bookworm container upgrade for lldp, net-snmp #18150

mohan-selvaraj · 2024-02-21T19:04:16Z

Why I did it

Update LLDP, net-snmp containers to Bookworm

Work item tracking

Microsoft ADO (number only):

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Tested branch (Please provide the tested image version)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

linux-foundation-easycla · 2024-02-21T19:04:20Z

The committers listed above are authorized under a signed CLA.

✅ login: mohan-selvaraj (1c93393, 8ac21a4, eee618a, 0dcf92a, c028a96, 4c22c94, 82a79ae, cb6b78a, 1c7ef70, a6a135f, 5b3195e, a167f06, c0f193c, e137b62, 1e4942e, 35b3a91, 786843d, 3ee040f, 16ff1a9, f2d05dd)
✅ login: Kalimuthu-Velappan (bd3032a, cdc6491, 486e018)

saiarcot895 · 2024-02-22T01:00:25Z

The PR build is failing because of build failures in net-snmp. Please look into these failures. The package may need to be updated to 5.9.3+dfsg-2 (similar to the upgrade done for Bullseye).

…e into lldp_bkwm

mohan-selvaraj · 2024-02-29T07:03:19Z

The PR build is failing because of build failures in net-snmp. Please look into these failures. The package may need to be updated to 5.9.3+dfsg-2 (similar to the upgrade done for Bullseye).

5.9.3+dfsg-2 wasn't available in the sonicblob. Hence download failed. Used the existing package 5.9+dfsg-4 for net-snmp. Now following errors are seen for bookworm build but not seen for bullseye build.

Any inputs on resolving this ?

2024-02-27T16:28:43.1201411Z [ FAIL LOG START ] [ target/debs/bookworm/libnl-nf-3-dev_3.5.0-1_amd64.deb-install ]
2024-02-27T16:28:43.1226890Z Build start time: Tue Feb 27 16:28:32 UTC 2024
2024-02-27T16:28:43.1227257Z dpkg: error: dpkg frontend lock was locked by another process with pid 197664
2024-02-27T16:28:43.1227605Z Note: removing the lock file is always wrong, can damage the locked area
2024-02-27T16:28:43.1228270Z and the entire system. See <https://wiki.debian.org/Teams/Dpkg/FAQ#db-lock>.
2024-02-27T16:28:43.1229042Z [  FAIL LOG END  ] [ target/debs/bookworm/libnl-nf-3-dev_3.5.0-1_amd64.deb-install ]

2024-02-27T16:28:43.3395027Z make: *** [slave.mk:848: target/debs/bookworm/libnl-nf-3-dev_3.5.0-1_amd64.deb-install] Error 1
2024-02-27T16:28:43.3395389Z make: *** Waiting for unfinished jobs....

2024-02-27T16:30:11.9310522Z transports/snmpTLSBaseDomain.c:59:22: error: static declaration of 'ERR_get_error_all' follows non-static declaration
2024-02-27T16:30:11.9310751Z    59 | static unsigned long ERR_get_error_all(const char **file, int *line,
2024-02-27T16:30:11.9310991Z       |                      ^~~~~~~~~~~~~~~~~

2024-02-27T16:30:11.9313469Z make[4]: *** [Makefile:101: transports/snmpTLSBaseDomain.lo] Error 1
2024-02-27T16:30:11.9313629Z make[4]: *** Waiting for unfinished jobs....

saiarcot895 · 2024-02-29T07:43:42Z

5.9.3+dfsg-2 should now be available from sonicblob, can you test it?

saiarcot895 · 2024-02-29T07:44:06Z

Makefile.work

@@ -311,8 +311,8 @@ endif
 ifeq ($(DOCKER_BUILDER_WORKDIR),)
 override DOCKER_BUILDER_WORKDIR := "/sonic"
 endif
-
-DOCKER_RUN := docker run --rm=true --privileged --init \


It looks like there are a lot of changes here, are these meant to be local/debug changes?

These changes are automatically added after rebase with master branch.

You may need to recheck the branch rebases, there are changes in this PR that shouldn't be here.

@maipbui to keep eyes on it.

Those changes were done to do a local build. Got pushed by mistake. Have removed those changes in the latest commit.

Build went through fine. Failure seen in kvmtest->Prepare testbed step. Unable to access the detailed logs. Do you have any inputs on this.

I thought the issue of not being able to see the test logs was fixed. At any rate, here is the failure:

12:54:21 recover.adaptive_recover L0169 WARNING| Restoring {'failed': True, 'check_item': 'processes', 'host': 'vlab-01', 'processes_status': {'pmon': {'status': True, 'exited_critical_process': [], 'running_critical_process': []}, 'snmp': {'status': False, 'exited_critical_process': ['snmp-subagent'], 'running_critical_process': ['snmpd']}, 'lldp': {'status': True, 'exited_critical_process': [], 'running_critical_process': ['lldp-syncd', 'lldpd', 'lldpmgrd']}, 'database': {'status': True, 'exited_critical_process': [], 'running_critical_process': ['redis']}, 'bgp': {'status': True, 'exited_critical_process': [], 'running_critical_process': ['bgpcfgd', 'bgpd', 'fpmsyncd', 'staticd', 'zebra']}, 'swss': {'status': True, 'exited_critical_process': [], 'running_critical_process': ['buffermgrd', 'coppmgrd', 'fabricmgrd', 'fdbsyncd', 'intfmgrd', 'nbrmgrd', 'neighsyncd', 'orchagent', 'portmgrd', 'portsyncd', 'tunnelmgrd', 'vlanmgrd', 'vrfmgrd', 'vxlanmgrd']}, 'syncd': {'status': True, 'exited_critical_process': [], 'running_critical_process': ['syncd']}, 'teamd': {'status': True, 'exited_critical_process': [], 'running_critical_process': ['teammgrd', 'teamsyncd', 'tlm_teamd']}}, 'services_status': {'pmon': True, 'snmp': False, 'lldp': True, 'database': True, 'bgp': True, 'swss': True, 'syncd': True, 'teamd': True}} with proposed action: config_reload, final action: config_reload

snmp-subagent process is supposed to be running, but doesn't appear to be running.

Following patches are available in 5.9.3 codebase. They need not be included.
0001-SNMP-Stop-spamming-logs-with-statfs-permission-denie.patch
0002-at.c-properly-check-return-status-from-realloc.-Than.patch
0003-CHANGES-BUG-2743-snmpd-crashes-when-receiving-a-GetN.patch
0006-From-Jiri-Cervenka-snmpd-Fixed-agentx-crashing-and-or-freezing-on-timeout.patch
0009-Makefile.in-agent-Makefile.in-Fix-parallel-compilati.patch
0010-Makefile.in-Make-sure-that-sedscript-is-built-before.patch
0011-agent-Makefile.in-Build-the-MIB-module-code-once.patch

Following patch looks specific to 5.7.3 version. Will not add this.
0007-Linux-VRF-5.7.3-Support.patch

Following patch is described as "Enable macro DEB_BUILD_ARCH_OS in order to build ipv6 feature". But it doesn't apply.
0008-Enable-macro-DEB_BUILD_ARCH_OS-in-order-to-build-ipv.patch
Following patch is required but doesn't apply
cross-compile-changes.patch

Following patches are required.
0012-agent-Makefile.in-Unbreak-the-enable-minimalist-buil.patch
0013-enable-parallel-build-for-net-snmp.patch

Next step is to add the following patches to 5.9.3.
0008-Enable-macro-DEB_BUILD_ARCH_OS-in-order-to-build-ipv.patch
0012-agent-Makefile.in-Unbreak-the-enable-minimalist-buil.patch
0013-enable-parallel-build-for-net-snmp.patch
cross-compile-changes.patch

snmp-subagent still isn't up. How to access the process logs?

"docker exec snmp supervisorctl status" dependent-startup RUNNING pid 7, uptime 0:00:10 rsyslogd RUNNING pid 20, uptime 0:00:08 snmp-subagent BACKOFF Exited too quickly (process log may have details) snmpd RUNNING pid 24, uptime 0:00:08 start EXITED Mar 15 11:03 AM supervisor-proc-exit-listener RUNNING pid 8, uptime 0:00:10

@qiluo-msft ,
Need your help to understand what exactly the failure is because of which snmp-subagent is exiting.

mohan-selvaraj · 2024-02-29T07:45:56Z

5.9.3+dfsg-2 should now be available from sonicblob, can you test it?

sure. will try and update

…e into lldp_bkwm

added the patch directory for snmpd

mohan-selvaraj · 2024-03-14T05:55:55Z

/azpw run

mssonicbld · 2024-03-14T05:55:57Z

/AzurePipelines run

azure-pipelines · 2024-03-14T05:56:04Z

You have several pipelines (over 10) configured to build pull requests in this repository. Specify which pipelines you would like to run by using /azp run [pipelines] command. You can specify multiple pipelines using a comma separated list.

added patches for snmp

mohan-selvaraj · 2024-03-22T17:36:35Z

@qiluo-msft ,

Have added the patches in snmpd based on code comparison.
Also rebased the lldp_bkwm branch to the latest master.
I still see that the snmp_subagent process doesn't come up.

"docker exec snmp supervisorctl status"
dependent-startup                RUNNING   pid 7, uptime 0:00:10
rsyslogd                         RUNNING   pid 20, uptime 0:00:08
snmp-subagent                    BACKOFF   Exited too quickly (process log may have details)
snmpd                            RUNNING   pid 24, uptime 0:00:08
start                            EXITED    Mar 15 11:03 AM
supervisor-proc-exit-listener    RUNNING   pid 8, uptime 0:00:10

Need your help to understand what exactly the failure is because of which snmp-subagent is exiting.

saiarcot895 · 2024-03-27T16:56:14Z

@mohan-selvaraj, you should be able to see this issue when loading the image on a setup (either KVM or physical). After the image is loaded, check /var/log/syslog to see why snmp-subagent is exiting/failing to start.

mohan-selvaraj · 2024-03-29T09:27:24Z

Additional pull request is created for changes in src/snmp-agent
sonic-net/sonic-snmpagent#313

With the changes in snmpagent, following is the process status

root@sonic:/home/admin# docker exec snmp supervisorctl status
dependent-startup                EXITED    Mar 29 09:00 AM
rsyslogd                         RUNNING   pid 20, uptime 0:24:55
snmp-subagent                    RUNNING   pid 25, uptime 0:24:53
snmpd                            RUNNING   pid 24, uptime 0:24:55
start                            EXITED    Mar 29 08:59 AM
supervisor-proc-exit-listener    RUNNING   pid 8, uptime 0:24:56
root@sonic:/home/admin#

root@sonic:/home/admin# docker exec -it snmp bash
root@sonic:/# ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 08:59 pts/0    00:00:00 /usr/bin/python3 /usr/local/bin/supervisord
root           8       1  0 08:59 pts/0    00:00:00 python3 /usr/bin/supervisor-proc-exit-listener --container-name snmp
root          20       1  0 08:59 pts/0    00:00:00 /usr/sbin/rsyslogd -n -iNONE
Debian-+      24       1  0 08:59 pts/0    00:00:00 /usr/sbin/snmpd -f -Ls0-2d -u Debian-snmp -g Debian-snmp -I -smux mteTrigger mteTriggerConf ifTable ifXTable inetCidrRouteTable ipCidrRouteTable ip disk_hw -p /run/snmpd.pid
root          25       1  5 08:59 pts/0    00:01:22 python3 -m sonic_ax_impl
root         172       0  2 09:23 pts/1    00:00:00 bash
root         178     172  0 09:23 pts/1    00:00:00 ps -ef

syslogs

...
Mar 29 09:00:00.981657 sonic INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.2.2.1.21]
Mar 29 09:00:00.981910 sonic INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.2.2.1.22]
Mar 29 09:00:00.982043 sonic INFO snmp#snmp-subagent [ax_interface] INFO: OID registration complete. Waiting to receive PDUs...
Mar 29 09:00:00.989625 sonic INFO snmp#supervisord 2024-03-29 09:00:00,988 INFO exited: dependent-startup (exit status 0; expected)

mohan-selvaraj · 2024-03-29T10:22:34Z

/azpw run

mohan-selvaraj · 2024-03-30T05:24:55Z

/azpw run

mohan-selvaraj · 2024-04-02T09:07:51Z

/AzurePipelines run

azure-pipelines · 2024-04-02T09:07:58Z

Commenter does not have sufficient privileges for PR 18150 in repo sonic-net/sonic-buildimage

mohan-selvaraj · 2024-04-03T06:19:33Z

/azpw run Azure.sonic-buildimage

mssonicbld · 2024-04-03T06:19:35Z

/AzurePipelines run Azure.sonic-buildimage

azure-pipelines · 2024-04-03T06:19:47Z

Azure Pipelines successfully started running 1 pipeline(s).

saiarcot895 · 2024-04-04T00:34:37Z

@mohan-selvaraj Until the snmpagent submodule gets merged, it doesn't make sense to rerun the pipeline here.

adyeung · 2024-04-04T19:40:02Z

Blocked on #313, expecting build test to pass after the merge of snmp subagent change

saiarcot895 · 2024-04-11T23:14:35Z

@mohan-selvaraj could you trigger a rebuild when you get a chance?

saiarcot895 · 2024-04-12T01:44:37Z

src/snmpd/patch-5.9.3+dfsg/series

@@ -0,0 +1,4 @@
+0008-Enable-macro-DEB_BUILD_ARCH_OS-in-order-to-build-ipv.patch  


I see 0001-SNMP-Stop-spamming-logs-with-statfs-permission-denie.patch, 0009-Makefile.in-agent-Makefile.in-Fix-parallel-compilati.patch, 0010-Makefile.in-Make-sure-that-sedscript-is-built-before.patch, 0011-agent-Makefile.in-Build-the-MIB-module-code-once.patch, and 0012-agent-Makefile.in-Unbreak-the-enable-minimalist-buil.patch were dropped. But it doesn't seem like the changes in those patches are in 5.9.3. Is there a reason these patches were dropped?

Can you confirm if the code in the dropped patches aren't present in the 5.9.3 codebase.
I had an earlier comment on which patches were required and which can be ignored.

0001-SNMP-Stop-spamming-logs-with-statfs-permission-denie.patch is needed, 0008-Enable-macro-DEB_BUILD_ARCH_OS-in-order-to-build-ipv.patch can be optionally dropped (since that logic is now coming from include /usr/share/dpkg/architecture.mk in debian/rules), 0012-agent-Makefile.in-Unbreak-the-enable-minimalist-buil.patch is needed. The others are correct.

@saiarcot895
As I understand, the only reason why we build snmp packages (libsnmp-base, snmptrapd, snmp, snmpd, libsnmp40, libsnmp-dev, libsnmp-perl, tkmib) from source is 0001-SNMP-Stop-spamming-logs-with-statfs-permission-denie.patch. Maybe it's possible to find another solution without modifyng source code.

Good point. Based on what's in sonic-net/sonic-snmpagent#22, there might not be any changes needed at all. For now, I prefer to let this get merged in, and then look at removing the snmp build (either in time for 202405 release or after the branch cutoff).

@saiarcot895

We should remove 0001-SNMP-Stop-spamming-logs-with-statfs-permission-denie.patch
fsys_mntctl.c is for AIX: https://github.com/net-snmp/net-snmp/blob/59acd6e6fcfccfd3456ec8a65816ca76036e142f/agent/mibgroup/hardware/fsys.h#L2
Original bug in fsys_mntent.c has been already fixed in upstream: bvanassche/net-snmp@5f1986c

…atch

…e into lldp_bkwm

bookworm container upgrade for lldp

c028a96

mohan-selvaraj requested review from qiluo-msft, xumia and lguohan as code owners February 21, 2024 19:04

Kalimuthu-Velappan added 2 commits February 21, 2024 11:46

Fix for build hooks debian permission error

cdc6491

Changes for building community master in broadcom LSF servers-Jan'24

486e018

Kalimuthu-Velappan and others added 2 commits February 21, 2024 22:45

Changes for building community master in broadcom LSF servers

bd3032a

bookworm container upgrade for net-snmp

8ac21a4

mohan-selvaraj changed the title ~~bookworm container upgrade for lldp~~ bookworm container upgrade for lldp, net-snmp Feb 27, 2024

mohan-selvaraj added 4 commits February 27, 2024 03:33

bookworm container upgrade for net-snmp

1c93393

Merge branch 'lldp_bkwm' of github.com:mohan-selvaraj/sonic-buildimag…

cb6b78a

…e into lldp_bkwm

Merge branch 'lldp_bkwm' of github.com:mohan-selvaraj/sonic-buildimag…

0dcf92a

…e into lldp_bkwm

Merge branch 'lldp_bkwm' of github.com:mohan-selvaraj/sonic-buildimag…

4c22c94

…e into lldp_bkwm

saiarcot895 reviewed Feb 29, 2024

View reviewed changes

mohan-selvaraj added 2 commits February 29, 2024 12:19

Merge branch 'lldp_bkwm' of github.com:mohan-selvaraj/sonic-buildimag…

eee618a

…e into lldp_bkwm

Merge branch 'lldp_bkwm' of github.com:mohan-selvaraj/sonic-buildimag…

c0f193c

…e into lldp_bkwm

qiluo-msft requested a review from maipbui March 7, 2024 21:04

saiarcot895 mentioned this pull request Mar 7, 2024

Frr bookworm upgrade #18233

Merged

11 tasks

mohan-selvaraj added 4 commits March 12, 2024 23:44

bookworm container upgrade for lldp, net-snmp

1c7ef70

Merge branch 'lldp_bkwm' of github.com:mohan-selvaraj/sonic-buildimag…

5b3195e

…e into lldp_bkwm

bookworm container upgrade for lldp, net-snmp

a6a135f

bookworm container upgrade for lldp, net-snmp

786843d

added the patch directory for snmpd

saiarcot895 added the Bookworm label Mar 13, 2024

mohan-selvaraj and others added 2 commits March 15, 2024 01:48

bookworm container upgrade for lldp, net-snmp

16ff1a9

added patches for snmp

Merge branch 'sonic-net:master' into lldp_bkwm

82a79ae

update supervisord.conf file to use correct stdout and stderr flags

a167f06

mohan-selvaraj mentioned this pull request Mar 29, 2024

snmp-subagent changes for bookworm upgrade sonic-net/sonic-snmpagent#313

Merged

mohan-selvaraj requested a review from saiarcot895 March 29, 2024 10:01

Merge branch 'sonic-net:master' into lldp_bkwm

35b3a91

Merge branch 'sonic-net:master' into lldp_bkwm

3ee040f

saiarcot895 reviewed Apr 12, 2024

View reviewed changes

mohan-selvaraj added 3 commits April 15, 2024 23:37

Add patch 0001-SNMP-Stop-spamming-logs-with-statfs-permission-denie.p…

1e4942e

…atch

Merge branch 'master' into lldp_bkwm

e137b62

Merge branch 'lldp_bkwm' of github.com:mohan-selvaraj/sonic-buildimag…

f2d05dd

…e into lldp_bkwm

saiarcot895 approved these changes Apr 18, 2024

View reviewed changes

yxieca merged commit df499b6 into sonic-net:master Apr 22, 2024
19 checks passed

adyeung mentioned this pull request Apr 26, 2024

Bookworm Upgrade LLDP, SNMP subagent, ICCPD, PDE, NAT, FRR sonic-net/SONiC#1677

Closed

saiarcot895 mentioned this pull request May 1, 2024

SONiC container upgrade to Bookworm (Debian 12) sonic-net/SONiC#1541

Closed

		@@ -0,0 +1,4 @@
		0008-Enable-macro-DEB_BUILD_ARCH_OS-in-order-to-build-ipv.patch

bookworm container upgrade for lldp, net-snmp #18150

bookworm container upgrade for lldp, net-snmp #18150

Conversation

mohan-selvaraj commented Feb 21, 2024 • edited Loading

Why I did it

Work item tracking

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Tested branch (Please provide the tested image version)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

linux-foundation-easycla bot commented Feb 21, 2024 • edited Loading

saiarcot895 commented Feb 22, 2024

mohan-selvaraj commented Feb 29, 2024

saiarcot895 commented Feb 29, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mohan-selvaraj commented Feb 29, 2024

mohan-selvaraj commented Mar 14, 2024

mssonicbld commented Mar 14, 2024

azure-pipelines bot commented Mar 14, 2024

mohan-selvaraj commented Mar 22, 2024

saiarcot895 commented Mar 27, 2024

mohan-selvaraj commented Mar 29, 2024

mohan-selvaraj commented Mar 29, 2024

mohan-selvaraj commented Mar 30, 2024

mohan-selvaraj commented Apr 2, 2024

azure-pipelines bot commented Apr 2, 2024

mohan-selvaraj commented Apr 3, 2024

mssonicbld commented Apr 3, 2024

azure-pipelines bot commented Apr 3, 2024

saiarcot895 commented Apr 4, 2024

adyeung commented Apr 4, 2024

saiarcot895 commented Apr 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

saiarcot895 Apr 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k-v1 Apr 19, 2024 • edited Loading

Choose a reason for hiding this comment

mohan-selvaraj commented Feb 21, 2024 •

edited

Loading

linux-foundation-easycla bot commented Feb 21, 2024 •

edited

Loading

saiarcot895 Apr 16, 2024 •

edited

Loading

k-v1 Apr 19, 2024 •

edited

Loading