[Logrotate] Add log rotate configuration #15116

fastiuk · 2023-05-17T15:05:30Z

depends-on #61

Why I did it

No be able to do the next things:

Perform log rotation separately for syslog and debug files
Configure log rotation parameters like rotation frequency, file size, file rotation count

Work item tracking

Microsoft ADO (number only):

How I did it

Add YANG model for a new configuration
Extend configuration templates to support new knobs

How to verify it

Which release branch to backport (provide reason below if selected)

Tested branch (Please provide the tested image version)

Description for the changelog

Add the ability Perform log rotation separately for syslog and debug files
Configure log rotation parameters like rotation frequency, file size, file rotation count

Link to config_db schema for YANG module changes

Logrotation config

A picture of a cute animal (it is my cat Finn)

files/build_templates/init_cfg.json.j2

src/sonic-yang-models/yang-models/sonic-logging.yang

dgsudharsan · 2023-05-19T20:50:55Z

files/image_config/logrotate/templates/logrotate-common.j2

@@ -8,7 +49,6 @@
 /var/log/kern.log
 /var/log/user.log
 /var/log/lpr.log
-/var/log/debug


What is the purpose of differentiating debug message alone through separate configuration?

We use the debug file separately, so we need to have a separation. By default that file has the same configuration as it had before.

In my opinion we need separate file only if debugs are much more and require special attention. I don't think that's the case in sonic as we have all debug logs as part of syslog itself and I am not sure if debug file is used.

If you don't want to rotate it, you just don't configure it. For the rest of people who want to use debug file and rotate they can configure it in the DB and rotate.

files/image_config/logrotate/logrotate-config.sh

dgsudharsan

Please update configuration guide Configuration.md

dgsudharsan · 2023-05-20T04:14:09Z

files/build_templates/defaults.j2

@@ -0,0 +1,13 @@
+{# Don't use that file to generate something, it is just for holding defaults #}


Why do we need a different file for setting defaults. Can't we incorporate this in init_cfg.j2?

The default is shared between two configuration files. I don't want to explicitly incorporate debug/syslog files in the configuration. The default values here are used to fill data if the configuration in init_cfg.j2 doesn't exist.

This is default only for syslog related configuration. I would prefer renaming the file to avoid configuration

I am going to remove that file. I remember we discussed that approach with Ying Xie and decided to remove defaults from that file and use defaults in-place

files/image_config/logrotate/templates/logrotate-syslog.j2

Meir-renford · 2023-09-04T06:56:16Z

We reviewed the code change needed, and will address it in the upcoming 2 weeks.

* Separate debug log rotation config * Separate syslog log rotation config * Add CONFIG_DB LOGGING table to hold logging configuration Signed-off-by: Yevhen Fastiuk <[email protected]>

Signed-off-by: Yevhen Fastiuk <[email protected]>

liat-grozovik · 2024-07-09T16:55:59Z

@dgsudharsan any further comments?

files/build_templates/init_cfg.json.j2

src/sonic-yang-models/yang-models/sonic-logging.yang

dgsudharsan · 2024-07-10T07:19:00Z

files/build_templates/defaults.j2

@@ -0,0 +1,13 @@
+{# Don't use that file to generate something, it is just for holding defaults #}


This is default only for syslog related configuration. I would prefer renaming the file to avoid configuration

dgsudharsan · 2024-07-10T07:21:40Z

files/image_config/logrotate/templates/logrotate-common.j2

@@ -8,7 +49,6 @@
 /var/log/kern.log
 /var/log/user.log
 /var/log/lpr.log
-/var/log/debug


In my opinion we need separate file only if debugs are much more and require special attention. I don't think that's the case in sonic as we have all debug logs as part of syslog itself and I am not sure if debug file is used.

src/sonic-yang-models/yang-models/sonic-logging.yang

files/image_config/logrotate/logrotate-config.sh

dgsudharsan · 2024-07-10T09:32:48Z

files/image_config/logrotate/logrotate-config.sh

+sonic-cfggen -d -t /usr/share/sonic/templates/logrotate-debug.j2 \
+             -a "$APPEND_DATA" > /etc/logrotate.d/debug
+
+sonic-cfggen -d -t /usr/share/sonic/templates/logrotate-syslog.j2 \


This file was not required previously. Why is this required now. Isn't rsyslog enough?

rsyslog was split into common and per-logfile config.
And the reason why it was done like that, because once config file will be generated, you can use to rotate corresponding log file manually, and not all files together.

dgsudharsan · 2024-07-10T09:37:06Z

files/image_config/logrotate/templates/logrotate-syslog.j2

+{% set size = (size_mb * 1024 * 1024) | int -%}
+/var/log/syslog
+{
+    {{ frequency }}


Can you clarify what does this frequency control? The frequency of logrotate itself is being controlled by timer. Why is frequency needed here?

this is weekly, daily, monthly etc. https://linux.die.net/man/8/logrotate#:~:text=Each%20log%20file%20may%20be%20handled%20daily%2C%20weekly%2C%20monthly%2C%20or%20when%20it%20grows%20too%20large.

dgsudharsan · 2024-07-10T09:51:23Z

@dgsudharsan any further comments?

@liat-grozovik Left few comments.

Signed-off-by: Yevhen Fastiuk <[email protected]>

fastiuk · 2024-08-14T12:18:39Z

/azpw run Azure.sonic-buidimage

mssonicbld · 2024-08-14T12:18:42Z

/AzurePipelines run Azure.sonic-buidimage

azure-pipelines · 2024-08-14T12:18:48Z

No pipelines are associated with this pull request.

Signed-off-by: Yevhen Fastiuk <[email protected]>

fastiuk · 2024-08-28T09:38:45Z

Tests will fail until sonic-net/sonic-host-services#61 is fixed

src/sonic-yang-models/yang-models/sonic-logging.yang

liat-grozovik · 2024-12-03T17:28:53Z

/azpw run Azure.sonic-buildimage

saiarcot895 · 2024-12-03T18:34:58Z

files/image_config/logrotate/templates/logrotate-debug.j2

+    {% set size_mb = (debug.size | d(0.001)) | float -%}
+{%- endif %}
+{% set size = (size_mb * 1024 * 1024) | int %}
+/var/log/debug


I don't think this log file is used at all on SONiC?

fastiuk requested review from qiluo-msft and lguohan as code owners May 17, 2023 15:05

dgsudharsan added the YANG YANG model related changes label May 18, 2023

dgsudharsan suggested changes May 20, 2023

View reviewed changes

fastiuk force-pushed the dev-logrotate-config branch from 586be93 to 2c99088 Compare July 4, 2023 22:09

fastiuk self-assigned this Jul 11, 2023

fastiuk force-pushed the dev-logrotate-config branch from e8258c1 to 2d17ed8 Compare July 22, 2023 11:12

SviatoslavBoichuk force-pushed the dev-logrotate-config branch 2 times, most recently from 11e2e6f to e3a3c6b Compare December 19, 2023 08:16

SviatoslavBoichuk force-pushed the dev-logrotate-config branch from e3a3c6b to 1e8368c Compare January 22, 2024 15:14

SviatoslavBoichuk force-pushed the dev-logrotate-config branch from 1e8368c to f7fad71 Compare February 27, 2024 12:03

SviatoslavBoichuk mentioned this pull request Feb 28, 2024

Added logrotate feature HLD SviatoslavBoichuk/SONiC#2

Closed

fastiuk mentioned this pull request Apr 27, 2024

[Logrotate] Add logrotate feature HLD sonic-net/SONiC#1683

Open

fastiuk added 2 commits April 28, 2024 02:06

Add log rotation configuration mechanism

4d9b427

* Separate debug log rotation config * Separate syslog log rotation config * Add CONFIG_DB LOGGING table to hold logging configuration Signed-off-by: Yevhen Fastiuk <[email protected]>

Pin max size of logging to static value

7f89174

Signed-off-by: Yevhen Fastiuk <[email protected]>

fastiuk force-pushed the dev-logrotate-config branch from f7fad71 to 7f89174 Compare April 27, 2024 23:07

dgsudharsan suggested changes Jul 10, 2024

View reviewed changes

fastiuk added 4 commits August 13, 2024 02:58

Logrotate: Fix according to GitHub comments

6868b21

Signed-off-by: Yevhen Fastiuk <[email protected]>

Merge branch 'master' into dev-logrotate-config

1e6c5dc

Logrotate: Don't copy defaults file anymore

91179d7

Signed-off-by: Yevhen Fastiuk <[email protected]>

Merge remote-tracking branch 'origin/master' into dev-logrotate-config

dfd71e6

fastiuk closed this Aug 14, 2024

fastiuk reopened this Aug 14, 2024

fastiuk closed this Aug 19, 2024

fastiuk reopened this Aug 19, 2024

fastiuk added 2 commits August 21, 2024 02:13

Logrotate: Fix logical mistake in template

b30a3fe

Signed-off-by: Yevhen Fastiuk <[email protected]>

Logrotate: Decrease syslog size to 16M

7807468

Signed-off-by: Yevhen Fastiuk <[email protected]>

fastiuk closed this Aug 22, 2024

fastiuk reopened this Aug 22, 2024

Merge branch 'master' into dev-logrotate-config

ae7b4c3

maipbui reviewed Oct 9, 2024

View reviewed changes

src/sonic-yang-models/yang-models/sonic-logging.yang Outdated Show resolved Hide resolved

Rename namespace in YANG model

544937b

saiarcot895 mentioned this pull request Nov 10, 2024

[Logrotate] Add config and show commands sonic-net/sonic-utilities#3293

Open

fastiuk added 2 commits November 12, 2024 00:47

Merge branch 'master' into dev-logrotate-config

c4c32e2

Merge branch 'master' into dev-logrotate-config

77e864f

saiarcot895 reviewed Dec 3, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Logrotate] Add log rotate configuration #15116

[Logrotate] Add log rotate configuration #15116

fastiuk commented May 17, 2023 •

edited

Loading

dgsudharsan May 19, 2023

fastiuk Jun 1, 2023

dgsudharsan Jul 10, 2024

fastiuk Aug 12, 2024

dgsudharsan left a comment

dgsudharsan May 20, 2023

fastiuk Jun 1, 2023

dgsudharsan Jul 10, 2024

fastiuk Aug 12, 2024

Meir-renford commented Sep 4, 2023

liat-grozovik commented Jul 9, 2024

dgsudharsan Jul 10, 2024

dgsudharsan Jul 10, 2024

dgsudharsan Jul 10, 2024

fastiuk Aug 12, 2024

dgsudharsan Jul 10, 2024

fastiuk Aug 12, 2024

dgsudharsan commented Jul 10, 2024

fastiuk commented Aug 14, 2024

mssonicbld commented Aug 14, 2024

azure-pipelines bot commented Aug 14, 2024

fastiuk commented Aug 28, 2024 •

edited

Loading

liat-grozovik commented Dec 3, 2024

saiarcot895 Dec 3, 2024

		@@ -0,0 +1,13 @@
		{# Don't use that file to generate something, it is just for holding defaults #}

[Logrotate] Add log rotate configuration #15116

Are you sure you want to change the base?

[Logrotate] Add log rotate configuration #15116

Conversation

fastiuk commented May 17, 2023 • edited Loading

Why I did it

Work item tracking

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Tested branch (Please provide the tested image version)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (it is my cat Finn)

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dgsudharsan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Meir-renford commented Sep 4, 2023

liat-grozovik commented Jul 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dgsudharsan commented Jul 10, 2024

fastiuk commented Aug 14, 2024

mssonicbld commented Aug 14, 2024

azure-pipelines bot commented Aug 14, 2024

fastiuk commented Aug 28, 2024 • edited Loading

liat-grozovik commented Dec 3, 2024

Choose a reason for hiding this comment

fastiuk commented May 17, 2023 •

edited

Loading

fastiuk commented Aug 28, 2024 •

edited

Loading