Skip to content

OCPBUGS-60077: Silence ovnkube-lib.sh to avoid mass of logs to collectors#2761

Merged
openshift-merge-bot[bot] merged 1 commit intoopenshift:masterfrom
michaelalang:script-lib-noise-reduce
Feb 6, 2026
Merged

OCPBUGS-60077: Silence ovnkube-lib.sh to avoid mass of logs to collectors#2761
openshift-merge-bot[bot] merged 1 commit intoopenshift:masterfrom
michaelalang:script-lib-noise-reduce

Conversation

@michaelalang
Copy link
Contributor

silenced default output of ovnkube-lib.sh and made it ENV dependning instead

on ~12 days running pods the amount of logs procuded is fairly high

$ oc -n openshift-ovn-kubernetes get po -l app=ovnkube-node 
NAME                 READY   STATUS    RESTARTS      AGE
ovnkube-node-464zd   8/8     Running   8             12d
ovnkube-node-9rrtb   8/8     Running   8             12d
...
$ oc -n openshift-ovn-kubernetes logs $(oc -n openshift-ovn-kubernetes get po -l app=ovnkube-node -oname | head -1) ovn-acl-logging | wc -l
431762

unfortunately those logs are not helpful as they are bash's Print a trace of simple commands output only.
With cloning the set -x behavior form ovnkube.sh the PR makes the output ENV controll able reducing logs in the default setup.

@openshift-ci openshift-ci bot requested review from dougbtv and jcaamano August 4, 2025 07:59
@openshift-ci openshift-ci bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Aug 4, 2025
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Aug 4, 2025

Hi @michaelalang. Thanks for your PR.

I'm waiting for a openshift member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@michaelalang michaelalang changed the title OCPBUGS-4595(GRPA): Silence ovnkube-lib.sh to avoid mass of logs to collectors OCPBUGS-60077: Silence ovnkube-lib.sh to avoid mass of logs to collectors Aug 4, 2025
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Aug 4, 2025
@openshift-ci-robot
Copy link
Contributor

@michaelalang: This pull request references Jira Issue OCPBUGS-60077, which is invalid:

  • expected the bug to target the "4.20.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

silenced default output of ovnkube-lib.sh and made it ENV dependning instead

on ~12 days running pods the amount of logs procuded is fairly high

$ oc -n openshift-ovn-kubernetes get po -l app=ovnkube-node 
NAME                 READY   STATUS    RESTARTS      AGE
ovnkube-node-464zd   8/8     Running   8             12d
ovnkube-node-9rrtb   8/8     Running   8             12d
...
$ oc -n openshift-ovn-kubernetes logs $(oc -n openshift-ovn-kubernetes get po -l app=ovnkube-node -oname | head -1) ovn-acl-logging | wc -l
431762

unfortunately those logs are not helpful as they are bash's Print a trace of simple commands output only.
With cloning the set -x behavior form ovnkube.sh the PR makes the output ENV controll able reducing logs in the default setup.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot added the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label Aug 4, 2025
@kyrtapz
Copy link
Contributor

kyrtapz commented Aug 4, 2025

silenced default output of ovnkube-lib.sh and made it ENV dependning instead

on ~12 days running pods the amount of logs procuded is fairly high

$ oc -n openshift-ovn-kubernetes get po -l app=ovnkube-node 
NAME                 READY   STATUS    RESTARTS      AGE
ovnkube-node-464zd   8/8     Running   8             12d
ovnkube-node-9rrtb   8/8     Running   8             12d
...
$ oc -n openshift-ovn-kubernetes logs $(oc -n openshift-ovn-kubernetes get po -l app=ovnkube-node -oname | head -1) ovn-acl-logging | wc -l
431762

unfortunately those logs are not helpful as they are bash's Print a trace of simple commands output only. With cloning the set -x behavior form ovnkube.sh the PR makes the output ENV controll able reducing logs in the default setup.

This is not a problem for containers other than ovn-acl-logging which runs the script in a loop.
Other containers only print some lines at the start before starting the main executable.
Please limit the changes to start-audit-log-rotation()

@michaelalang michaelalang force-pushed the script-lib-noise-reduce branch from e7251d3 to 7f2b3e6 Compare August 4, 2025 14:53
@michaelalang michaelalang force-pushed the script-lib-noise-reduce branch from 7f2b3e6 to d69bd30 Compare August 4, 2025 14:54
@michaelalang
Copy link
Contributor Author

This is not a problem for containers other than ovn-acl-logging which runs the script in a loop. Other containers only print some lines at the start before starting the main executable. Please limit the changes to start-audit-log-rotation()

@kyrtapz done and thanks for the hint on being able to limit only the particular function <3

@kyrtapz
Copy link
Contributor

kyrtapz commented Aug 19, 2025

/ok-to-test

@openshift-ci openshift-ci bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 19, 2025
@openshift-bot
Copy link
Contributor

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

@openshift-ci openshift-ci bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Nov 18, 2025
@coderabbitai
Copy link

coderabbitai bot commented Nov 18, 2025

Walkthrough

Modified the start-audit-log-rotation() function in the OVN-Kubernetes script library to conditionally disable shell tracing when the OVNKUBE_SH_VERBOSE environment variable is not set to "true", reducing verbosity during audit log rotation.

Changes

Cohort / File(s) Summary
Script Library Configuration
bindata/network/ovn-kubernetes/common/008-script-lib.yaml
Disable shell tracing in start-audit-log-rotation() function when OVNKUBE_SH_VERBOSE is not "true"

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

  • Verify the conditional logic correctly checks the OVNKUBE_SH_VERBOSE variable state
  • Confirm the shell tracing disable command (set +x) is properly positioned within the function
✨ Finishing touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

Cache: Disabled due to data retention organization setting

Knowledge base: Disabled due to Reviews -> Disable Knowledge Base setting

📥 Commits

Reviewing files that changed from the base of the PR and between ad26055 and d69bd30.

📒 Files selected for processing (1)
  • bindata/network/ovn-kubernetes/common/008-script-lib.yaml (1 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
**

⚙️ CodeRabbit configuration file

-Focus on major issues impacting performance, readability, maintainability and security. Avoid nitpicks and avoid verbosity.

Files:

  • bindata/network/ovn-kubernetes/common/008-script-lib.yaml
🔇 Additional comments (1)
bindata/network/ovn-kubernetes/common/008-script-lib.yaml (1)

129-131: Solid targeted fix for audit log verbosity.

The conditional check correctly disables shell tracing for the start-audit-log-rotation() function when OVNKUBE_SH_VERBOSE is not explicitly set to "true". This directly addresses the excessive log volume from the audit log rotation loop (which runs every 30 seconds) while preserving verbose tracing when needed for debugging. The use of parameter expansion ${OVNKUBE_SH_VERBOSE:-} is safe and idiomatic.


Comment @coderabbitai help to get the list of available commands and usage tips.

@kyrtapz
Copy link
Contributor

kyrtapz commented Dec 8, 2025

/lgtm
/jira-refresh

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Dec 8, 2025
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Dec 8, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kyrtapz, michaelalang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 8, 2025
@kyrtapz
Copy link
Contributor

kyrtapz commented Dec 8, 2025

/jira refresh

@openshift-ci-robot openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Dec 8, 2025
@openshift-ci-robot
Copy link
Contributor

@kyrtapz: This pull request references Jira Issue OCPBUGS-60077, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.21.0) matches configured target version for branch (4.21.0)
  • bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @anuragthehatter

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-bot
Copy link
Contributor

/jira refresh

The requirements for Jira bugs have changed (Jira issues linked to PRs on main branch need to target different OCP), recalculating validity.

@kyrtapz
Copy link
Contributor

kyrtapz commented Feb 4, 2026

the bot seems to be ignoring my commands...
/shrug

@openshift-ci openshift-ci bot added the ¯\_(ツ)_/¯ ¯\\\_(ツ)_/¯ label Feb 4, 2026
@kyrtapz
Copy link
Contributor

kyrtapz commented Feb 4, 2026

/remove-lifecycle rotten

@kyrtapz
Copy link
Contributor

kyrtapz commented Feb 4, 2026

/jira refresh

@openshift-ci-robot
Copy link
Contributor

@kyrtapz: This pull request references Jira Issue OCPBUGS-60077, which is invalid:

  • expected the bug to target either version "4.22." or "openshift-4.22.", but it targets "4.21.z" instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@kyrtapz
Copy link
Contributor

kyrtapz commented Feb 4, 2026

/jira refresh

@openshift-ci-robot openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Feb 4, 2026
@openshift-ci-robot
Copy link
Contributor

@kyrtapz: This pull request references Jira Issue OCPBUGS-60077, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.22.0) matches configured target version for branch (4.22.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @anuragthehatter

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@kyrtapz
Copy link
Contributor

kyrtapz commented Feb 4, 2026

/retest-required

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Feb 4, 2026
@openshift-ci-robot
Copy link
Contributor

@kyrtapz: This PR has been marked as verified by CI.

Details

In response to this:

/verified by CI
https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_cluster-network-operator/2761/pull-ci-openshift-cluster-network-operator-master-e2e-aws-ovn-single-node/1957815852371808256/artifacts/e2e-aws-ovn-single-node/gather-extra/artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-qk2gg_ovn-acl-logging.log

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link
Contributor

/retest-required

Remaining retests: 0 against base HEAD 2b7b23c and 2 for PR HEAD d69bd30 in total

@openshift-ci-robot
Copy link
Contributor

/retest-required

Remaining retests: 0 against base HEAD f0bdee3 and 1 for PR HEAD d69bd30 in total

@kyrtapz
Copy link
Contributor

kyrtapz commented Feb 5, 2026

This failure is unrelated
/override ci/prow/e2e-aws-ovn-upgrade

@michaelalang
Copy link
Contributor Author

/cherry-pick release-4.21 release-4.20 release-4.19 release-4.18 release-4.17 release-4.16

@openshift-cherrypick-robot

@michaelalang: only openshift org members may request cherry picks. If you are already part of the org, make sure to change your membership to public. Otherwise you can still do the cherry-pick manually.

Details

In response to this:

/cherry-pick release-4.21 release-4.20 release-4.19 release-4.18 release-4.17 release-4.16

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@kyrtapz
Copy link
Contributor

kyrtapz commented Feb 5, 2026

/cherry-pick release-4.21 release-4.20 release-4.19 release-4.18 release-4.17 release-4.16

@openshift-cherrypick-robot

@kyrtapz: once the present PR merges, I will cherry-pick it on top of release-4.21 in a new PR and assign it to you.

Details

In response to this:

/cherry-pick release-4.21 release-4.20 release-4.19 release-4.18 release-4.17 release-4.16

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Feb 5, 2026

@kyrtapz: Overrode contexts on behalf of kyrtapz: ci/prow/e2e-aws-ovn-upgrade

Details

In response to this:

This failure is unrelated
/override ci/prow/e2e-aws-ovn-upgrade

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@kyrtapz
Copy link
Contributor

kyrtapz commented Feb 5, 2026

/retest-required

@michaelalang
Copy link
Contributor Author

/retest

@kyrtapz
Copy link
Contributor

kyrtapz commented Feb 6, 2026

/retest-required

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Feb 6, 2026

@michaelalang: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/4.20-upgrade-from-stable-4.19-e2e-azure-ovn-upgrade d69bd30 link false /test 4.20-upgrade-from-stable-4.19-e2e-azure-ovn-upgrade
ci/prow/e2e-aws-ovn-serial d69bd30 link false /test e2e-aws-ovn-serial
ci/prow/4.20-upgrade-from-stable-4.19-e2e-gcp-ovn-upgrade d69bd30 link false /test 4.20-upgrade-from-stable-4.19-e2e-gcp-ovn-upgrade
ci/prow/4.20-upgrade-from-stable-4.19-e2e-aws-ovn-upgrade d69bd30 link false /test 4.20-upgrade-from-stable-4.19-e2e-aws-ovn-upgrade
ci/prow/security d69bd30 link false /test security
ci/prow/e2e-ovn-hybrid-step-registry d69bd30 link false /test e2e-ovn-hybrid-step-registry
ci/prow/e2e-aws-hypershift-ovn-kubevirt d69bd30 link false /test e2e-aws-hypershift-ovn-kubevirt
ci/prow/4.22-upgrade-from-stable-4.21-e2e-aws-ovn-upgrade d69bd30 link false /test 4.22-upgrade-from-stable-4.21-e2e-aws-ovn-upgrade
ci/prow/4.22-upgrade-from-stable-4.21-e2e-azure-ovn-upgrade d69bd30 link false /test 4.22-upgrade-from-stable-4.21-e2e-azure-ovn-upgrade

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-merge-bot openshift-merge-bot bot merged commit 9d540eb into openshift:master Feb 6, 2026
43 of 48 checks passed
@openshift-ci-robot
Copy link
Contributor

@michaelalang: Jira Issue Verification Checks: Jira Issue OCPBUGS-60077
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-60077 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

Details

In response to this:

silenced default output of ovnkube-lib.sh and made it ENV dependning instead

on ~12 days running pods the amount of logs procuded is fairly high

$ oc -n openshift-ovn-kubernetes get po -l app=ovnkube-node 
NAME                 READY   STATUS    RESTARTS      AGE
ovnkube-node-464zd   8/8     Running   8             12d
ovnkube-node-9rrtb   8/8     Running   8             12d
...
$ oc -n openshift-ovn-kubernetes logs $(oc -n openshift-ovn-kubernetes get po -l app=ovnkube-node -oname | head -1) ovn-acl-logging | wc -l
431762

unfortunately those logs are not helpful as they are bash's Print a trace of simple commands output only.
With cloning the set -x behavior form ovnkube.sh the PR makes the output ENV controll able reducing logs in the default setup.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-cherrypick-robot

@kyrtapz: new pull request created: #2900

Details

In response to this:

/cherry-pick release-4.21 release-4.20 release-4.19 release-4.18 release-4.17 release-4.16

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-merge-robot
Copy link
Contributor

Fix included in accepted release 4.22.0-0.nightly-2026-02-08-124411

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. verified Signifies that the PR passed pre-merge verification criteria ¯\_(ツ)_/¯ ¯\\\_(ツ)_/¯

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants