Skip to content

Bugfix/28537460 auto assessment timeout 10min - #384

Open
SathishMSFT wants to merge 7 commits into
masterfrom
bugfix/28537460-auto-assessment-timeout-10min
Open

Bugfix/28537460 auto assessment timeout 10min#384
SathishMSFT wants to merge 7 commits into
masterfrom
bugfix/28537460-auto-assessment-timeout-10min

Conversation

@SathishMSFT

Copy link
Copy Markdown
Member

The MsftLinuxPatchAutoAssess.service uses Type=forking. systemd keeps the unit in its startup phase until it observes the expected forking lifecycle. If assessment startup and package-manager work exceed the default TimeoutStartSec=90s, systemd terminates the service before the extension writes terminal assessment status. As the result, the status remains in-progress.

Fix:
Increased the timeout from 90 seconds ( default) to 10 minutes. This 10 minutes is inline with p99 time taken for on-demand operations. This avoids changing the established lifecycle model while allowing slow assessments to finish. The timeout remains bounded so a process that never reaches the expected state (or takes longer time than 10 minutes) is still terminated.

Testing:
Verified this by adding (>90 seconds) timeouts in the respective apt, yum,zypper package manager commands. Adding command used for apt for reference:

DELAY_SECONDS=120
sudo test ! -e /usr/bin/apt-get.distrib || {
    echo "apt-get diversion already exists; inspect before continuing."
    exit 1
}
sudo dpkg-divert --local --rename --add /usr/bin/apt-get
sudo tee /usr/bin/apt-get >/dev/null <<EOF
#!/usr/bin/env bash
MARKER=/var/tmp/lpe-28537460-apt-delay-used
if [ ! -e "\$MARKER" ]; then
    touch "\$MARKER"
    logger -t lpe-28537460-repro \
        "Delaying first apt-get invocation for ${DELAY_SECONDS} seconds"
    sleep ${DELAY_SECONDS}
fi
exec /usr/bin/apt-get.distrib "\$@"
EOF
sudo chmod 755 /usr/bin/apt-get
sudo rm -f /var/tmp/lpe-28537460-apt-delay-used

SathishMSFT and others added 6 commits August 4, 2026 10:52
Keep Type=forking and extend TimeoutStartSec to 10 minutes for slow assessments.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Document the four-distro 600-second boundary test and retained evidence location.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add reusable APT, YUM, and Zypper delay and cleanup commands for the 90-second reproduction and 10-minute boundary validation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 91444677-8618-4dc0-8cf1-f415f7cb7185

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The change is small, targeted, and validated by an updated unit-file content test that confirms the new systemd timeout behavior.

Pull request overview

This PR addresses a systemd service startup timeout problem for the auto-assessment service by ensuring the generated unit file has a longer bounded TimeoutStartSec, preventing systemd from terminating slow-starting assessments before status is written.

Changes:

  • Extend the generated systemd unit’s start timeout to TimeoutStartSec=10min.
  • Update/strengthen the ServiceManager unit-file test to assert the unit content includes Type=forking and TimeoutStartSec=10min.
File summaries
File Description
src/core/src/core_logic/ServiceManager.py Adds TimeoutStartSec=10min (configurable via parameter) to the generated systemd service unit content.
src/core/tests/Test_ServiceManager.py Enhances the unit-file creation test to validate the written path and confirm the timeout directive is present.
Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@codecov

codecov Bot commented Sep 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.94%. Comparing base (cbd9ac9) to head (b172b44).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #384   +/-   ##
=======================================
  Coverage   94.93%   94.94%           
=======================================
  Files         111      111           
  Lines       20883    20890    +7     
=======================================
+ Hits        19826    19833    +7     
  Misses       1057     1057           
Flag Coverage Δ
python27 94.91% <100.00%> (+<0.01%) ⬆️
python312 94.91% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.


# region - Service Unit Management
def create_service_unit_file(self, exec_start, desc, after="network.target", service_type="forking", wanted_by="multi-user.target"):
def create_service_unit_file(self, exec_start, desc, after="network.target", service_type="forking", timeout_start_sec="10min", wanted_by="multi-user.target"):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that the potential collision surface between Auto Assessment and imperative patching calls has increased, could you please validate that this scenario is handled correctly? Specifically, it would be useful to initiate an InstallPatches call while an assessment is in progress and verify the behavior.

Although this is a relatively small change, I believe we should let it bake sufficiently in the canary region to observe and address any unforeseen issues before broader rollout.

As I understand it, with this approach, if an assessment completes quickly, say within 60 seconds, the process exits immediately and gracefully. The 10-minute timeout only comes into play for assessments that are delayed or take longer than expected to complete.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants