systemd TimeoutStopSec: Diagnose SIGKILL After 90 Seconds

Symptoms and Exact Error

A systemd TimeoutStopSec problem often leaves a service in deactivating (stop-sigterm) after a stop or restart request, followed by State 'stop-sigterm' timed out. Killing. or another SIGKILL log. If the delay repeatedly measures about 90 seconds, verify whether the effective timeout for that unit is also 90 seconds before changing it.

systemd generally sends the signal defined in KillSignal when terminating a service, with SIGTERM being the default. If the process does not terminate within the timeout, systemd cleans up the remaining processes with SIGKILL in a standard configuration where SendSIGKILL=yes is active. Therefore, the final SIGKILL is not the root cause but a result of the normal termination failing to complete within the allotted time.

While significantly increasing TimeoutStopSec can help services that actually require time for data flushing or task draining, it may also mask issues where the SIGTERM handler is hung or ExecStop fails to return. First, synchronize the unit’s stop commands with the application logs to identify which stage is consuming the time.

Observed stops with the same two-second limit

On 2026-09-06 KST, two disposable units were run on a separate Linux host using systemd 255 (255.4-1ubuntu8.17). They ran Python fixtures with a 20-second runtime limit, not production services. Both used TimeoutStopSec=2s, KillMode=control-group, and SendSIGKILL=yes. Only the signal handler differed; the units were cleaned up afterward.

SIGTERM ignored:  stop took 2.150 s, systemctl stop exit=0
  State 'stop-sigterm' timed out. Killing.
  Main process exited, code=killed, status=9/KILL
  Failed with result 'timeout'.
SIGTERM handled:  stop took 0.033 s, systemctl stop exit=0
  Deactivated successfully.

The significant detail is that systemctl stop returned 0 in both cases. A successful command alone would miss the forced termination. Check the journal for timeout, 9/KILL, and graceful completion. This experiment tests the signal-handling boundary only; it does not reproduce a database flush, worker drain, or a 90-second production incident.

Values to Check First

Checkpoints	Command or Property	Criteria
Applied Termination Limits	systemctl show app.service -p TimeoutStopUSec	Check if the observed 90 seconds matches the actual applied value of 1min 30s.
Termination Signal Policy	systemctl show app.service -p KillMode -p KillSignal -p SendSIGKILL	Verify the basic control-group, SIGTERM, and SIGKILL cleanup procedures.
Unit Source and Drop-in	systemctl cat app.service	Look for TimeoutStopSec, ExecStop, and KillMode applied after the vendor unit.
Stop Timeline	journalctl -u app.service --since '-10 min' -o short-precise	Compare the stop request, SIGTERM processing start, drain completion, timeout, and SIGKILL timestamps in sequence.
Remaining Processes	systemctl status app.service	Check the status tree to see if child processes remain after the main process or if the ExecStop process has stalled.

Replace app.service with the actual unit name. Even if 90 seconds is not specified in the configuration file, it may inherit the manager’s DefaultTimeoutStopSec. Conversely, if a unit or drop-in has overridden the value, do not assume the distribution default; instead, judge based on the systemctl show results.

Cause Branches

Before or after systemd TimeoutStopSec expiration

• SIGTERM handler is not executed: If the application only handles other signals or the PID 1 wrapper fails to propagate signals to children, the shutdown log itself will not start. For container services, also check the init·entrypoint layers.
• Handler started but did not finish: This occurs when network requests, queue drains, file flushes, or worker joins wait indefinitely. Check the final stage of the shutdown log along with the number of open tasks.
• ExecStop does not finish synchronously: The stop script may not wait for the actual shutdown, or conversely, it may continue to wait for an unresponsive management endpoint. Separate the ExecStop execution time from the main process termination time.
• Only child processes remain: The default KillMode=control-group cleans up the entire process group of the unit. If changed to handle only the main process like KillMode=process, children may remain, making it difficult to recommend as a general solution.
• Normal drain time exceeds the limit: If progress continues to change and slightly exceeds the limit, you can increase TimeoutStopSec to a reasonable upper bound. If the Type=notify service supports shutdown extensions, also consider a design that extends the timeout according to a defined protocol.

Fix Sequence

1. Save the current unit and status. Run systemctl cat app.service > app.service.before-stop-fix.txt and systemctl show app.service > app.service.before-stop-fix.properties, and save the recent journal to a separate file.
2. Measure the stop process once during the maintenance window. Check the journal from the time systemctl stop app.service is executed until the timeout. If the service receives operational requests, drain the traffic first before proceeding, and do not repeatedly perform forced kills.
3. First, fix the application shutdown path. Ensure that upon receiving SIGTERM, the application stops accepting new tasks, sets a limit on ongoing tasks, closes resources, and then terminates the main process. If a wrapper script exists, use exec to pass the actual process so that the signal reaches the correct PID.
4. Minimize the role of ExecStop. If the application can terminate with only SIGTERM, remove unnecessary long ExecStop entries. If a separate stop command is required, ensure the command waits synchronously for the actual service to stop and returns a clear exit code upon failure.
5. Apply timeouts only to normal drains. When measurements show that slightly more than 90 seconds are required and progress is confirmed, set a reasoned upper limit such as [Service] and TimeoutStopSec=3min in the drop-in file. Do not hide problems with infinite waits or TimeoutStopSec=infinity.
6. Verify and apply the settings. Inspect the unit with systemd-analyze verify app.service, then run systemctl daemon-reload. Perform start and stop again during the maintenance window and verify that SIGTERM termination completes within the new timeout.

Verification and Rollback

The success criteria are that the SIGKILL logs disappear and the service leaves a completion log within the designated time. Verify that no failure results remain after stopping via systemctl show app.service -p Result -p ExecMainStatus -p ActiveState -p SubState, and ensure that data recovery or reprocessing of incomplete tasks is unnecessary upon restart.

• Verify that the number of tasks in progress immediately before stopping matches the number of completed or retried tasks immediately after termination.
• Check that timeout and SIGKILL messages do not reappear in journalctl -u app.service -b.
• Verify that no child processes remain after the unit terminates in systemctl status app.service.
• Record the actual stop duration to ensure the new timeout does not excessively extend the failure recovery time.

If termination becomes slower or restart fails after the change, compare the created drop-in with the backup, roll back only this change, and execute systemctl daemon-reload. If no SIGTERM logs appear even after returning to the original configuration, do not continue adjusting the timeout value; instead, re-examine the execution wrapper and the application signal handler.

Related Articles

• Server and Linux Troubleshooting
• View All IT Troubleshooting

Official Documentation and Verification Criteria

• systemd.service manual — Scope of TimeoutStopSec application to ExecStop and service termination, and SIGKILL behavior after expiration
• systemd.service source manual — Implementation documentation on service termination limits applied to ExecStop and the main process, and how they can be extended
• systemd.kill source manual — Process group termination order based on KillMode and KillSignal

Earlier verification notes preserved from this article

Written by: Nole IT Editorial Team

Verification level: Checked against official documentation

The signal order and unit properties in this guide were checked against the current systemd service and kill documentation. Confirm effective values on the affected host before changing a timeout.

Editorial and revision policy · Report an error