Proxmox Diagnosis Script
This script has now become center stage in my projects at the moment for the following reasons.
- It tells me if the network is overloaded - its currently a 2.5Gbps test network
- Do I have a thrash problem on the server with my IO
The reason all this is happening is because my servers are starting to push their storage beyond capacity. So a NAS is on the budget - but do I go 10Gbps or do I just go pro-consumer at 2.5Gb - which is working rather well. Cost is a problem not form factor.
I can fit a Ubiquiti NAS 4 bay in my entertainment unit along with a Pro switch and the UDM SE all single 1U. If I go this way I can go 10Gbps internal till I hit the NTD, but is it the most cost effective solution. My server cant do 10Gbps, unless I add a NIC card.
2.5bps is handling everything I throw at it.
Hence the script. Am I getting every ounce of performance out of a dated server.
At the moment baden is running at an average of 90% as I overload the multimedia server I posted about previously. It is a perfect test case of pushing my hardware to the limit by using every screen in the house and pushing streaming media to them. So far I have not broke the server.
However some problems have presented.
Clear answer: Jellyfin, at 203% CPU (???) — it's the only thing not near-idle, and its 17.33 GB network-out + 206 GB block-read say it's transcoding (or several streams at once), not stuck. That's baden earning its keep, not a fault. Everything else totals ~8%.
The transcoding is not registering in Beszel either which is a concern. Ollama registers as did Immich when I was running it, but not Jellyfin. Logs say otherwise.
So: nothing to fix on the server right now. If the pegging bothers you later, the real lever is hardware transcoding — that RTX 3070 passthrough should be doing this work on the GPU instead of four CPU cores (Jellyfin → Dashboard → Transcoding → enable NVENC). Offloads almost all of it.
IDIOT - Can I hardcode that in the docker compose file.
That ONLY pushes the GPU to 2% and the CPU's still hit 90+%. That is on all protocols listed above on at least 4 screens.
Backup and take a breath.
The Script - now in development - all my scripts are in development.
Here is the latest version. Its going to change.
#!/usr/bin/env bash
#===============================================================================
# Script : diagnose-host-incident.sh
# Version : 3.3.0
# Created : 2026-07-05
# Updated : 2026-07-08
#
# Purpose : Forensic collector AND automated interpreter for a service outage on
# a Proxmox host or LXC, where services stopped responding / transfers
# timed out but the host recovered WITHOUT a reboot. It pulls the
# systemd/kernel JOURNAL for a window back from now, PLUS cumulative,
# boot-persistent evidence that survives a self-recovery, across two
# exhaustion domains (MEMORY/OOM and NETWORK), then ends with an
# AUTOMATED DIAGNOSIS that scores each domain and prints prioritised
# RECOMMENDATIONS so the report is self-interpreting.
#
# MEMORY / OOM (usual root cause of a whole-host brownout):
# - every kernel OOM-kill in the window, AUTO-RESOLVED to the Podman
# container or LXC it hit, with that cgroup's memory/swap ceilings;
# - cumulative per-cgroup oom_kill tallies from memory.events;
# - PSI memory/io stall pressure + swap-thrash counters;
# - which containers run UNBOUNDED (no mem_limit).
# NETWORK (rule it in or out): conntrack utilisation + drop counters,
# TCP listen/accept-queue overflows, ARP/neighbour, softnet drops,
# Podman network health, NIC.
#
# Host-agnostic: auto-detects Podman and Proxmox context. Run it on
# EACH host involved (backend first, then proxy, then the PVE host).
# NOTE: kernel OOM logs for an LXC appear on the PVE HOST journal, not
# inside the LXC - run on the PVE host to see cgroup OOM kills.
#
# Usage : sudo ./diagnose-host-incident.sh [LOOKBACK_MIN] [BRIDGE] [REPORT_DIR]
# LOOKBACK_MIN - minutes of journal history to pull (default: 90)
# BRIDGE - bridge to inspect for NIC section (default: vmbr0)
# REPORT_DIR - where to write the report (default: cwd)
# Precise window via env (overrides LOOKBACK_MIN):
# SINCE="2026-07-05 10:15:00" UNTIL="2026-07-05 11:30:00" \
# sudo ./diagnose-host-incident.sh
#
# Prereqs : Run as root. Core: ip, journalctl, awk, ss. Useful: podman, pct,
# conntrack-tools, ethtool, nstat. Degrades gracefully if absent.
# Requires an awk with strtonum() (gawk, or Debian mawk >= 1.3.4).
#
# Warnings: Strictly READ-ONLY. Changes no config, loads no modules, restarts
# nothing. Safe to re-run on a production host. Sections labelled
# (LIVE)/(now) are current state; incident evidence is the JOURNAL,
# OOM FORENSICS, and CUMULATIVE COUNTER sections. AUTOMATED DIAGNOSIS
# findings are heuristic - confirm against the raw sections above.
# Colour is applied ON SCREEN ONLY (set NO_COLOR=1 to disable); the
# saved report file stays plain ASCII. RECOMMENDATIONS are drawn from a
# SINGLE run - baseline them across several runs before persisting.
#
# Changelog:
# 3.3.0 (2026-07-08) - Added LXC LIVE MEMORY PRESSURE: on the PVE host, reads
# each guest's memory.current vs memory.max, anon working
# set, memory.swap.current and memory.events straight from
# /sys/fs/cgroup/lxc/<ctid>/ - the actual cgroup pressure
# that host-level 'free' hides. The verdict now flags any
# guest near its cap and recommends raising THAT guest's
# RAM (a capped guest cannot use host free RAM), correcting
# the earlier host-swappiness-first advice. Host-swap
# finding reworded to point at capped-guest swapping.
# 3.2.2 (2026-07-08) - Cosmetic: fixed two stale "ACTION ITEMS" references (the
# header note and the END OF REPORT footer) left over from
# the 3.2.0 rename to RECOMMENDATIONS.
# 3.2.1 (2026-07-08) - Fix: "CRITS: unbound variable" under set -u when the run
# produced WARN findings but no CRIT ones. Arrays are now
# initialised empty (local -a CRITS=() WARNS=()) and the
# RECOMMENDATIONS loops are guarded with ${#arr[@]} count
# checks instead of the unreliable "${arr[@]:-}" idiom.
# 3.2.0 (2026-07-08) - Terminal colour output (severity tags coloured, section
# dividers highlighted) applied ON SCREEN ONLY via a token
# colouriser - the tee'd report file stays plain ASCII per
# house style. NO_COLOR=1 disables it. Renamed ACTION ITEMS
# to RECOMMENDATIONS (rendered green), fine-tuned each to a
# concrete command, and added a caveat that they come from
# a single sample and want a larger baseline before being
# persisted.
# 3.1.0 (2026-07-08) - Added AUTOMATED DIAGNOSIS engine: re-reads the raw
# sources and emits [CRIT]/[WARN]/[INFO]/[ OK ] findings
# per domain (window incident, OOM cumulative + in-window,
# swap thrash + needless-swapping, memory PSI, I/O PSI,
# conntrack, TCP listen-queue, softnet, ARP) each with a
# specific remediation, followed by a prioritised ACTION
# ITEMS summary. Cumulative OOM downgraded to WARN when no
# kills fall in the window (historical, clears on reboot).
# Plain ASCII severity tags (no ANSI) to keep the tee'd
# report clean.
# 3.0.0 (2026-07-05) - Renamed from diagnose-net-incident.sh; added MEMORY/OOM
# domain (auto-resolved OOM kills, memory.events tallies,
# PSI, swap counters, unbounded-container audit, per-LXC
# ceilings). Reordered so memory evidence reads first.
# 2.0.0 (2026-07-05) - Reframed NIC tool into windowed service-outage forensics
# (journal window, conntrack, listen-queue, ARP, softnet,
# Podman net health). Stripped benign veth/loop noise.
# 1.0.0 (2026-07-05) - Initial release as diagnose-nic-pxe.sh (NIC snapshot).
#===============================================================================
set -u
LOOKBACK_MIN="${1:-90}"
BRIDGE="${2:-vmbr0}"
REPORT_DIR="${3:-$(pwd)}"
SINCE="${SINCE:-$(date -d "${LOOKBACK_MIN} minutes ago" '+%Y-%m-%d %H:%M:%S' 2>/dev/null)}"
UNTIL="${UNTIL:-$(date '+%Y-%m-%d %H:%M:%S')}"
STAMP="$(date +%Y%m%d-%H%M%S)"
HOSTN="$(hostname -s 2>/dev/null || echo unknown)"
REPORT="${REPORT_DIR%/}/host-incident-${HOSTN}-${STAMP}.txt"
SIGNAL='nf_conntrack|table full|out of memory|oom-killer|oom-kill:|Out of memory|Killed process|invoked oom|Memory cgroup out of memory|neighbour: |neighbor table overflow|ntable overflow|TCP: out of memory|too many orphan|SYN flooding|Ran out of|no buffer space|too many open files|EMFILE|Connection timed out|conntrack|netavark|aardvark|link is Down|Link is Down|reset adapter|NETDEV WATCHDOG|Hardware Error|segfault|general protection|traps:|hung_task|blocked for more than'
NOISE='renamed from veth|detected capacity change|entered (blocking|forwarding|disabled) state|entered promiscuous mode|entered allmulticast mode|port [0-9]+\(veth'
#--- helpers -------------------------------------------------------------------
section() {
printf '\n===============================================================================\n'
printf ' %s\n' "$1"
printf '===============================================================================\n'
}
have() { command -v "$1" >/dev/null 2>&1; }
note() { printf '[note] %s\n' "$1"; }
sub() { printf -- '--- %s ---\n' "$1"; }
# float greater-than: exit 0 (true) if a > b
fgt() { awk -v a="${1:-0}" -v b="${2:-0}" 'BEGIN{exit !(a>b)}'; }
resolve_iface() {
local br="$1" cand
[ -d "/sys/class/net/${br}" ] || return 1
cand=$(ls "/sys/class/net/${br}/brif" 2>/dev/null | grep -vE 'tap|veth|fwln|fwpr' | head -1)
printf '%s' "$cand"
}
#--- colour (SCREEN ONLY) ------------------------------------------------------
# The report file is written plain (tee happens BEFORE the colouriser), so ANSI
# never lands on disk. Colour is added to the terminal stream by matching the
# severity/recommendation tokens the sections already emit. NO_COLOR=1 disables.
if [ -n "${NO_COLOR:-}" ]; then USE_COLOR=0; else USE_COLOR=1; fi
C_CRIT=$'\033[1;31m' # red - act now
C_WARN=$'\033[0;33m' # yellow - worth fixing
C_OK=$'\033[0;32m' # green - healthy
C_INFO=$'\033[0;36m' # cyan - context
C_REC=$'\033[1;32m' # green - recommendations (as requested)
C_HEAD=$'\033[1;36m' # bold cyan - section dividers
C_NOTE=$'\033[0;90m' # grey - notes
C_OFF=$'\033[0m'
colourize() {
if [ "${USE_COLOR:-1}" -ne 1 ]; then cat; return; fi
sed -uE \
-e "s/\[CRIT\]/${C_CRIT}[CRIT]${C_OFF}/g" \
-e "s/\[WARN\]/${C_WARN}[WARN]${C_OFF}/g" \
-e "s/\[ OK \]/${C_OK}[ OK ]${C_OFF}/g" \
-e "s/\[INFO\]/${C_INFO}[INFO]${C_OFF}/g" \
-e "s/^(\[REC!?\].*)$/${C_REC}\1${C_OFF}/g" \
-e "s/^(===============.*)$/${C_HEAD}\1${C_OFF}/g" \
-e "s/^(\[note\].*)$/${C_NOTE}\1${C_OFF}/g"
}
#--- automated diagnosis engine (reads raw sources, scores, recommends) ---------
verdict() {
section "AUTOMATED DIAGNOSIS"
note "Tags: [CRIT] act now [WARN] worth fixing [INFO] context [ OK ] healthy"
note "Cumulative findings persist until the next reboot; (window)/(now) mark live state."
local -a CRITS=() WARNS=()
local pk=4 # page size KiB
# -- window: was there an acute event? ----------------------------------
local oom_win net_win
oom_win=$(journalctl -k --since "$SINCE" --until "$UNTIL" --no-pager 2>/dev/null | grep -c 'oom-kill:')
net_win=$(journalctl --since "$SINCE" --until "$UNTIL" --no-pager 2>/dev/null | grep -iE "$SIGNAL" | grep -vEi "$NOISE" | grep -c .)
oom_win=${oom_win:-0}; net_win=${net_win:-0}
if [ "$oom_win" -eq 0 ] && [ "$net_win" -eq 0 ]; then
printf '[INFO] window : no acute incident in window (%s .. %s); findings below are since boot.\n' "$SINCE" "$UNTIL"
else
printf '[WARN] window : %s OOM + %s network signal line(s) in window - see JOURNAL/OOM sections.\n' "$oom_win" "$net_win"
WARNS+=("Acute event inside the window: inspect JOURNAL + OOM FORENSICS above.")
fi
# -- OOM (cumulative, with in-window escalation) ------------------------
local oomk
oomk=$(awk '/^oom_kill /{print $2}' /proc/vmstat 2>/dev/null); oomk=${oomk:-0}
if [ "$oomk" -gt 0 ] 2>/dev/null; then
if [ "$oom_win" -gt 0 ]; then
printf '[CRIT] oom : %s OOM-kill(s) since boot, %s of them IN WINDOW - active problem.\n' "$oomk" "$oom_win"
CRITS+=("Cap the OOM'd container: add mem_limit + memswap_limit to its compose and redeploy. Its name is in OOM FORENSICS (run on the backend host).")
else
printf '[WARN] oom : %s OOM-kill(s) since boot but none in window - historical (clears on reboot). Verify the fix held.\n' "$oomk"
WARNS+=("Confirm the earlier cap held: podman inspect --format '{{.HostConfig.Memory}}' <name> (0 = still unbounded); watch its memory.events oom_kill.")
fi
while IFS= read -r f; do
local k; k=$(awk '/^oom_kill /{print $2}' "$f" 2>/dev/null)
[ -n "$k" ] && [ "$k" -gt 0 ] 2>/dev/null && printf ' -> %s (oom_kill=%s)\n' "${f%/memory.events}" "$k"
done < <(find /sys/fs/cgroup -name memory.events 2>/dev/null)
else
printf '[ OK ] oom : no OOM-kills since boot.\n'
fi
# -- swap: needless swapping + churn ------------------------------------
local pout swp_used ma mt swpi pout_gb swp_gb ma_gb
pout=$(awk '/^pswpout /{print $2}' /proc/vmstat 2>/dev/null); pout=${pout:-0}
swp_used=$(awk '/^SwapTotal:/{t=$2}/^SwapFree:/{f=$2}END{print (t-f)+0}' /proc/meminfo 2>/dev/null)
ma=$(awk '/^MemAvailable:/{print $2}' /proc/meminfo 2>/dev/null); ma=${ma:-0}
mt=$(awk '/^MemTotal:/{print $2}' /proc/meminfo 2>/dev/null); mt=${mt:-1}
swpi=$(sysctl -n vm.swappiness 2>/dev/null)
pout_gb=$(awk -v p="$pout" -v k="$pk" 'BEGIN{printf "%.1f", p*k/1048576}')
swp_gb=$(awk -v s="${swp_used:-0}" 'BEGIN{printf "%.1f", s/1048576}')
ma_gb=$(awk -v s="$ma" 'BEGIN{printf "%.1f", s/1048576}')
local swap_flag=0
if [ "${swp_used:-0}" -gt 524288 ] && awk -v a="$ma" -v t="$mt" 'BEGIN{exit !(a>t*0.25)}'; then
printf '[WARN] swap : %s GiB in swap though the HOST has %s GiB RAM free - on a PVE host this usually means a CAPPED guest swapping against its own limit (see lxc mem), not host waste.\n' "$swp_gb" "$ma_gb"
WARNS+=("Do NOT just lower host swappiness - a capped guest cannot use host free RAM. Raise the maxed guest's memory (see lxc mem); treat vm.swappiness=10 as a secondary reducer only.")
swap_flag=1
fi
if fgt "$pout_gb" 8; then
printf '[WARN] swapchurn: ~%s GiB paged OUT since boot - heavy thrash; feeds I/O stall and worsens OOM brownouts.\n' "$pout_gb"
[ "$swap_flag" -eq 0 ] && WARNS+=("Cut swap churn: lower vm.swappiness and/or raise the busiest guest's RAM (pct set <ctid> -memory <MB>).")
swap_flag=1
fi
[ "$swap_flag" -eq 0 ] && printf '[ OK ] swap : swap use and churn within normal range.\n'
# -- memory PSI ---------------------------------------------------------
local mfull
mfull=$(awk '/^full/{for(i=1;i<=NF;i++) if($i ~ /^total=/){sub("total=","",$i); print $i/1000000}}' /proc/pressure/memory 2>/dev/null)
if fgt "${mfull:-0}" 120; then
printf '[WARN] mem psi : ~%.0f s cumulative FULL memory stall since boot (brownout fingerprint).\n' "$mfull"
WARNS+=("Memory-stall history present - correlate its timing (journalctl -k | grep oom-kill) with the OOM; keep the offender capped.")
else
printf '[ OK ] mem psi : low cumulative memory stall (~%.0f s).\n' "${mfull:-0}"
fi
# -- I/O PSI ------------------------------------------------------------
local iofull io_a3
iofull=$(awk '/^full/{for(i=1;i<=NF;i++) if($i ~ /^total=/){sub("total=","",$i); print $i/3600000000}}' /proc/pressure/io 2>/dev/null)
io_a3=$(awk '/^some/{for(i=1;i<=NF;i++) if($i ~ /^avg300=/){sub("avg300=","",$i); print $i}}' /proc/pressure/io 2>/dev/null)
if fgt "${iofull:-0}" 1; then
printf '[WARN] io psi : ~%.1f h cumulative FULL I/O stall since boot (some avg300=%s%% now) - real disk contention.\n' "$iofull" "${io_a3:-0}"
WARNS+=("Trace disk contention under load: iostat -x 5 (needs sysstat); usual writers are qbittorrent/radarr/snapotter. Move swap off any HDD.")
else
printf '[ OK ] io psi : low cumulative I/O stall (~%.1f h).\n' "${iofull:-0}"
fi
# -- conntrack ----------------------------------------------------------
local cc cm cp
cc=$(cat /proc/sys/net/netfilter/nf_conntrack_count 2>/dev/null)
cm=$(cat /proc/sys/net/netfilter/nf_conntrack_max 2>/dev/null)
if [ -n "${cc:-}" ] && [ "${cm:-0}" -gt 0 ] 2>/dev/null; then
cp=$(( cc * 100 / cm ))
if [ "$cp" -ge 80 ]; then
printf '[CRIT] conntrack: %s%% of table used (now) - near overflow.\n' "$cp"
CRITS+=("Raise net.netfilter.nf_conntrack_max (e.g. 2x) via /etc/sysctl.d/; find the storm via 'conntrack -L' grouped by dst/src.")
elif [ "$cp" -ge 50 ]; then
printf '[WARN] conntrack: %s%% of table used (now).\n' "$cp"
WARNS+=("Watch nf_conntrack_max headroom.")
else
printf '[ OK ] conntrack: %s%% table used; no drops.\n' "$cp"
fi
fi
# -- TCP listen/accept-queue (cumulative) -------------------------------
if have nstat; then
local lov
lov=$(nstat -az 2>/dev/null | awk '/ListenOverflows|ListenDrops|TCPBacklogDrop/{s+=$2}END{print s+0}')
if [ "${lov:-0}" -gt 0 ]; then
printf '[WARN] tcp queue: %s listen/accept-queue drops since boot - connections were refused.\n' "$lov"
WARNS+=("Raise net.core.somaxconn / application backlog if this recurs under load.")
else
printf '[ OK ] tcp queue: no listen/accept-queue drops.\n'
fi
fi
# -- softnet backlog ----------------------------------------------------
local snd
snd=$(awk '{s+=strtonum("0x"$2)}END{print s+0}' /proc/net/softnet_stat 2>/dev/null)
if [ "${snd:-0}" -gt 0 ]; then
printf '[WARN] softnet : %s input-backlog drops since boot.\n' "$snd"
WARNS+=("Raise net.core.netdev_max_backlog (e.g. 8192).")
else
printf '[ OK ] softnet : no input-backlog drops.\n'
fi
# -- ARP / neighbour ----------------------------------------------------
local nn g3
nn=$(ip -4 neigh show 2>/dev/null | wc -l)
g3=$(sysctl -n net.ipv4.neigh.default.gc_thresh3 2>/dev/null)
if [ -n "${g3:-}" ] && awk -v n="${nn:-0}" -v g="${g3:-1}" 'BEGIN{exit !(n>g*0.8)}'; then
printf '[WARN] arp : %s neighbour entries near gc_thresh3=%s.\n' "$nn" "$g3"
WARNS+=("Raise net.ipv4.neigh.default.gc_thresh1/2/3.")
else
printf '[ OK ] arp : %s neighbour entries (well under gc_thresh3=%s).\n' "${nn:-0}" "${g3:-?}"
fi
# -- prioritised recommendations (green; single-sample caveat) ----------
section "RECOMMENDATIONS"
note "Drawn from a SINGLE run - treat these as leads, not settled fixes."
note "Baseline them first: repeat across idle AND peak load, or cross-check Beszel"
note "history, before persisting any sysctl / memory change. [REC!] = urgent."
if [ "${#CRITS[@]}" -eq 0 ] && [ "${#WARNS[@]}" -eq 0 ]; then
printf '[REC] Nothing to change - all monitored domains healthy in this sample.\n'
else
local i=1 x
if [ "${#CRITS[@]}" -gt 0 ]; then
for x in "${CRITS[@]}"; do printf '[REC!] %d. %s\n' "$i" "$x"; i=$((i+1)); done
fi
if [ "${#WARNS[@]}" -gt 0 ]; then
for x in "${WARNS[@]}"; do printf '[REC] %d. %s\n' "$i" "$x"; i=$((i+1)); done
fi
fi
}
#--- main body (tee'd to report) -----------------------------------------------
main() {
HAS_PODMAN=0; have podman && HAS_PODMAN=1
IS_PVE=0; have pveversion && IS_PVE=1
section "HOST INCIDENT FORENSIC REPORT"
printf 'Host : %s\n' "$(hostname -f 2>/dev/null || hostname)"
printf 'Generated : %s\n' "$(date -Is)"
printf 'Kernel : %s\n' "$(uname -r)"
printf 'Uptime : %s\n' "$(uptime -p 2>/dev/null || uptime)"
printf 'Boot time : %s\n' "$(uptime -s 2>/dev/null || echo '?')"
printf 'Window SINCE: %s\n' "$SINCE"
printf 'Window UNTIL: %s\n' "$UNTIL"
printf 'Podman here : %s | Proxmox here: %s\n' \
"$([ $HAS_PODMAN -eq 1 ] && echo yes || echo no)" \
"$([ $IS_PVE -eq 1 ] && echo yes || echo no)"
printf 'Report : %s\n' "$REPORT"
note "Kernel OOM kills for an LXC log on the PVE HOST - run on the PVE host to see them."
note "Read the AUTOMATED DIAGNOSIS + RECOMMENDATIONS at the end for the verdict."
#========================= JOURNAL / OUTAGE EVIDENCE =======================
section "JOURNAL PERSISTENCE"
if have journalctl; then
[ -d /var/log/journal ] && note "Persistent journal present (/var/log/journal)." \
|| note "Journal may be volatile (/run) - still intact this boot (no reboot)."
journalctl --disk-usage 2>/dev/null
else
note "journalctl not available - not a systemd host?"
fi
section "INCIDENT KERNEL/JOURNAL LINES IN WINDOW (noise stripped)"
note "conntrack table full, OOM, SYN flooding, neighbour overflow, resets."
if have journalctl; then
out=$(journalctl --since "$SINCE" --until "$UNTIL" --no-pager -o short-iso 2>/dev/null \
| grep -iE "$SIGNAL" | grep -vEi "$NOISE")
if [ -n "$out" ]; then printf '%s\n' "$out"; else note "No smoking-gun lines in window. Widen LOOKBACK_MIN or set SINCE/UNTIL."; fi
fi
section "SYSTEMD ERRORS + FAILED UNITS IN WINDOW"
if have journalctl; then
sub "priority err+ in window (container noise excluded)"
journalctl --since "$SINCE" --until "$UNTIL" -p err --no-pager -o short-iso 2>/dev/null \
| grep -vEi "$NOISE" | tail -40 || note "none"
fi
if have systemctl; then
sub "currently failed units"
systemctl --failed --no-legend --no-pager 2>/dev/null || note "none"
fi
#========================= MEMORY / OOM (PRIME SUSPECT) ====================
section "OOM FORENSICS IN WINDOW (auto-resolved to container / LXC)"
note "Each kernel OOM-kill mapped to the container/LXC it hit, plus that cgroup's ceilings."
if have journalctl; then
oom_lines=$(journalctl -k --since "$SINCE" --until "$UNTIL" --no-pager -o short-iso 2>/dev/null \
| grep -iE 'oom-kill:|Killed process|Memory cgroup out of memory')
if [ -z "$oom_lines" ]; then
note "No kernel OOM-kill lines in window here."
note "If services died and this is an LXC, run on the PVE host - that is where the kill logs."
else
printf '%s\n' "$oom_lines"; echo
sub "resolved targets"
printf '%s\n' "$oom_lines" | grep 'oom-kill:' | while IFS= read -r l; do
lxcid=$(printf '%s' "$l" | sed -nE 's/.*oom_memcg=\/lxc\/([0-9]+).*/\1/p')
cid=$(printf '%s' "$l" | sed -nE 's/.*libpod-([0-9a-f]{12,}).scope.*/\1/p')
task=$(printf '%s' "$l" | sed -nE 's/.*task=([^,]+),.*/\1/p')
printf ' task=%s' "${task:-?}"
[ -n "$lxcid" ] && printf ' | lxc=%s' "$lxcid"
[ -n "$cid" ] && printf ' | container-id=%s' "${cid:0:12}"
printf '\n'
if [ -n "$cid" ]; then
if have podman; then
nm=$(podman ps -a --filter "id=${cid:0:12}" --format '{{.Names}} ({{.Image}})' 2>/dev/null)
[ -n "$nm" ] && printf ' -> podman container: %s\n' "$nm" \
|| printf ' -> container id %s not found by podman here.\n' "${cid:0:12}"
else
printf ' -> resolve on the Podman host: podman ps -a --filter id=%s\n' "${cid:0:12}"
fi
fi
if [ -n "$lxcid" ] && have pct; then
printf ' -> LXC %s: %s\n' "$lxcid" \
"$(pct config "$lxcid" 2>/dev/null | grep -E '^(hostname|memory|swap):' | tr '\n' ' ')"
fi
done
fi
fi
section "CGROUP OOM-KILL TALLIES (CUMULATIVE - survives recovery)"
note "Every cgroup that has OOM-killed at least once since boot. Non-empty = something exceeds a limit."
if [ -d /sys/fs/cgroup ]; then
found=0
while IFS= read -r f; do
k=$(awk '/^oom_kill /{print $2}' "$f" 2>/dev/null)
if [ -n "$k" ] && [ "$k" -gt 0 ] 2>/dev/null; then
printf ' oom_kill=%-4s %s\n' "$k" "${f%/memory.events}"; found=1
fi
done < <(find /sys/fs/cgroup -name memory.events 2>/dev/null)
[ "$found" -eq 0 ] && note "No cgroup shows a non-zero oom_kill count since boot."
else
note "cgroup v2 filesystem not found."
fi
section "MEMORY PRESSURE (PSI) + SWAP THRASH (cumulative)"
note "PSI 'total' = microseconds stalled since boot (survives recovery). High = the brownout."
for f in /proc/pressure/memory /proc/pressure/io; do
[ -r "$f" ] && { sub "$f"; cat "$f"; }
done
sub "swap + oom activity since boot (/proc/vmstat)"
awk '/^(oom_kill|pswpin|pswpout|pgmajfault) /{printf " %-12s %s\n",$1,$2}' /proc/vmstat 2>/dev/null
sub "swap state + swappiness now"
printf ' vm.swappiness = %s\n' "$(sysctl -n vm.swappiness 2>/dev/null)"
free -h 2>/dev/null | awk 'NR==1 || /Mem:|Swap:/'
section "MEMORY LIMITS - BOUNDED vs UNBOUNDED"
note "An UNLIMITED container can OOM the whole host cgroup. Give heavy/AI services a mem_limit."
if [ $HAS_PODMAN -eq 1 ]; then
sub "podman containers: memory limit (UNLIMITED = risk)"
podman ps --format '{{.ID}} {{.Names}}' 2>/dev/null | while read -r id nm; do
m=$(podman inspect --format '{{.HostConfig.Memory}}' "$id" 2>/dev/null)
if [ "${m:-0}" = "0" ]; then
printf ' %-24s UNLIMITED <-- unbounded\n' "$nm"
else
printf ' %-24s limit=%.1f GiB\n' "$nm" "$(awk "BEGIN{print ${m}/1073741824}")"
fi
done
sub "live usage snapshot (top by memory)"
podman stats --no-stream --format '{{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}' 2>/dev/null \
| sort -t$'\t' -k3 -rh | head -15
fi
if have pct; then
sub "LXC guests: configured memory / swap ceilings"
pct list 2>/dev/null | awk 'NR>1{print $1}' | while read -r ct; do
printf ' CT %-5s %s\n' "$ct" \
"$(pct config "$ct" 2>/dev/null | grep -E '^(hostname|memory|swap):' | tr '\n' ' ')"
done
fi
if [ -d /sys/fs/cgroup/lxc ]; then
section "LXC LIVE MEMORY PRESSURE (cgroup usage vs cap)"
note "Real usage vs cap per guest. A CAPPED guest CANNOT use host free RAM, so it"
note "swaps against its OWN limit even when 'free' on the host looks relaxed."
note "anon = true working set; swap.current > 0 = it was forced to swap."
for d in /sys/fs/cgroup/lxc/*/; do
[ -r "${d}memory.current" ] || continue
id=$(basename "$d")
cur=$(cat "${d}memory.current" 2>/dev/null)
max=$(cat "${d}memory.max" 2>/dev/null)
scur=$(cat "${d}memory.swap.current" 2>/dev/null)
anon=$(awk '/^anon /{print $2}' "${d}memory.stat" 2>/dev/null)
evt=$(awk '/^(high|max|oom|oom_kill) /{printf "%s=%s ",$1,$2}' "${d}memory.events" 2>/dev/null)
host=$(have pct && pct config "$id" 2>/dev/null | awk -F': ' '/^hostname/{print $2; exit}')
printf -- '--- CT %s (%s) ---\n' "$id" "${host:-?}"
if printf '%s' "${max:-}" | grep -qE '^[0-9]+$'; then
p=$(awk -v c="${cur:-0}" -v m="$max" 'BEGIN{printf "%d",(m>0)?c*100/m:0}')
printf ' memory.current = %.1f / %.1f GiB (%s%% of cap)\n' \
"$(awk -v x="${cur:-0}" 'BEGIN{print x/1073741824}')" \
"$(awk -v x="$max" 'BEGIN{print x/1073741824}')" "$p"
else
printf ' memory.current = %.1f GiB (cap: unlimited)\n' "$(awk -v x="${cur:-0}" 'BEGIN{print x/1073741824}')"
fi
[ -n "${anon:-}" ] && printf ' anon (workset) = %.1f GiB\n' "$(awk -v x="$anon" 'BEGIN{print x/1073741824}')"
printf ' swap.current = %.1f GiB\n' "$(awk -v x="${scur:-0}" 'BEGIN{print x/1073741824}')"
[ -n "${evt:-}" ] && printf ' events = %s\n' "$evt"
done
fi
#========================= CONNTRACK ======================================
section "CONNECTION TRACKING - UTILISATION"
if [ -r /proc/sys/net/netfilter/nf_conntrack_count ]; then
cnt=$(cat /proc/sys/net/netfilter/nf_conntrack_count 2>/dev/null)
max=$(cat /proc/sys/net/netfilter/nf_conntrack_max 2>/dev/null)
printf 'nf_conntrack_count = %s\n' "$cnt"
printf 'nf_conntrack_max = %s\n' "$max"
if [ -n "${cnt:-}" ] && [ -n "${max:-}" ] && [ "$max" -gt 0 ] 2>/dev/null; then
pctu=$(( cnt * 100 / max ))
printf 'utilisation (LIVE) = %s%%\n' "$pctu"
[ "$pctu" -ge 80 ] && note "HIGH now. If it hit 100%% during the outage, this is a cause."
fi
else
note "nf_conntrack not loaded here (host may not NAT/filter)."
fi
section "CONNTRACK DROP COUNTERS (CUMULATIVE - survives recovery)"
note "insert_failed / drop / early_drop > 0 = the table overflowed at some point since boot."
if have conntrack; then
conntrack -S 2>/dev/null | tr ' ' '\n' | grep -E 'insert_failed|drop|early_drop|invalid|error' | grep -vE '=0$' \
|| note "conntrack -S reports no non-zero drop/insert_failed counters."
elif [ -r /proc/net/stat/nf_conntrack ]; then
awk 'NR==1{for(i=1;i<=NF;i++)h[i]=$i; next}
{for(i=1;i<=NF;i++)s[i]+=strtonum("0x"$i)}
END{for(i=1;i<=NF;i++) if(h[i] ~ /insert_failed|drop|invalid|error/) printf " %-16s = %d\n", h[i], s[i]}' \
/proc/net/stat/nf_conntrack
else
note "No conntrack CLI and no /proc/net/stat/nf_conntrack."
fi
#========================= SOCKET / LISTEN-QUEUE STRESS ====================
section "TCP LISTEN/ACCEPT-QUEUE STRESS (CUMULATIVE - survives recovery)"
note "ListenOverflows/ListenDrops/TCPBacklogDrop > 0 = new connections dropped (proxy timeouts)."
if have nstat; then
nstat -az 2>/dev/null | grep -iE 'ListenOverflows|ListenDrops|TCPBacklogDrop|SyncookiesSent|TCPReqQFull' | grep -vE ' 0 ' \
|| note "No non-zero listen-queue stress counters."
sub "retransmits / pruning / orphans"
nstat -az 2>/dev/null | grep -iE 'RetransSegs|TCPLostRetransmit|PruneCalled|RcvPruned|OfoPruned|TCPAbortOnMemory' | grep -vE ' 0 '
elif have netstat; then
netstat -s 2>/dev/null | grep -iE 'listen|overflow|SYNs to LISTEN|pruned|retransmit|out of memory' || note "nothing relevant."
fi
section "CURRENT LISTEN QUEUES + BACKLOG LIMITS (LIVE)"
printf 'net.core.somaxconn = %s\n' "$(sysctl -n net.core.somaxconn 2>/dev/null)"
printf 'net.ipv4.tcp_max_syn_backlog = %s\n' "$(sysctl -n net.ipv4.tcp_max_syn_backlog 2>/dev/null)"
if have ss; then
sub "listening sockets (Recv-Q=backlog used, Send-Q=backlog max)"
ss -ltnH 2>/dev/null | awk '{printf " %-6s used=%-6s max=%-6s %s\n",$1,$2,$3,$4}' | head -25
sub "socket summary"; ss -s 2>/dev/null
fi
#========================= ARP / SOFTNET / PODMAN NET =====================
section "ARP / NEIGHBOUR TABLE (overflow -> host-unreachable timeouts)"
for k in gc_thresh1 gc_thresh2 gc_thresh3; do
printf 'ipv4.neigh.default.%-10s = %s\n' "$k" "$(sysctl -n net.ipv4.neigh.default.$k 2>/dev/null)"
done
if have ip; then
printf 'current neighbour entries : ipv4=%s ipv6=%s\n' \
"$(ip -4 neigh show 2>/dev/null | wc -l)" "$(ip -6 neigh show 2>/dev/null | wc -l)"
fi
section "PER-CPU SOFTNET DROPS (CUMULATIVE - col2; survives recovery)"
note "Non-zero 'dropped' = input backlog overflowed -> raise netdev_max_backlog."
[ -r /proc/net/softnet_stat ] && awk '{printf "cpu%-2d processed=%-12d dropped=%-10d squeezed=%d\n", NR-1, strtonum("0x"$1), strtonum("0x"$2), strtonum("0x"$3)}' /proc/net/softnet_stat
if [ $HAS_PODMAN -eq 1 ]; then
section "PODMAN NETWORK + CONTAINER HEALTH"
sub "networks"; podman network ls 2>/dev/null
sub "containers: restart counts + status (highest restarts first)"
podman ps -a --format '{{.Names}}\t{{.RestartCount}}\t{{.Status}}' 2>/dev/null | sort -t$'\t' -k2 -rn | head -30
sub "container events in window (die/oom/restart)"
podman events --since "$SINCE" --until "$UNTIL" --stream=false 2>/dev/null \
| grep -iE 'die|oom|stop|start|restart|health_status' | tail -40 || note "no events in window."
fi
#========================= NIC HARDWARE (demoted) ========================
section "NIC HARDWARE (secondary)"
IFACE="$(resolve_iface "$BRIDGE")"
if [ -z "${IFACE:-}" ]; then
note "No physical NIC behind '${BRIDGE}' (expected on an LXC using a veth)."
else
printf 'Physical NIC behind %s: %s\n' "$BRIDGE" "$IFACE"
if have ethtool; then
ethtool -i "$IFACE" 2>/dev/null | grep -E 'driver|firmware'
ethtool "$IFACE" 2>/dev/null | grep -E 'Speed|Duplex|Link detected'
sub "non-zero NIC error counters"
ethtool -S "$IFACE" 2>/dev/null | grep -iE 'err|drop|miss|fifo|over|timeout|reset|nobuf' | grep -vE ':\s*0$' \
|| note "No non-zero NIC error counters."
fi
ip -s link show "$IFACE" 2>/dev/null
fi
#========================= SYSCTL SNAPSHOT ===============================
section "RELEVANT SYSCTLS (current)"
for k in vm.swappiness vm.overcommit_memory \
net.netfilter.nf_conntrack_max net.core.somaxconn \
net.ipv4.tcp_max_syn_backlog net.core.netdev_max_backlog \
net.core.rmem_max net.core.wmem_max \
net.bridge.bridge-nf-call-iptables fs.file-nr; do
printf '%-40s = %s\n' "$k" "$(sysctl -n "$k" 2>/dev/null || echo 'n/a')"
done
#========================= AUTOMATED DIAGNOSIS ===========================
verdict
section "END OF REPORT"
printf 'Saved to: %s\n' "$REPORT"
printf 'Verdict + fixes are in AUTOMATED DIAGNOSIS and RECOMMENDATIONS above.\n'
}
#--- run: display live + persist -----------------------------------------------
mkdir -p "$REPORT_DIR" 2>/dev/null || { echo "Cannot create report dir: $REPORT_DIR" >&2; exit 1; }
# tee writes the PLAIN report to disk; colourize adds ANSI to the terminal only.
main 2>&1 | tee "$REPORT" | colourizeMy issue was a run away AI that quickly overwhelmed the LXC application server baden.
Happy hunting
#enoughsaid