Welcome to Podman

Yep - I have given Docker the flick.

Just too many problems with the latest updates and too many compromises.
I have edited my vent post a little but leaving it in situ.

🚀 Working Theory

• All servers are LXC Debian 13 privileged instances running Podman to conserve resources. Hypervisor is Proxmox 9 with no compromises.
• Portainer CE is deployed on the master server
• Portainer Agents run on secondary servers
• Both Portainer CE and agents are configured to restart automatically on server reboots.
• Existing code and stacks will restart seamlessly as long as Portainer restarts. Wrong - Podman quadlets are required for this - cronjob ??
• All existing Docker Compose stacks remain compatible with Podman
  Wrong - networking changes for a start

✅ Issues Resolved by Moving to Podman

• AppArmor conflicts eliminated
• Docker v29 privileged port issues resolved - ditched 😄
• No need to lower container security levels just to run servers
• Foundation laid for further research and development

No further issues yet discovered. Monitoring servers.

I have attached my code as a guide - updated 20-11-2025

#!/bin/bash
# ============================================================
# Proxmox LXC Podman + Conditional Portainer CE Setup Script
# ============================================================
# Purpose:
#   - Install base packages and Podman
#   - Configure firewall + logging
#   - Configure Podman registries
#   - Deploy Portainer CE (primary host) or Agent (secondary hosts)
#   - Ensure services auto-start via quadlets
#   - Base network setup for Podman containers
#   - Idempotent and robust error handling
#
# Created: 14-11-2025
# Updated: 20-11-2025
# ============================================================

set -euo pipefail

# -------------------------------
# Utility Functions
# -------------------------------
log() { echo "[INFO] $*"; }
success() { echo "[SUCCESS] $*"; }
error() { echo "[ERROR] $*"; }

configure_firewall_ports() {
    local ports=("$@")
    log "Opening firewall ports: ${ports[*]}"
    for port in "${ports[@]}"; do
        iptables -I INPUT -p tcp --dport "${port}" -j ACCEPT
        ip6tables -I INPUT -p tcp --dport "${port}" -j ACCEPT
    done
    netfilter-persistent save
    success "Firewall rules for ports ${ports[*]} persisted."
}

# -------------------------------
# 1. Base Packages
# -------------------------------
log "Installing base packages..."
apt-get update -qq || warn "apt-get update failed, check DNS/network"
apt-get install -y -qq \
    bind9-dnsutils iptables-persistent netfilter-persistent rsyslog sudo curl gpg net-tools \
    apt-transport-https cron mtr git openssh-server openssh-client \
    podman podman-docker podman-compose

log "Removing conflicting packages..."
apt-get purge --auto-remove -y -qq ufw inetutils-telnet || true

# -------------------------------
# 2. Miscellaneous Configurations
# -------------------------------
log "Configuring automatic updates via cron..."
sed -i '/apt-get .*upgrade/d' /etc/crontab
cat <<'EOF' >> /etc/crontab
0 23 */2 * * root apt-get update -qq && apt-get -y -qq upgrade && apt-get -y -qq autoremove && apt-get -y -qq autoclean
EOF
systemctl restart cron || { error "Cron restart failed"; exit 1; }

log "Set the timezone manually to ensure consistency..."
timedatectl set-timezone Australia/Perth

# Ensure timezone is set in /etc/containers/containers.conf
CONF_FILE="/etc/containers/containers.conf"

# Create file if missing
if [ ! -f "$CONF_FILE" ]; then
    echo "[engine]" > "$CONF_FILE"
    echo 'tz = "Australia/Perth"' >> "$CONF_FILE"
else
    # Ensure [engine] section exists
    if ! grep -q "^

\[engine\]

" "$CONF_FILE"; then
        echo "[engine]" >> "$CONF_FILE"
    fi
    # Update or append tz line idempotently
    if grep -q "^tz" "$CONF_FILE"; then
        sed -i 's|^tz.*|tz = "Australia/Perth"|' "$CONF_FILE"
    else
        sed -i '/^

\[engine\]

/a tz = "Australia/Perth"' "$CONF_FILE"
    fi
fi

# -------------------------------
# 3. Firewall Setup
# -------------------------------
log "Configuring firewall rules..."
iptables -F; ip6tables -F
iptables -P INPUT DROP; iptables -P FORWARD ACCEPT; iptables -P OUTPUT ACCEPT
ip6tables -P INPUT DROP; ip6tables -P FORWARD ACCEPT; ip6tables -P OUTPUT ACCEPT


# IPv4 INPUT rules
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A INPUT -p tcp --dport 22 -s 192.168.0.0/16 -j ACCEPT
iptables -A INPUT -p udp --dport 53 -j ACCEPT
iptables -A INPUT -p tcp --dport 53 -j ACCEPT
iptables -A INPUT -p udp --sport 67:68 --dport 67:68 -j ACCEPT
iptables -A INPUT -p udp --dport 5353 -j ACCEPT
iptables -A INPUT -p icmp -j ACCEPT
iptables -A INPUT -d 255.255.255.255 -j ACCEPT
iptables -A INPUT -m conntrack --ctstate INVALID -m limit --limit 5/min -j LOG --log-prefix "Invalid v4: "
iptables -A INPUT -m conntrack --ctstate INVALID -j DROP

# Reverse proxy / web - only allow from local network - adjust as needed
iptables -A INPUT -p tcp --dport 80  -s 192.168.0.0/16 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -s 192.168.0.0/16 -j ACCEPT
iptables -A INPUT -p udp --dport 443 -s 192.168.0.0/16 -j ACCEPT

# IPv6 INPUT rules
ip6tables -A INPUT -i lo -j ACCEPT
ip6tables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
ip6tables -A INPUT -p tcp --dport 22 -s ::/0 -j ACCEPT
ip6tables -A INPUT -p udp --dport 53 -j ACCEPT
ip6tables -A INPUT -p tcp --dport 53 -j ACCEPT
ip6tables -A INPUT -p udp --sport 546:547 --dport 546:547 -j ACCEPT
ip6tables -A INPUT -p udp --dport 5353 -j ACCEPT
ip6tables -A INPUT -p ipv6-icmp -j ACCEPT
ip6tables -A INPUT -m conntrack --ctstate INVALID -m limit --limit 5/min -j LOG --log-prefix "Invalid v6: "
ip6tables -A INPUT -m conntrack --ctstate INVALID -j DROP

# Reverse proxy / web - allow from anywhere (adjust as needed)
ip6tables -A INPUT -p tcp --dport 80  -s ::/0 -j ACCEPT
ip6tables -A INPUT -p tcp --dport 443 -s ::/0 -j ACCEPT
ip6tables -A INPUT -p udp --dport 443 -s ::/0 -j ACCEPT

netfilter-persistent save
systemctl enable --now netfilter-persistent

# -------------------------------
# 4. Logging Setup
# -------------------------------
log "Configuring iptables logging..."
cat <<'EOF' > /etc/rsyslog.d/iptables.conf
:msg, contains, "Invalid" -/var/log/iptables-dropped.log
& stop
EOF
systemctl restart rsyslog

cat <<'EOF' > /etc/logrotate.d/iptables
/var/log/iptables-dropped.log {
    daily
    missingok
    rotate 7
    compress
    delaycompress
    notifempty
    create 0640 root adm
    postrotate
        systemctl reload rsyslog >/dev/null 2>&1 || true
    endscript
}
EOF

# -------------------------------
# 5. Podman Registry Config & Network Setup
# -------------------------------
log "Configuring Podman registries..."
mkdir -p /etc/containers/registries.conf.d
cat <<'EOF' > /etc/containers/registries.conf.d/99-unqualified.conf
unqualified-search-registries = ["docker.io", "quay.io", "ghcr.io", "registry.fedoraproject.org"]

[aliases]
# Map unqualified image names to fully qualified ones
"portainer/agent"      = "docker.io/portainer/agent"
"portainer/portainer-ce" = "docker.io/portainer/portainer-ce"
EOF

# Create Podman networks - these can be used by containers as needed
log "Creating Podman networks..."
podman network create backend-net
podman network create proxy-net
podman network create tunnel-net

systemctl enable --now podman.socket

# -------------------------------
# 6. Conditional Portainer Deployment
# -------------------------------
HOSTNAME_FQDN=$(hostname -f || echo "unknown")
log "Detected hostname: $HOSTNAME_FQDN"

mkdir -p /etc/containers/systemd

if [[ "$HOSTNAME_FQDN" == "alex" ]]; then
    log "Primary host — deploying Portainer CE..."
    cat <<'EOF' > /etc/containers/systemd/portainer-ce.container
[Unit]
Description=Portainer CE container
After=network-online.target podman.socket
Wants=network-online.target

[Service]
Restart=always

[Container]
Image=docker.io/portainer/portainer-ce:latest
PublishPort=9000:9000
PublishPort=9443:9443
Volume=/run/podman/podman.sock:/var/run/docker.sock
# Check this volume path for correctness
Volume=portainer_data:/data
PodmanArgs=--privileged

[Install]
WantedBy=multi-user.target
EOF

    # Set permissions, reload systemd, and start service
    chmod 644 /etc/containers/systemd/portainer-ce.container
    systemctl daemon-reexec  && systemctl daemon-reload
    systemctl start portainer-ce.service
    systemctl status portainer-ce.service
    configure_firewall_ports 9000 9443
    success "Portainer CE deployed and managed by systemd."


else


    log "Secondary host — deploying Portainer Agent..."
    cat <<'EOF' > /etc/containers/systemd/portainer-agent.container
# /etc/containers/systemd/portainer-agent.container
[Unit]
Description=Portainer Agent container
After=network-online.target podman.socket
Wants=network-online.target

[Service]
Restart=always

[Container]
Image=docker.io/portainer/agent:latest
PublishPort=9001:9001
Volume=/run/podman/podman.sock:/var/run/docker.sock
Volume=/var/lib/containers/storage/volumes:/var/lib/docker/volumes
Volume=/:/host
PodmanArgs=--privileged

[Install]
WantedBy=multi-user.target

EOF

    # Set permissions, reload systemd, and start service
    chmod 644 /etc/containers/systemd/portainer-agent.container
    systemctl daemon-reexec && systemctl daemon-reload
    systemctl start portainer-agent.service
    systemctl status portainer-agent.service
    configure_firewall_ports 9001
    success "Portainer Agent deployed and managed by systemd."
fi

# -------------------------------
# 7. Final Notes
# -------------------------------
success "Deployment complete."
log "Verify services with:"
log "  systemctl status portainer-ce.service"
log "  systemctl status portainer-agent.service"
log "Access Portainer UI at https://<host-ip>:9443 (primary) or https://<host-ip>:9001 (secondary)."

⚙️ Current Status & Next Steps

• Codebase is functional but not final — enough to get you moving forward
• Firewall rules still require fine‑tuning for optimal security and performance
• Remember to update the public IP section in Portainer to reflect the host’s FQDN
• Existing Docker stacks - need reviewing based on learnings.
• Setup scripts revised, rewritten and redeployed
• Base stack irrelevant due to code review - now only contains Beszel agent.
• Beszel and agents redeployed - monitor server and services logs
• Watchtower requires reviewing
• All stacks now require reviewing

Well at least the servers are up and running.

#enoughsaid