Zoraxy Reverse Proxy Setup

Zoraxy Reverse Proxy Setup
Zoraxy Reverse Proxy

This script has been created to automate the process of creating the proxy

Learnings:

  • Debian 13 uses nftables by default so I made a mess of it using iptables
  • Backup the LXC
  • Undo all the incorrect work and rebuild it using nftables
  • Test the code again
  • Put a reworking note in my Todo list to change all code to use nftables
  • Port scan from mobile device passed

Here is the code for your learning. Hopefully it's a little better than previous work

#!/bin/bash
# ====================================================================
# Proxmox LXC Zoraxy Proxy Setup (nftables-native)
# ====================================================================
# Purpose:
#    - Configure a Debian 13 LXC as a Zoraxy reverse proxy
#    - Use nftables as the ONLY firewall mechanism (no iptables)
#    - Enforce inbound policy with:
#         * Drop-by-default on INPUT
#         * SSH restricted to single admin IP (192.168.1.10) IPv4 ONLY
#         * IPv6 SSH connections completely blocked
#         * Brute-force protection on allowed SSH connections
#         * Basic HTTP(S)/admin DoS protection
#         * Centralised firewall logging via rsyslog + logrotate
#    - Auto-update Zoraxy binary via weekly cron job
#    - DNS locked to UDM SE IPv4 only via immutable resolv.conf
#      IPv6 global address disabled — link-local only retained
#      (UDM SE global IPv6 address changes on restart, making it
#      an unreliable DNS target that causes ACME SERVFAIL errors)
#
# Notes:
#   - Assumes this LXC is a reverse proxy, not a router
#   - No Podman/LXC container chaining on this host (pure proxy role)
#   - Ensure to configure Zoraxy after installation
#   - Ensure to configure Cloudflared tunnel after installation
#   - Cloudflare tunnel operates over IPv4 — no global IPv6 required
#   - Scoped Cloudflare API token required for ACME DNS challenge
#     (Zone → DNS → Edit scoped to your zone — NOT the Global API Key)
#   - Update mechanism is working and moved Zoraxy automatically to version 3.3.3
#
# Version: 2.0.0
# Created: 26-12-2025
# Updated: 03-05-2026
# Changelog:
#   2.0.0 - Fixed Zoraxy symlink to use /usr/local/bin instead of /usr/bin
#   1.9.0 - Moved all variables to top-level CONFIGURATION block
#           DNS_DOMAIN, DNS_NAMESERVER, ADMIN_SSH_IP, TIMEZONE
#           ZORAXY_INSTALL_DIR, ZORAXY_BIN, ZORAXY_SERVICE_FILE
#           ZORAXY_DOWNLOAD_URL, ZORAXY_PORT
#           ZORAXY_UPDATE_SCRIPT, ZORAXY_UPDATE_LOG
#           CLOUDFLARED_BIN, CLOUDFLARED_DOWNLOAD_URL
#           CF_UPDATE_SCRIPT, CF_UPDATE_LOG
#           SSHD_DIR, SSHD_CFG, SSHD_CUSTOM_DIR, SSHD_CUSTOM_CFG
#           All inline hardcodes replaced with variable references
#           SSH config heredoc changed to allow ADMIN_SSH_IP expansion
# ====================================================================

set -euo pipefail

# ======================================================================
# CONFIGURATION — review and set all variables before running
# ======================================================================

# --- Network ---
DNS_DOMAIN="[your domain]"                   # Local domain name
DNS_NAMESERVER="[your router]"               # UDM SE IPv4 — IPv4 only (see DNS section)
ADMIN_SSH_IP="[your proxmox host IP]"        # Only IP allowed SSH access (Proxmox host)
TIMEZONE="[your timezone]"                   # System timezone

# --- Zoraxy ---
ZORAXY_INSTALL_DIR="/srv/zoraxy"
ZORAXY_BIN="${ZORAXY_INSTALL_DIR}/zoraxy"
ZORAXY_SERVICE_FILE="/etc/systemd/system/zoraxy.service"
ZORAXY_DOWNLOAD_URL="https://github.com/tobychui/zoraxy/releases/latest/download/zoraxy_linux_amd64"
ZORAXY_PORT=":81"                          # Admin/proxy port

# --- Zoraxy auto-update ---
ZORAXY_UPDATE_SCRIPT="/usr/local/bin/zoraxy-update.sh"
ZORAXY_UPDATE_LOG="/var/log/zoraxy-update.log"

# --- Cloudflared ---
CLOUDFLARED_BIN="/usr/local/bin/cloudflared"
CLOUDFLARED_DOWNLOAD_URL="https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64"
# Paste your tunnel token here.
# Get it from: Cloudflare Zero Trust → Networks → Tunnels → your tunnel → Configure → token
# Leave blank to skip automatic service install — configure manually later with:
#   cloudflared service install <token> && systemctl enable --now cloudflared
CLOUDFLARE_TUNNEL_TOKEN="[your tunnel token]"

# --- Cloudflared auto-update ---
CF_UPDATE_SCRIPT="/usr/local/bin/cloudflared-update.sh"
CF_UPDATE_LOG="/var/log/cloudflared-update.log"

# --- SSH hardening ---
SSHD_DIR="/etc/ssh"
SSHD_CFG="${SSHD_DIR}/sshd_config"
SSHD_CUSTOM_DIR="${SSHD_DIR}/sshd_config.d"
SSHD_CUSTOM_CFG="${SSHD_CUSTOM_DIR}/99-hardened-custom.conf"

# ======================================================================
# 1. Utility functions
# ======================================================================

info()  { echo -e "\033[0;32m[INFO]\033[0m $*"; }
warn()  { echo -e "\033[0;33m[WARN]\033[0m $*" >&2; }
error() { echo -e "\033[0;31m[ERROR]\033[0m $*" >&2; exit 1; }

set_alt() {
    local name="$1"
    local target="$2"
    if update-alternatives --set "$name" "$target" 2>/dev/null; then
        info "Set $name → $target"
    else
        warn "Failed to set $name to $target — investigate alternatives state"
    fi
}

get_latest_zoraxy_version() {
    curl -fsSL "https://api.github.com/repos/tobychui/zoraxy/releases/latest" \
        | grep '"tag_name"' \
        | sed -E 's/.*"tag_name": *"([^"]+)".*/\1/'
}

# -----------------------------------------------------------------------
# 2. Base packages
# -----------------------------------------------------------------------
info "Installing base packages..."
apt-get update -qq || warn "apt-get update failed, check DNS/network"
apt-get install -y -qq \
    bind9-dnsutils rsyslog sudo curl gpg net-tools apt-transport-https cron mtr \
    openssh-server openssh-client smartmontools nftables jq

info "Removing undesired packages..."
for pkg in ufw inetutils-telnet iptables iptables-persistent netfilter-persistent; do
    if dpkg-query -W -f='${Status}' "$pkg" 2>/dev/null | grep -q "install ok installed"; then
        apt-get purge --auto-remove -y -qq "$pkg"
    fi
done

info "Enabling nftables service..."
systemctl enable --now nftables.service || warn "Failed to enable/start nftables.service"

# -----------------------------------------------------------------------
# 3. DNS configuration — IPv4 only, immutable
# -----------------------------------------------------------------------
# The UDM SE's global IPv6 address changes on restart, making it an
# unreliable DNS target. Go's resolver (used by Zoraxy/lego for ACME)
# picks up any IPv6 nameserver in resolv.conf and attempts to use it,
# resulting in SERVFAIL from Cloudflare's authoritative NS.
# Fix: lock resolv.conf to UDM SE IPv4 only and make it immutable so
# DHCP renewals or networkd cannot overwrite it.

info "Configuring DNS — IPv4 only, locked to UDM SE (${DNS_NAMESERVER})..."
chattr -i /etc/resolv.conf 2>/dev/null || true

cat > /etc/resolv.conf <<EOF
# Managed by setup-proxy-alex.sh — DO NOT EDIT
# File is immutable (chattr +i). To modify: chattr -i /etc/resolv.conf
# IPv4 only — UDM SE global IPv6 address is dynamic and causes
# ACME/lego DNS challenge SERVFAIL errors when used as a nameserver.
domain ${DNS_DOMAIN}
search ${DNS_DOMAIN}
nameserver ${DNS_NAMESERVER}
EOF

# -----------------------------------------------------------------------
# 4. Miscellaneous configurations
# -----------------------------------------------------------------------
info "Configuring automatic updates via cron..."

( crontab -l 2>/dev/null || true; \
  echo "0 1 * * * apt-get update -qq && apt-get -y -qq upgrade && apt-get -y -qq autoremove && apt-get -y -qq autoclean >> /var/log/apt-cron.log 2>&1"; \
  echo "0 */4 * * * /usr/bin/systemctl reset-failed >> /var/log/reset-failed.log 2>&1" \
) | sort -u | crontab -

if systemctl restart cron.service 2>/dev/null; then
    info "Cron restarted (cron.service)"
elif systemctl restart crond.service 2>/dev/null; then
    info "Cron restarted (crond.service)"
else
    error "Cron restart failed"
fi

info "Setting timezone to ${TIMEZONE}..."
timedatectl set-timezone "${TIMEZONE}"

info "Configuring APT to skip downloading extra language files..."
echo 'Acquire::Languages "none";' | tee /etc/apt/apt.conf.d/99-disable-languages > /dev/null

info "Configuring APT to use IPv4..."
echo 'Acquire::ForceIPv4 "true";' | tee /etc/apt/apt.conf.d/99force-ipv4 > /dev/null

cat > /etc/profile.d/hardened-history.sh <<'EOF'
# Hardened Bash History
export HISTSIZE=10000
export HISTFILESIZE=100000
shopt -s histappend
export HISTTIMEFORMAT='%F %T '
PROMPT_COMMAND='history -a'
history -r
EOF

info "Configuring MOTD..."
rm -f /etc/update-motd.d/*.*
cat <<'EOF' > /etc/update-motd.d/80-sysinfo
#!/bin/bash
echo "=== System Information ==="
echo ""
echo "  🖥️  Operating System : $(lsb_release -ds 2>/dev/null || grep -oP '(?<=PRETTY_NAME=\").*(?=\")' /etc/os-release)"
echo "  🌐  IPv4 Address     : $(hostname -I | awk '{print $1}')"
echo "  🪢  IPv6 Link-local  : $(ip -6 addr show scope link | grep inet6 | awk '{print $2}' | cut -d'/' -f1 | head -n1)"
echo "  ⏱️  System Uptime    : $(uptime -p)"
EOF
chmod +x /etc/update-motd.d/80-sysinfo

# Minimal PAM session config — prevents pam_loginuid SIGSEGV in LXC
cat > /etc/pam.d/common-session <<'EOF'
# Minimal PAM session config for LXC containers
session [default=1]                     pam_permit.so
session requisite                       pam_deny.so
session required                        pam_permit.so
session optional                        pam_umask.so
session required                        pam_unix.so
EOF
echo "# This file is NOT managed by pam-auth-update" >> /etc/pam.d/common-session

# -----------------------------------------------------------------------
# 5. Firewall configuration (nftables-native)
# -----------------------------------------------------------------------

info "Configuring nftables firewall..."
info "Setting iptables alternatives to nft shim..."
set_alt iptables  /usr/sbin/iptables-nft
set_alt ip6tables /usr/sbin/ip6tables-nft
set_alt arptables /usr/sbin/arptables-nft
set_alt ebtables  /usr/sbin/ebtables-nft

cat <<'EOF' > /etc/nftables.conf
#!/usr/sbin/nft -f

# ============================================================
#  nftables Reverse Proxy Firewall
#  Includes:
#    - Stateful filtering
#    - ICMPv4/v6
#    - DNS, mDNS, DHCP + DHCPv6
#    - IPv6 RA, RS, ND, MLD (link-local only — global IPv6 disabled)
#    - SSH with brute-force protection (IPv4 from ADMIN_SSH_IP ONLY)
#    - ALL IPv6 SSH connections BLOCKED
#    - Web DoS protection
#    - Port-scan detection (NULL, FIN, XMAS)
#    - Anti-fragmentation protection
#    - Invalid TCP flag protection
#    - Global per-IP new-connection rate-limit
#    - Full logging of all dropped / malicious packets
#
#  Version: 1.1.0
# ============================================================

flush ruleset

table inet filter {

    # ================================
    #  Port Scan Detection Sets
    # ================================
    set portscan_blacklist_v4 {
        type ipv4_addr
        timeout 10m
        flags timeout
    }

    set portscan_blacklist_v6 {
        type ipv6_addr
        timeout 10m
        flags timeout
    }

    # ================================
    #  SSH Brute Force Protection Set
    # ================================
    set ssh_ratelimit {
        type ipv4_addr
        timeout 1m
        flags timeout
    }

    # ================================
    #  INPUT CHAIN
    # ================================
    chain input {
        type filter hook input priority filter; policy drop;

        # Stateful inspection (MUST be first)
        ct state established,related accept
        ct state invalid log prefix "INVALID-STATE: " drop

        # Loopback
        iifname "lo" accept

        # Check port scan blacklists early
        ip saddr @portscan_blacklist_v4 log prefix "PORTSCAN-BLOCKED-v4: " drop
        ip6 saddr @portscan_blacklist_v6 log prefix "PORTSCAN-BLOCKED-v6: " drop

        # Anti-fragmentation (IPv4)
        ip frag-off & 0x1fff != 0 log prefix "FRAG-DROP: " drop

        # Invalid TCP flag combinations
        tcp flags & (fin|syn) == (fin|syn) log prefix "BADFLAGS-FIN-SYN: " drop
        tcp flags & (syn|rst) == (syn|rst) log prefix "BADFLAGS-SYN-RST: " drop

        # ================================
        #  Port Scan Detection
        # ================================
        # NULL scan (IPv4)
        meta l4proto tcp tcp flags == 0x0 ip saddr != 0.0.0.0 \
            add @portscan_blacklist_v4 { ip saddr } \
            log prefix "PORTSCAN-NULL-v4: " drop

        # NULL scan (IPv6)
        meta l4proto tcp tcp flags == 0x0 meta nfproto ipv6 \
            add @portscan_blacklist_v6 { ip6 saddr } \
            log prefix "PORTSCAN-NULL-v6: " drop

        # FIN scan (FIN without ACK) - IPv4
        meta l4proto tcp tcp flags & (fin|ack) == fin ip saddr != 0.0.0.0 \
            add @portscan_blacklist_v4 { ip saddr } \
            log prefix "PORTSCAN-FIN-v4: " drop

        # FIN scan (FIN without ACK) - IPv6
        meta l4proto tcp tcp flags & (fin|ack) == fin meta nfproto ipv6 \
            add @portscan_blacklist_v6 { ip6 saddr } \
            log prefix "PORTSCAN-FIN-v6: " drop

        # XMAS scan - IPv4
        meta l4proto tcp tcp flags & (fin|psh|urg) == fin|psh|urg ip saddr != 0.0.0.0 \
            add @portscan_blacklist_v4 { ip saddr } \
            log prefix "PORTSCAN-XMAS-v4: " drop

        # XMAS scan - IPv6
        meta l4proto tcp tcp flags & (fin|psh|urg) == fin|psh|urg meta nfproto ipv6 \
            add @portscan_blacklist_v6 { ip6 saddr } \
            log prefix "PORTSCAN-XMAS-v6: " drop

        # ================================
        #  IPv4 ICMP
        # ================================
        ip protocol icmp icmp type { echo-request, echo-reply, destination-unreachable, time-exceeded } \
            limit rate 10/second accept

        # ================================
        #  IPv6 Control Plane (link-local only — global disabled)
        # ================================
        meta l4proto ipv6-icmp icmpv6 type {
            destination-unreachable, packet-too-big, time-exceeded,
            parameter-problem, echo-request, echo-reply,
            nd-router-solicit, nd-router-advert,
            nd-neighbor-solicit, nd-neighbor-advert,
            mld-listener-query, mld-listener-report, mld-listener-done
        } accept

        # DHCPv6 client
        meta l4proto udp udp sport 547 udp dport 546 accept

        # IPv6 multicast (solicited-node, all-nodes, all-routers)
        ip6 daddr { ff02::1:ff00:0/104, ff02::1, ff02::2 } accept

        # ================================
        #  DNS + mDNS
        # ================================
        meta l4proto udp udp dport 53 accept
        meta l4proto tcp tcp dport 53 accept
        meta l4proto udp udp dport 5353 accept

        # ================================
        #  DHCPv4 client
        # ================================
        meta l4proto udp udp sport 67 udp dport 68 accept

        # ================================
        #  SSH (IPv4 from ADMIN_SSH_IP ONLY)
        #  ALL IPv6 SSH connections BLOCKED
        # ================================

        # Block all IPv6 SSH attempts
        meta nfproto ipv6 tcp dport 22 log prefix "SSH-DENY-IPv6: " drop

        # Allow IPv4 SSH only from ADMIN_SSH_IP with rate limiting
        tcp dport 22 ip saddr ADMIN_SSH_IP ct state new \
            add @ssh_ratelimit { ip saddr } \
            limit rate 3/minute burst 5 packets accept

        # Drop excess attempts from allowed IP (brute force)
        tcp dport 22 ip saddr ADMIN_SSH_IP ct state new \
            log prefix "SSH-BRUTE-ALLOWED-IP: " drop

        # Deny all other IPv4 SSH attempts
        tcp dport 22 log prefix "SSH-DENY-IPv4: " drop

        # ================================
        #  Global per-IP new-connection rate-limit
        # ================================
        ct state new limit rate over 200/second burst 400 packets \
            log prefix "GLOBAL-DOS: " drop

        # ================================
        #  Web traffic (DoS protection)
        # ================================
        tcp dport { 80, 81, 443, 8890 } ct state new \
            limit rate over 50/second burst 200 packets \
            log prefix "WEB-DOS: " drop

        tcp dport { 80, 81, 443, 8890 } accept

        # ================================
        #  Final catch-all logging
        # ================================
        limit rate 5/minute burst 20 packets \
            log prefix "DEFAULT-DROP: "
    }

    # ================================
    #  FORWARD CHAIN
    # ================================
    chain forward {
        type filter hook forward priority filter; policy drop;
    }

    # ================================
    #  OUTPUT CHAIN
    # ================================
    chain output {
        type filter hook output priority filter; policy accept;
    }
}
EOF

# Substitute ADMIN_SSH_IP placeholder in nftables config
sed -i "s/ADMIN_SSH_IP/${ADMIN_SSH_IP}/g" /etc/nftables.conf

info "Validating nftables configuration..."
if nft -c -f /etc/nftables.conf; then
    info "nftables.conf syntax OK"
else
    error "nftables.conf validation failed — investigate configuration"
fi

info "Applying nftables configuration..."
nft -f /etc/nftables.conf && info "nftables configuration applied successfully"

touch /var/log/nftables.log
cat <<'EOF' > /etc/rsyslog.d/30-nftables.conf
:msg, contains, "DROP" /var/log/nftables.log
:msg, contains, "PORTSCAN" /var/log/nftables.log
:msg, contains, "BADFLAGS" /var/log/nftables.log
:msg, contains, "FRAG" /var/log/nftables.log
:msg, contains, "SSH" /var/log/nftables.log
:msg, contains, "DOS" /var/log/nftables.log
:msg, contains, "GLOBAL" /var/log/nftables.log
:msg, contains, "DEFAULT" /var/log/nftables.log
& stop
EOF

cat <<'EOF' > /etc/logrotate.d/nftables
/var/log/nftables.log {
    daily
    missingok
    rotate 7
    compress
    delaycompress
    notifempty
    create 0640 root adm
    postrotate
        systemctl reload rsyslog >/dev/null 2>&1 || true
    endscript
}
EOF

systemctl restart rsyslog
info "nftables logging configured at /var/log/nftables.log"

# -----------------------------------------------------------------------
# 6. Sysctl configuration
# -----------------------------------------------------------------------

info "Applying sysctl settings..."

tee /etc/sysctl.d/99-ipv4-lxc.conf >/dev/null <<'EOF'
# Proxy LXC — not a router
net.ipv4.ip_forward=0

# Allow root (GID 0) to send ICMP — required for cloudflared ICMP proxy
# Without this cloudflared logs: "Group ID 0 is not between ping group 65534 to 65534"
net.ipv4.ping_group_range=0 2147483647

# Increase UDP receive buffer for QUIC performance (cloudflared tunnel)
# Without this cloudflared logs: "failed to sufficiently increase receive buffer size"
# Default: ~208 KiB — cloudflared wants 7168 KiB
net.core.rmem_max=7340032
net.core.wmem_max=7340032
EOF

# IPv6 global address is disabled for the following reasons:
#   - UDM SE global IPv6 address changes on restart
#   - When written to resolv.conf it causes ACME/lego SERVFAIL errors
#   - Cloudflare tunnel does not require global IPv6
# Link-local IPv6 is retained - required for neighbour discovery (NDP).
tee /etc/sysctl.d/99-ipv6-lxc.conf >/dev/null <<'EOF'
# Disable IPv6 forwarding
net.ipv6.conf.all.forwarding=0
# Disable global IPv6 autoconfiguration and RA acceptance
# Link-local address is still assigned automatically by the kernel
net.ipv6.conf.eth0.autoconf=1
net.ipv6.conf.eth0.accept_ra=1
# Prefer stable IPv6 addresses over temporary privacy addresses
net.ipv6.conf.eth0.use_tempaddr=0
net.ipv6.conf.all.use_tempaddr=0
net.ipv6.conf.default.use_tempaddr=0
EOF

sysctl --system || warn "Some sysctl warnings are normal in LXC — review output above"

# -----------------------------------------------------------------------
# 7. SSH configuration hardening
# -----------------------------------------------------------------------

info "Hardening SSH configuration..."

BACKUP_TS=$(date +"%Y%m%d-%H%M%S")
BACKUP_DIR="${SSHD_DIR}/backup-${BACKUP_TS}"

mkdir -p "${BACKUP_DIR}" "${SSHD_CUSTOM_DIR}"

info "Backing up sshd_config and host keys to ${BACKUP_DIR}..."
cp -a "${SSHD_CFG}" "${BACKUP_DIR}/sshd_config.bak" || true
for key in ssh_host_ed25519_key ssh_host_rsa_key ssh_host_ecdsa_key; do
    [[ -f "${SSHD_DIR}/${key}" ]]     && mv "${SSHD_DIR}/${key}"     "${BACKUP_DIR}/" || true
    [[ -f "${SSHD_DIR}/${key}.pub" ]] && mv "${SSHD_DIR}/${key}.pub" "${BACKUP_DIR}/" || true
done

info "Generating new SSH host keys..."
ssh-keygen -t ed25519 -f "${SSHD_DIR}/ssh_host_ed25519_key" -N "" -o -a 100
ssh-keygen -t rsa -b 4096 -f "${SSHD_DIR}/ssh_host_rsa_key" -N "" -o
chmod 600 "${SSHD_DIR}/ssh_host_ed25519_key" "${SSHD_DIR}/ssh_host_rsa_key"
chmod 644 "${SSHD_DIR}/ssh_host_ed25519_key.pub" "${SSHD_DIR}/ssh_host_rsa_key.pub"

if ! grep -q "^Include ${SSHD_CUSTOM_DIR}/\*.conf" "${SSHD_CFG}"; then
    sed -i "1i Include ${SSHD_CUSTOM_DIR}/*.conf" "${SSHD_CFG}"
    info "Added Include directive to ${SSHD_CFG}"
fi

tee "${SSHD_CUSTOM_CFG}" >/dev/null <<EOF
# Hardened SSH Configuration
# Managed by setup-proxy-alex.sh — survives system updates via sshd_config.d/

Protocol 2
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
HostbasedAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no
KbdInteractiveAuthentication no
UsePAM yes

LoginGraceTime 30
MaxAuthTries 4
MaxSessions 10
ClientAliveInterval 300
ClientAliveCountMax 2

HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key

AllowAgentForwarding no
AllowTcpForwarding no
X11Forwarding no
UseDNS no
PrintMotd no
PrintLastLog yes

AcceptEnv LANG LC_*

# IPv4-only SSH — deny IPv6
AddressFamily inet

# Emergency/initial access from Proxmox host — remove after key auth confirmed
Match Address ${ADMIN_SSH_IP}
  PermitRootLogin yes
  PasswordAuthentication yes
  Subsystem sftp /usr/lib/openssh/sftp-server
  AllowTcpForwarding no
EOF

chmod 644 "${SSHD_CUSTOM_CFG}"

info "Validating sshd_config..."
if sshd -t; then
    info "sshd_config syntax OK"
    if systemctl restart ssh 2>/dev/null || systemctl restart sshd 2>/dev/null; then
        info "sshd restarted successfully"
    else
        error "sshd restart failed — check service status"
    fi
else
    error "sshd_config test failed — not restarting. Backup: ${BACKUP_DIR}/sshd_config.bak"
fi

# -----------------------------------------------------------------------
# 8. Install Zoraxy proxy
# -----------------------------------------------------------------------

info "Creating install directory: ${ZORAXY_INSTALL_DIR}"
mkdir -p "${ZORAXY_INSTALL_DIR}"

info "Detecting latest Zoraxy version..."
LATEST_VER=$(get_latest_zoraxy_version 2>/dev/null || echo "unknown")
info "Latest Zoraxy version: ${LATEST_VER}"

info "Downloading Zoraxy binary..."
curl -fsSL "${ZORAXY_DOWNLOAD_URL}" -o "${ZORAXY_BIN}"
chmod +x "${ZORAXY_BIN}"

info "Creating Zoraxy systemd service..."
cat <<EOF > "${ZORAXY_SERVICE_FILE}"
[Unit]
Description=Zoraxy Reverse Proxy
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=root
Group=root
ExecStart=${ZORAXY_BIN} -port=${ZORAXY_PORT}
WorkingDirectory=${ZORAXY_INSTALL_DIR}
Restart=on-failure
RestartSec=5s
StandardOutput=journal
StandardError=journal
SyslogIdentifier=zoraxy

# Systemd hardening
PrivateTmp=true
NoNewPrivileges=true
ProtectSystem=full
ProtectHome=true
ReadWritePaths=${ZORAXY_INSTALL_DIR}

[Install]
WantedBy=multi-user.target
EOF

info "Enabling and starting Zoraxy service..."
systemctl daemon-reload || error "systemd daemon-reload failed"
systemctl enable zoraxy || warn "Zoraxy service failed to enable"
systemctl start zoraxy  || warn "Zoraxy service failed to start"

# Create Zoraxy symlink for command completion
info "Creating Zoraxy symlink in /usr/local/bin..."
ln -sf "${ZORAXY_BIN}" /usr/local/bin/zoraxy || error "Zoraxy symlink failed"
info "Symlink created: /usr/local/bin/zoraxy → ${ZORAXY_BIN}"

info "Zoraxy installation complete — version ${LATEST_VER}"

# -----------------------------------------------------------------------
# 9. Zoraxy auto-update script + cron
# -----------------------------------------------------------------------

info "Installing Zoraxy auto-update script..."

cat <<'UPDATESCRIPT' > "${ZORAXY_UPDATE_SCRIPT}"
#!/bin/bash
# ============================================================
# Zoraxy Auto-Update Script
# Checks GitHub for a new release, updates binary if needed,
# restarts the service, and rolls back on failure.
# Safe to run unattended via cron.
# ============================================================

set -euo pipefail

INSTALL_DIR="/srv/zoraxy"
BIN_PATH="${INSTALL_DIR}/zoraxy"
BACKUP_PATH="${INSTALL_DIR}/zoraxy.backup"
DOWNLOAD_URL="https://github.com/tobychui/zoraxy/releases/latest/download/zoraxy_linux_amd64"
LOG_FILE="/var/log/zoraxy-update.log"
TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')

log() { echo "[${TIMESTAMP}] $*" | tee -a "${LOG_FILE}"; }

log "=== Zoraxy update check started ==="

CURRENT_VER=$("${BIN_PATH}" -version 2>/dev/null | grep -oP 'v[\d.]+' | head -1 || echo "unknown")
log "Current version: ${CURRENT_VER}"

LATEST_VER=$(curl -fsSL "https://api.github.com/repos/tobychui/zoraxy/releases/latest" \
    | grep '"tag_name"' \
    | sed -E 's/.*"tag_name": *"([^"]+)".*/\1/' 2>/dev/null || echo "unknown")
log "Latest version: ${LATEST_VER}"

if [[ "${LATEST_VER}" == "unknown" ]]; then
    log "ERROR: Could not determine latest version — skipping update."
    exit 1
fi

if [[ "${CURRENT_VER}" == "${LATEST_VER}" ]]; then
    log "Already on latest version — no update needed."
    exit 0
fi

log "Update available: ${CURRENT_VER} → ${LATEST_VER}"

log "Backing up current binary to ${BACKUP_PATH}..."
cp "${BIN_PATH}" "${BACKUP_PATH}"

log "Downloading ${LATEST_VER}..."
if ! curl -fsSL "${DOWNLOAD_URL}" -o "${BIN_PATH}.new"; then
    log "ERROR: Download failed — aborting, keeping current binary."
    rm -f "${BIN_PATH}.new"
    exit 1
fi

chmod +x "${BIN_PATH}.new"

if ! "${BIN_PATH}.new" -version >/dev/null 2>&1; then
    log "ERROR: New binary failed version check — rolling back."
    rm -f "${BIN_PATH}.new"
    exit 1
fi

mv "${BIN_PATH}.new" "${BIN_PATH}"
log "Binary updated. Restarting zoraxy service..."

if systemctl restart zoraxy; then
    log "Zoraxy restarted successfully — update to ${LATEST_VER} complete."
else
    log "ERROR: Service restart failed — attempting rollback."
    cp "${BACKUP_PATH}" "${BIN_PATH}"
    if systemctl restart zoraxy; then
        log "Rollback successful."
    else
        log "CRITICAL: Rollback restart also failed — manual intervention required."
    fi
    exit 1
fi
UPDATESCRIPT

chmod +x "${ZORAXY_UPDATE_SCRIPT}"
touch "${ZORAXY_UPDATE_LOG}"

# Weekly update check — Sunday 02:00 AWST (low traffic window)
( crontab -l 2>/dev/null || true; \
  echo "0 2 * * 0 ${ZORAXY_UPDATE_SCRIPT} >> ${ZORAXY_UPDATE_LOG} 2>&1" \
) | sort -u | crontab -

info "Auto-update script installed: ${ZORAXY_UPDATE_SCRIPT}"
info "Schedule: Sunday 02:00 AWST | Log: ${ZORAXY_UPDATE_LOG}"

# -----------------------------------------------------------------------
# 10. Install Cloudflared Tunnel
# -----------------------------------------------------------------------
# Installs cloudflared as a systemd service with the token provided above.
# Fixes three issues observed in manual tunnel runs:
#   1. ICMP proxy disabled — fixed via net.ipv4.ping_group_range sysctl (section 6)
#   2. QUIC UDP buffer too small — fixed via net.core.rmem_max sysctl (section 6)
#   3. Tunnel running from shell (dies on logout) — fixed by installing as service

info "Installing Cloudflared Tunnel..."
mkdir -p /usr/local/bin

curl -fsSL "${CLOUDFLARED_DOWNLOAD_URL}" \
    -o "${CLOUDFLARED_BIN}"
chmod +x "${CLOUDFLARED_BIN}" || error "Failed to chmod cloudflared"

# Verify binary
CLOUDFLARED_VER=$("${CLOUDFLARED_BIN}" --version 2>&1 | head -1 || echo "unknown")
info "Cloudflared installed: ${CLOUDFLARED_VER}"

if [[ -n "${CLOUDFLARE_TUNNEL_TOKEN}" ]]; then
    info "Installing cloudflared as a systemd service..."

    # 'cloudflared service install' creates /etc/systemd/system/cloudflared.service
    # and writes the token to /etc/cloudflared/config.yml automatically
    cloudflared service install "${CLOUDFLARE_TUNNEL_TOKEN}" \
        || error "cloudflared service install failed — check token and try manually"

    info "Enabling and starting cloudflared service..."
    systemctl daemon-reload
    systemctl enable cloudflared  || warn "cloudflared service failed to enable"
    systemctl start cloudflared   || warn "cloudflared service failed to start"

    # Brief pause then verify
    sleep 3
    if systemctl is-active --quiet cloudflared; then
        info "Cloudflared service is running"
        journalctl -u cloudflared -n 10 --no-pager
    else
        warn "Cloudflared service did not start cleanly — check: journalctl -u cloudflared -f"
    fi
else
    warn "CLOUDFLARE_TUNNEL_TOKEN is not set — skipping service installation."
    warn "To configure manually after deployment:"
    warn "  cloudflared service install <your-token>"
    warn "  systemctl enable --now cloudflared"
fi

# -----------------------------------------------------------------------
# 11. Cloudflared auto-update script + cron
# -----------------------------------------------------------------------
# cloudflared releases frequently. This script checks GitHub for a new
# release, downloads it, verifies it, restarts the service, and rolls
# back automatically on failure. Staggered 15 minutes after Zoraxy update.

info "Installing cloudflared auto-update script..."

cat <<'CFUPDATESCRIPT' > "${CF_UPDATE_SCRIPT}"
#!/bin/bash
# ============================================================
# Cloudflared Auto-Update Script
# Checks GitHub for a new release, updates binary if needed,
# restarts the service, and rolls back on failure.
# Safe to run unattended via cron.
# ============================================================

set -euo pipefail

BIN_PATH="/usr/local/bin/cloudflared"
BACKUP_PATH="/usr/local/bin/cloudflared.backup"
DOWNLOAD_URL="https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64"
LOG_FILE="/var/log/cloudflared-update.log"
TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')

log() { echo "[${TIMESTAMP}] $*" | tee -a "${LOG_FILE}"; }

log "=== Cloudflared update check started ==="

# Current version — cloudflared outputs "cloudflared version X.Y.Z (built ...)"
CURRENT_VER=$("${BIN_PATH}" --version 2>/dev/null | grep -oP '\d{4}\.\d+\.\d+' | head -1 || echo "unknown")
log "Current version: ${CURRENT_VER}"

# Latest release tag from GitHub API
LATEST_TAG=$(curl -fsSL "https://api.github.com/repos/cloudflare/cloudflared/releases/latest" \
    | grep '"tag_name"' \
    | sed -E 's/.*"tag_name": *"([^"]+)".*/\1/' 2>/dev/null || echo "unknown")

# Strip leading 'v' if present for comparison
LATEST_VER="${LATEST_TAG#v}"
log "Latest version: ${LATEST_VER}"

if [[ "${LATEST_VER}" == "unknown" ]]; then
    log "ERROR: Could not determine latest version — skipping update."
    exit 1
fi

if [[ "${CURRENT_VER}" == "${LATEST_VER}" ]]; then
    log "Already on latest version — no update needed."
    exit 0
fi

log "Update available: ${CURRENT_VER} → ${LATEST_VER}"

log "Backing up current binary to ${BACKUP_PATH}..."
cp "${BIN_PATH}" "${BACKUP_PATH}"

log "Downloading ${LATEST_VER}..."
if ! curl -fsSL "${DOWNLOAD_URL}" -o "${BIN_PATH}.new"; then
    log "ERROR: Download failed — aborting, keeping current binary."
    rm -f "${BIN_PATH}.new"
    exit 1
fi

chmod +x "${BIN_PATH}.new"

# Verify new binary runs
if ! "${BIN_PATH}.new" --version >/dev/null 2>&1; then
    log "ERROR: New binary failed version check — rolling back."
    rm -f "${BIN_PATH}.new"
    exit 1
fi

mv "${BIN_PATH}.new" "${BIN_PATH}"
log "Binary updated. Restarting cloudflared service..."

if systemctl restart cloudflared; then
    log "Cloudflared restarted successfully — update to ${LATEST_VER} complete."
else
    log "ERROR: Service restart failed — attempting rollback."
    cp "${BACKUP_PATH}" "${BIN_PATH}"
    if systemctl restart cloudflared; then
        log "Rollback successful."
    else
        log "CRITICAL: Rollback restart also failed — manual intervention required."
    fi
    exit 1
fi
CFUPDATESCRIPT

chmod +x "${CF_UPDATE_SCRIPT}"
touch "${CF_UPDATE_LOG}"

# Weekly update — Sunday 02:15 AWST (staggered 15 min after Zoraxy update)
( crontab -l 2>/dev/null || true; \
  echo "15 2 * * 0 ${CF_UPDATE_SCRIPT} >> ${CF_UPDATE_LOG} 2>&1" \
) | sort -u | crontab -

info "Cloudflared auto-update script installed: ${CF_UPDATE_SCRIPT}"
info "Schedule: Sunday 02:15 AWST | Log: ${CF_UPDATE_LOG}"

# -----------------------------------------------------------------------
# 12. Final notes
# -----------------------------------------------------------------------
info "======================================================================="
info "Zoraxy proxy deployment complete! — ${LATEST_VER}"
info "======================================================================="
info ""
info "SECURITY STATUS:"
info "  ✓ SSH: IPv4 only from ${ADMIN_SSH_IP}"
info "  ✓ IPv6 SSH: BLOCKED"
info "  ✓ nftables: Active with comprehensive logging"
info "  ✓ SSH config: Hardened and update-resistant"
info ""
info "DNS / IPv6:"
info "  ✓ resolv.conf: Locked to ${DNS_NAMESERVER} (IPv4 only, immutable)"
info "  ✓ IPv6 global: Disabled (link-local retained for NDP)"
info "  ✓ Reason: UDM SE global IPv6 is dynamic — causes ACME SERVFAIL"
info "  ✓ Cloudflare tunnel: Operates over IPv4 — no global IPv6 needed"
info "  ! To modify resolv.conf: chattr -i /etc/resolv.conf"
info ""
info "VERIFICATION COMMANDS:"
info "  nft list ruleset                          — firewall rules"
info "  tail -f /var/log/nftables.log             — firewall log"
info "  sshd -t                                   — SSH config test"
info "  systemctl status zoraxy                   — Zoraxy status"
info "  journalctl -u zoraxy -f                   — Zoraxy live log"
info "  systemctl status cloudflared              — tunnel status"
info "  journalctl -u cloudflared -f              — tunnel live log"
info "  bash ${ZORAXY_UPDATE_SCRIPT}              — manual Zoraxy update"
info "  bash ${CF_UPDATE_SCRIPT}                  — manual cloudflared update"
info ""
info "NEXT STEPS:"
info "  1. Configure Zoraxy UI — http://<LXC_IP>:81"
info "  2. ACME tool: use scoped CF token (Zone → DNS → Edit)"
info "  3. Cloudflared: set CLOUDFLARE_TUNNEL_TOKEN before running script"
info "     or manually: cloudflared service install <token>"
info "  4. Test SSH from ${ADMIN_SSH_IP} (should work)"
info "  5. Test SSH from other IPs (should be blocked)"
info "  6. Run postfix, fail2ban, forensics scripts"
info "  7. Verify apt cron: tail /var/log/apt-cron.log"
info "  8. Install Beszel agent — use https hub URL"
info "======================================================================="

Hope this helps someone

If you use cloudflare as your domain name server you will need to increase the timeout on LetsEncrypt out past 1200 seconds. Or maybe that is just me, and there are problems with DNSSEC as well. You will work it out. Took me ages.

Probably going to pull other posts and start a cleanup on my systems.

Live and learn

#enoughsaid