Projects

Project - Spiderfoot

Leon SCOTT

03 Sep 2026 • 7 min read

Project - Spiderfoot

SpiderFoot is an open-source OSINT automation tool. Point it at a domain, IP, email address or username and it runs a chain of specialised modules against public sources and APIs — DNS records, subdomains, breach databases, Shodan, VirusTotal and so on — to build up a picture of a target's footprint. It's used on both sides of the fence: red teams and attackers use it to map an attack surface before doing anything with it, blue teams use the exact same output to find what needs patching first. It doesn't break into anything itself; it just automates the recon legwork.

I wanted it running in my own homelab, and getting there turned into a proper project — migrating off a dead upstream, building my own image pipeline, and chasing down a run of genuinely interesting bugs before landing on a working AI-report feature. This is that story.

Why a fork, and why not just run the container

The original smicallef/spiderfoot image on Docker Hub is the one most people run, and it worked fine when I first stood it up. The problem is upstream: no commits to master since late 2023. Rather than sit on a dead project, I moved to the actively maintained poppopjmp/spiderfoot fork — a genuine rewrite, not a patch: Postgres, Redis, FastAPI and Celery behind a React frontend, split into per-service containers instead of one monolithic image.

That meant building my own images rather than pulling a ready one, so the first decision was where to build. I stood up a disposable box purely for building and testing, kept the network isolated (its own bridge network, only the services I actually needed enabled — I skipped the storage, monitoring, scheduler and SSO profiles for a first pass), and left Traefik out entirely since my own reverse proxy already handles TLS and routing.

Building the pipeline

build-spiderfoot.sh does the actual building: clones and tracks the fork's source, builds each service image, tags everything explicitly as <version>-g<shortsha> — never :latest — and pushes to my own GitHub Container Registry namespace. That tagging discipline isn't cosmetic; a reused floating tag was the exact root cause of an image-confusion bug I'd already hit on a separate project, where a container's running image and its tag quietly drifted apart. Explicit tags mean that can't happen here.

The build box compiles a genuinely heavy workload — the active-scanner service alone builds 30+ tools from source, some in Go and Rust — so the script prunes Docker's build cache both before and after every run. I'd originally only pruned beforehand, which left each run's own cache sitting there afterward; disk usage crept from a healthy amount up to the low 30s of GB before I noticed and fixed it, and pruning immediately after the build dropped it straight back down.

GHCR itself had one sharp edge worth knowing: every package defaults to private on its first-ever push, with no way to make it public up front. Each of the four images needed a manual one-time flip to public — now documented directly in the script.

I also moved secrets out of the build script's hands and into the shell's own environment.

Getting the stack itself to run

This is where most of the real debugging happened, once images existed and it was time to actually stand the stack up:

The core stack silently couldn't reach Postgres or Redis at all — turned out my firewall's forward chain only allowlisted specific known container subnets, and this stack's auto-assigned one wasn't on the list, so its own inter-container traffic was being dropped without a trace. Fixed at the source by widening the allowlist to a full private supernet rather than maintaining a specific list forever.

With the frontend loading, every API call from it returned a 502. The frontend's baked-in nginx config proxies to a hostname the compose file never actually defined — a naming mismatch between what nginx expected and what I'd called the API service. Fixed by adding the expected name as a network alias rather than renaming the service everywhere.

Scans crashed outright with a permission error writing to a runtime cache directory. The tmpfs mount backing it wasn't using the world-writable default I expected, so the process couldn't write to its own path — fixed by setting the mode explicitly rather than relying on any default.

Health checks across two of the services reported unhealthy despite clean logs — wrong port, wrong path, and in one case a healthcheck relying on a binary that isn't even in the image. All three fixed by pointing the check at the actual dedicated health endpoint the app exposes internally.

And twice, login locked out after a handful of failed attempts and then stayed locked even after the lockout window passed, because the app tracks lockout via two separate fields (a timer and a status flag) and only its own successful-login path resets both together — nothing else clears the status flag once the timer expires. Both times: clear the stale account row directly in the database and let the app's own bootstrap logic recreate it cleanly.

None of these were exotic — every one was found by reading the actual running container's logs and state rather than guessing, which is really the whole methodology here.

Moving from a Docker to Podman

Once the core stack was proven, I moved the whole thing off the disposable Docker box and onto my actual Podman host — the build pipeline now just pushes tagged images to GHCR, and the production stack pulls them with no local build step, no build toolchain, and none of the disk pressure that comes with compiling from source.

The move surfaced one more real bug: the frontend's nginx has its DNS resolver hardcoded to Docker's fixed internal DNS address, which doesn't exist under Podman — every internal API call failed a silent DNS lookup, which is also why the login page never even rendered (the very first auth check just hung). The image actually ships a proper fix for this already — an entrypoint script that detects the real resolver address at startup — it's just gated behind a flag that's off by default and wired to a variable nothing sets. I patched the one line in the actual frontend source (not just at deploy time) so every future build carries the fix automatically, with a hard failure at build time if that line ever stops matching — worth carrying as a real patch rather than a workaround, and worth raising upstream.

Adding AI-generated reports

The fork ships an optional AI reporting feature — a set of agents that summarise scan results into a written report. Getting it running was its own small saga.

My first instinct was to route it through a local LLM the same way another project in my homelab already does, on a disposable Ollama container sitting next to the stack rather than touching my actual production inference host. That plan stalled on a GPU passthrough problem specific to the build box (turned out to be a reboot away from fixed, and then a missing container runtime configuration once it was), and once it was working I hit a model-sizing mismatch — one of the seven report agents was hardcoded to request a model I'd never configured.

At that point I made the call to skip local inference entirely and route everything through Anthropic's API instead, the same pattern I already use elsewhere in the homelab. It's a genuine simplification: no GPU passthrough, no VRAM budgeting, no model tag drift, just an API key. LiteLLM sits in front as the routing layer and needed exactly one behavioural fix — the agents send a temperature value the newer Claude models don't accept, so LiteLLM is now configured to silently drop parameters a given model doesn't support rather than erroring the whole request.

That unlocked the last and most interesting bug: every report attempt was getting cut off at almost exactly 30 seconds, no matter what I changed server-side. I ruled out the reverse proxy, then the app's own backend timeout config, then bypassed my reverse proxy entirely to rule out anything in between — same cutoff every time, which meant it had to be client-side. It was: the frontend's bundled JavaScript has a hardcoded 30-second timeout on every API call, not exposed as a setting anywhere. Patching that (again, at build time, in source) revealed a second ceiling underneath it — the reverse-proxy-facing nginx location for the AI endpoints had its own timeout, separately configured and never revisited. And underneath that was a third: the report generator itself hardcodes its own internal timeout in Python, after environment variables are already read, invisible to any config check. All three are now patched consistently, each comfortably above the one below it, so a genuine hang surfaces as an honest error instead of a mysterious silent cutoff.

Once all three were aligned, I measured actual generation speed directly against the model rather than guessing: consistently in the range of 90-100 tokens/second, meaning a substantial report can legitimately take over a minute to generate. Every earlier "timeout" was correct behaviour being cut short, not a real failure.

Where it stands now

The stack runs end-to-end: core services, scanning, and AI-generated reports via Anthropic's API, all built from tagged images with no local build step on the host that actually runs it.
A handful of the fixes above are genuinely upstream-worthy rather than homelab-specific — the DNS resolver gap and the hardcoded frontend timeout especially — so I've drafted a consolidated writeup to raise with the fork's maintainer rather than splitting it into several disconnected pull requests.

We are now working on the GPU problem. Due to the massive amounts of data depending on the scan this is filtered via sf_agent to an internal model that reduces the amount of data whilst keeping it real before its sent to the AI for processing the report.
There is also the modules and settings tab that need an overhaul as its rather ugly
Then there is checking every module for functionality. All of them.