Project - Spiderfoot

Yikes this is interesting.

For the past several weeks I've been running SpiderFoot as the manual investigation dashboard in my homelab — a self-hosted OSINT correlation platform that fans a single domain or IP out across dozens of data sources and stitches the results into an attack-surface picture. It sits alongside the rest of my monitoring stack, and every so often I point it at one of my own domains just to see what the internet currently knows about it.

The scan itself always worked. It came back with thousands of events — DNS records, WHOIS data, affiliate infrastructure, blacklist hits, the works. The problem was what happened next: SpiderFoot's AI-generated summary report. It read fluently, it was formatted like a proper CTI deliverable, and it said almost nothing. Every specific field — IP addresses, ASN numbers, SPF record contents, the actual blacklist source — came back as "not provided in the summarized data" or a flat "N/A." The report looked like intelligence. It wasn't.

That gap between "the scan worked" and "the AI told me anything useful about it" turned into a proper investigation: a forked repository, nine confirmed bugs, and a rebuilt reporting pipeline that now produces reports worth reading. This post is the write-up — what SpiderFoot is, what the fork changed, what it says about where AI-augmented security tooling actually stands right now, and an honest look at what we gained and what we gave up to get there.

1. The Source: SpiderFoot

SpiderFoot is one of the older, more established names in open source OSINT automation. It was created by Steve Micallef in 2012 and has been maintained continuously since, released under the MIT license at github.com/smicallef/spiderfoot. In its original form it's a single Python 3 application with an embedded web server, over 200 modules, and integrations across more than 100 public data sources — DNS, WHOIS, threat intelligence feeds, breach databases, social media, certificate transparency logs, and more. Point it at a domain, IP, email address, or name, pick your modules, and it correlates everything it finds into a browsable event graph.

It's a genuinely useful tool, and it's earned its reputation: SpiderFoot shows up repeatedly in OSINT tutorials, red-team tooling lists, and threat-intel writeups as a reliable way to automate the reconnaissance phase that would otherwise mean manually querying a dozen separate services by hand.

The specific codebase behind the deployment discussed in this post, however, is not the original. It's a substantial architectural fork — poppopjmp/spiderfoot — that reimagines SpiderFoot as a microservices platform: a FastAPI backend, Celery-based distributed scanning workers, PostgreSQL in place of SQLite, a React single-page frontend, and — most relevant to this story — a Qdrant vector database and a set of specialised LLM-backed "agents" (report generation, credential risk analysis, threat-intel cross-referencing, and more) sitting on top of the classic scan engine. It's an ambitious modernisation, and it's the version we run.

2. The Fork: Why We Branched Off

We didn't set out to fork anything. The plan was to run poppopjmp/spiderfoot as-is, on our own infrastructure, fronted by our own LLM proxy. The thin-report problem changed that.

Diagnosing and fixing issues at this depth — inside the request-handling code, the database access layer, and the vector-indexing pipeline — isn't something you do safely against a production deployment, and it isn't something you can responsibly hand to a shared upstream project as a drive-by pull request without first proving the fix actually works. So the work happened in three stages: a local, fully containerised development environment where scans could be run and reports regenerated freely without touching anything live; a personal fork of the upstream repository at github.com/braedach/spiderfoot to hold the fixes with full commit history and reasoning attached; and, once each fix was verified end-to-end against a real completed scan, a pull request back to poppopjmp/spiderfoot proposing the same changes upstream.

That last part matters. A fork that never talks back to its source just accumulates drift. Ours is meant to be temporary scaffolding — a place to prove fixes are correct before asking the maintainer to fold them back in — not a permanent parallel project.

3. What Changed Between Them

Here's the thing that made this investigation interesting rather than routine: every one of the nine bugs behind the thin-report problem failed silently. Not one of them threw an error a user would ever see in the UI. Each one degraded gracefully into "return nothing" or "return the wrong thing," which is exactly why a report that should have been rich with real data instead came back reading like a template with the blanks unfilled.

A representative sample, described at the level a non-engineer can follow:

The accessor that didn't exist. The code responsible for fetching a scan's stored events called a method on a service-lookup object — ServiceRegistry.get_instance() — that simply isn't defined anywhere in the class. Calling it raises a Python AttributeError. That error was caught by a generic except Exception block a few lines later and logged at a level nobody was watching, so instead of failing loudly, the export function just returned an empty list. Every downstream consumer of that function — the JSON/CSV export feature, the streaming export, and the AI reporting pipeline — inherited the same silent emptiness.

The database connection nobody wired up. Digging one layer deeper: even with the accessor fixed, the object that's supposed to hold the connection to the scan database is only ever initialised in one specific process role (the dedicated scan-worker process). The API service and the AI-agents service — the two places that actually need to read that data back out for reporting — never call that initialisation step at all. It's not a bug in the traditional sense; it's a piece of application wiring that was correct for one deployment shape and simply never got extended to cover the others.

Two independently wrong column mappings. Raw scan events come back from the database as plain tuples — position 0 is this field, position 1 is that one, and so on. Two different pieces of code guessed at that ordering, and both guessed wrong, in different ways, because neither was checked against the actual SQL query that produces the rows. One of the two "wrong" mappings had even been used as the reference point for fixing the other — a case of trusting a neighbour's homework instead of checking the textbook.

The vector database that was never fed. This was the deepest issue. SpiderFoot's newer architecture includes a component purpose-built to stream scan findings into the Qdrant vector database in real time as a scan runs, so the reporting AI can later run semantic search over everything a scan found. It's a complete, well-written piece of code. It is also never started anywhere in the actual scanning code path — a capability that exists but has no wiring connecting it to anything that runs. The AI agent asking Qdrant for context was, every single time, asking an empty room.

The missing dependency. The component responsible for generating the embeddings that make semantic search possible tries to import a Python package that was never added to the project's dependency list. When the import fails, it quietly falls back to producing meaningless placeholder ("mock") vectors instead of raising an error — so even the parts of the system that were wired together correctly were operating on fake data.

The ID format Qdrant rejected. Once the vector database was actually being written to, it turned out the identifier being used for each stored point — a cryptographic hash from the scan engine — isn't a format Qdrant's API accepts; it requires either a plain integer or a UUID. A one-line fix (deterministically deriving a UUID from the hash) resolved it, but it's the kind of thing that would have silently broken the very component built specifically to solve the problem, had it ever been switched on.

Timeouts sized for the wrong job. The reverse proxy in front of the API, and the browser-side HTTP client behind the dashboard, both had timeouts (30 and 120 seconds) tuned for typical API calls. Generating a genuinely thorough intelligence report by feeding a large language model several thousand words of context is not a typical API call — it can legitimately take several minutes. Every report that took longer than the timeout was recorded as a failure, regardless of whether the model was still working correctly behind the scenes.

Put together, none of these were exotic. They were the ordinary failure modes of software that was built correctly in isolation and never fully connected end-to-end — the kind of thing that only surfaces once someone actually traces a real request through the entire system, from a completed scan sitting in Postgres, through a database read, through vector indexing, through an LLM call, and back out to a browser. Once traced and fixed, a scan of one of my own domains — the same scan, no re-run required — went from a report citing nothing but "N/A" to one correctly citing real autonomous system numbers, real DNS infrastructure, a real SPF policy gap, and a genuinely notable dark-web credential-exposure match, appropriately caveated rather than overstated.

4. Where This Sits in the Broader Story of AI in Security Testing

This project is small, but it's a fairly clean illustration of a pattern showing up across the security industry right now: AI is being folded into penetration testing and OSINT tooling faster than most organisations' processes for validating it are maturing.

The scale of that shift is real. The global AI-assisted penetration testing services market was valued at roughly USD 3.56 billion in 2025 and is projected to keep growing sharply through the end of the decade, and Gartner has forecast that more than 40% of penetration testing activity at large enterprises will incorporate some form of AI-assisted automation by 2027 (ReversingLabs; Intel Market Research). AI tooling has also demonstrably sped up the reporting phase specifically — one analysis put the reduction in time-to-report on mid-scope engagements at around 35%, driven mostly by faster reconnaissance and synthesis (ReversingLabs).

But the industry's own data tells a more cautious second half of the story. The share of organisations relying solely on AI automation for security testing actually fell — from 29% in 2025 down to just 9% in 2026 — as teams ran into the limits of fully automated approaches. Seventy-eight percent of organisations using fully automated AI scanning report that it misses critical vulnerabilities and produces false negatives, and close to half of security teams now deliberately favour a hybrid model that pairs AI-driven testing with human review rather than trusting either one alone (Aikido, State of AI in Pentesting 2026).

That's not a coincidence, and it's not really a surprise once you've watched it happen firsthand. What we found in SpiderFoot's reporting pipeline is a small, contained instance of exactly the failure mode the wider industry is reporting at scale: an AI component that produces fluent, confident, professionally-formatted output regardless of whether the data underneath it is actually there. A large language model asked to write a threat intelligence report will write one — the prompt doesn't fail if the retrieved context is empty, it just gets terser and more generic, and unless someone is checking the substance rather than the shape of the output, that degradation is very easy to miss. We're also not the only ones to have hit rough edges in this specific codebase — an independently maintained fork of the same upstream project exists specifically to patch a different set of self-hosted deployment issues (kolezka/spiderfoottoe), which is a useful reminder that "AI-augmented" and "production-hardened" are not the same claim.

None of this is an argument against using AI in security tooling — the productivity gains are real and the pentesting-as-AI-agent space is developing fast, with more than three dozen distinct tools now competing in that category alone (AppSec Santa, AI Pentesting Agents 2026). It's an argument for treating the AI layer of a security tool with the same scepticism you'd apply to any other unverified data source, and for building — or demanding — the kind of end-to-end verification that catches "the report is technically well-formed but empty" before it reaches an analyst who trusts it at face value.

5. Advantages and Disadvantages of This Work

In the interest of the professional honesty this kind of write-up deserves, here's a balanced look at what forking and fixing this pipeline actually bought us, and what it cost.

Advantages

• Root causes, not symptoms. Every fix traced back to the actual source code and, where possible, the actual SQL — not a guess. Several early attempts to fix a symptom turned out to be built on an incorrect assumption about the underlying data format, which only came to light by checking it against ground truth rather than trusting an existing (but equally wrong) reference implementation elsewhere in the same file.
• Verified against real production data, not a synthetic test case. The proof of success wasn't a unit test — it was the same real scan, against a real domain, producing a materially better report once each fix landed, confirmed on both a local development stack and the actual production deployment.
• Reproducible without touching production. Every one of these fixes was found, applied, and confirmed working in an isolated, fully containerised local environment before a single line changed on the production system. That's the difference between "debugging in production" and doing the debugging safely somewhere else first.
• Given back, not just kept. The fixes were submitted as a pull request to the upstream project rather than silently kept in a private fork — the whole point of open source collaboration is that a fix one person needs is usually a fix several people need.
• Immediate, measurable value. Reports now cite real infrastructure detail instead of placeholders, which is the entire point of running an OSINT platform in the first place. A CTI report an analyst can't act on is worse than no report at all, because it creates false confidence that the domain was actually assessed.

Disadvantages

• Fork maintenance is now a standing cost. Every future upstream change to this project needs to be manually reconciled against the fork until (and unless) the pull request is accepted. That's ongoing overhead, not a one-time cost.
• One fix is a workaround, not a cure. The vector-database indexing gap was patched by having the reporting step pull data on demand at report-generation time, rather than by resurrecting the real-time indexing pipeline the original architecture intended. It works, and it's honestly documented as a stopgap, but it isn't the design the system was originally meant to have — the deeper problem (an unstarted background component) still needs a proper fix.
• New dependencies aren't free. Fixing the missing-embeddings bug meant adding a genuinely heavy machine learning dependency to the build. That's a larger container image and a longer build time, permanently, in exchange for search results that are actually meaningful rather than mathematically fake.
• Narrow scope, by necessity. This investigation covered the reporting and export pipeline specifically, because that's where the symptom was. It did not amount to an audit of a codebase with several hundred modules. Given how many of these nine issues were "dead code that was never exercised end-to-end," it would be naive to assume they're the only ones of their kind still sitting undiscovered elsewhere in the same project.
• Upstream acceptance isn't guaranteed. A pull request is a proposal, not a merge. Until (or unless) the maintainer accepts it, production has to keep tracking a fork rather than the canonical upstream project, which is exactly the ongoing overhead described above.

6. Closing Thoughts

The most useful lesson from this whole exercise isn't really about SpiderFoot specifically — it's a reminder that "the AI produced a report" and "the AI produced a correct report" are two different claims, and only one of them is easy to verify by glancing at the output. As AI components get folded into more and more of the security tooling we rely on — and the market data says that's happening quickly — the discipline that catches the difference between those two claims is going to matter more, not less.

For now: the reports are fixed, the fixes are documented, and they're on their way back upstream where anyone else running this platform can benefit from them too.

References

• Micallef, S. SpiderFoot (original project). github.com/smicallef/spiderfoot
• poppopjmp/spiderfoot (microservices fork used in this deployment). github.com/poppopjmp/spiderfoot
• braedach/spiderfoot (fork containing the fixes described in this post). github.com/braedach/spiderfoot
• ReversingLabs. "Security teams are ditching AI-only penetration testing." reversinglabs.com/blog/automated-ai-pen-testing-out
• Intel Market Research. "AI Penetration Testing Service Market 2026 to 2034." intelmarketresearch.com/ai-penetration-testing-service-market-57810
• Aikido. "State of AI in Pentesting 2026." aikido.dev/state-of-ai-pentesting
• AppSec Santa. "AI Pentesting Agents 2026: The Rise of 39+ Tools Tested." appsecsanta.com/research/ai-pentesting-agents-2026
• kolezka/spiderfoottoe — an independently maintained fork addressing separate self-hosted deployment issues in the same upstream project. github.com/kolezka/spiderfoottoe

Im about to make an arse out of myself. I have seen the original AI reports and I have seen the new reports based on AI work and constant prompting and testing by "us"

To show what I am seeing I run a test based against our modified fork on my own domain. I have included it as "proof of concept" (that is normally so wrong in this field)

I am about to rebuild my images and then update my Portainer images and see what gives.
I have been advised of the following.

No — you don't need to re-scan anything. That's actually one of the nice properties of how tonight's fix works: the backfill reads straight from Postgres on demand, at report-generation time, not from anything captured while the scan was running. As long as a scan's raw events are still sitting in tbl_scan_results on baden, requesting a report for it — old or new — will trigger the same backfill-then-generate flow we just watched work.

61E3AD0E itself proves this: that scan completed before any of tonight's fixes existed, and it still produced the rich report you just reviewed once we asked for it with the fixed code.

Two things worth keeping in mind, not blockers:

1. First report on an old scan is slower — the backfill/embedding pass only happens once per scan (it checks Qdrant first and skips if already indexed), so the first generation for a given old scan pays the CPU-embedding cost; regenerating that same scan's report afterward should be much faster.
2. Only works if the data hasn't been purged — there's a cleanup_expired_results maintenance task in the Celery queue list; if a retention policy has already deleted a very old scan's rows, there's nothing left to backfill from. Anything still visible in the scan list should be fine.

One thing that does not retroactively fix itself: any report you already generated and saved under the old broken code stays exactly as thin as it was — you'd need to explicitly hit "Generate Report" again for that scan to get a fresh one.

Notes:

1. My UDM IDS/IPS lit up big time when running this work
2. You might need to adjust yours to allow this work to be done
3. Just saying

This build is a significant improvement on the original source.

I have given the AI the report I have shared above and we are in agreement. It rocks against the original AI report.

I am about to rebuild the images and will have to jump through a few hoops to get them where I need them and shared the way I need them. I then have to validate the production server which is Podman against a development platform which is docker.

So why are you sharing, if you havent yet pushed from development to production.

Because if you are interested in this field, its days like this make your world "rock"

So, back to Davros and let the build begin. This might hurt - alot.

Crap - I just had a look at the adjusted yml, .env files and I am not fresh enough to deal with this tonight. I still have one project burning in the back ground.

Tomorrow. Post to be updated.

Shut down cleanly — all 10 containers stopped and removed, network gone.

To your assumption: correct, no data loss either way, but the more important point is that nothing here mattered for keeping the actual work safe:

• The fixes themselves are safe regardless — they're committed to braedach/spiderfoot (master, pushed), submitted as PR #393 upstream, and the deployment/tooling changes are merged into Braedach/Code's main. None of that lived in these containers.
• The bind-mounted data (Postgres, Qdrant, MinIO, the embeddings cache — 88MB–199MB worth) is still sitting on disk under volume/ untouched, since we only ran docker compose down without -v. If you want it back later, docker compose up -d from Dev/spiderfoot-dev/ picks up exactly where it left off, same scan history and vector data intact.
• Production on baden was never touched by any of this — that's an entirely separate stack.

Nothing lost either way. Standing by.

Love this shit.

#enoughsaid