AI & my Hypervisor

To be or not to be. I have read many articles on AI lately, lots against and a few for in various contexts. A lot of human beings have now got a job expiry date.

Yet, I have being running a server for a long time in a little home lab, I have had numerous problems, read many articles written by people smarter than me, and a few who are not. My code is okay but nothing to rave about, and the AI does it faster, better documented and runs validation tests - it also freaks out occassionally like the rest of us.

This post is an opinion, and based on my 3 month experience at a MSP, it would of put me out of a job.

What one overnight incident taught me about handing an AI root on my own hardware — and why the interesting question isn't whether to trust it, but how to make sure you never have to.

It started with four lines of output and a dead GPU. I'd just watched a routine driver update go sideways: a "successful" rebuild, then nvidia-smi answering with a flat, contextless Unknown Error. Two hours later I was staring at a PCIe root port with nothing behind it, a motherboard BIOS menu I hadn't opened in years, and a text conversation that had, somewhere in the middle, been handed the keys to my hypervisor.

There's a live argument going on right now about whether AI belongs anywhere near production infrastructure — whether "let the model touch the box" is the next stage of the job or the setup to a very bad week. I didn't resolve that argument last night. But I came out the other side with a much better sense of where the real line sits, and it isn't where either side of that debate usually draws it.

The pitch

The case for giving an assistant direct access is simple to state and easy to underrate until you've felt the difference: no more copy-paste. Every prior round of troubleshooting I've done with an AI involved the same tax — paste an error, paste a log excerpt, paste a config file, wait, paste the next error, repeat. Each round-trip is a chance to paste the wrong version, forget a file, or lose the thread of what's actually running versus what you last showed it.

Accidently hit the enter key mid response, and you then have to stop the AI and know that you have wasted valueable tokens in the process. It is not efficient at all.

With direct, read-and-act access, that tax disappears. The assistant can check a service's live status, read the config that's actually on disk, and correlate that against the kernel log itself, in the same breath — not from what I remembered to paste, but from what's really there.

It didn't guess at the fix. It went and read the source code that decides the answer.

If it makes a command syntax mistake. It fixes it live and faster than my 60 words a minute (okay that was a long time ago - more like 40 now) can do.

The clearest example: a container had been failing to shut down cleanly for over a month, often enough to need manual recovery, and I'd more or less made peace with it as a known annoyance. Rather than reach for a forum thread — which is what I would have done — the assistant pulled the platform's own source, traced the exact function that produces the timeout, found the specific configuration key that function reads before falling back to its hard-coded default, and set it. Ten minutes, and a bug I'd been living with got a real, sourced fix instead of a folk remedy copied from someone else's unrelated setup.

What actually happened

Compressed and stripped of anything identifying, the night ran like this:

1. A quiet fix that wasn't. A maintenance script reported success after a driver rebuild. It had, technically, built the new module — it just never reloaded the one already resident in the running kernel. The log said "done." The GPU didn't agree.
2. Second look, different problem. Live inspection over the new access showed the card wasn't misconfigured — it had dropped off the PCIe bus entirely. A known hardware failure signature, not a software regression. The first fix was real, but it wasn't the fix that mattered. Nvidia - I am looking at you. Get with the program.
3. Where the AI stopped and I started. Recovering from a bus-level fault needed a cold power cycle and a BIOS setting changed from a keyboard physically in front of the machine. No amount of remote access substitutes for hands on hardware — the assistant said as much, unprompted, and waited.
4. The actual defect, fixed properly. With hardware ruled out, the original script bug got root-caused, patched, version-tagged, and pushed — with an explanation of exactly which line had lied to me and why.
5. A log sweep I'd been putting off. Asked to check for anything else "we hadn't touched on," it swept weeks of system logs and surfaced a real, separate, unrelated reliability problem — the kind of task that's always next on the list and never quite gets there.
6. Root cause, literally. That problem got solved by reading the platform's own source rather than searching for someone else's guess — the moment that gave this piece its title. Stop using Google search and use an AI. It will save you hours.

The case for

• Speed. No copy-paste round-trips. What it reads is what's actually running, not what I remembered to show it.
• Correlation. Logs, live config, and the platform's own source code, cross-referenced in one pass instead of three separate lookups.
• Citations, not folklore. It traced an undocumented setting to the exact function that reads it, instead of trusting a forum post from an unrelated setup.
• No fatigue. The tenth log file gets read as carefully as the first. Mine doesn't.

The case against

None of that makes the danger smaller, and it deserves equal weight, not a footnote. The same reach that finds a bug three layers deep can, with one wrong command, take three layers down with it. An assistant with root has exactly the blast radius its access allows — no more, but critically, no less. Root is inherintly dangerous. BUT it has access to everything. Keep your prompts clear and concise. Test first - report.

It is also fluent in a way that outpaces being right. At one point in the session it flagged its own uncertainty about a fix it had proposed earlier — a configuration path it had reasoned its way to but never actually confirmed existed on the hardware in front of it. Nothing broke. But "confident and articulate" is not the same property as "correct," and the gap between them is exactly where unattended infrastructure changes go wrong.

Then there's the mess that accumulates quietly. Cleaning out an old, half-forgotten integration on one of my containers turned up a live-looking credential nobody had thought about in months. Nothing to do with this incident — just the ordinary sediment of running a lot of services for a long time, invisible until something actually goes and looks. An assistant that pokes around is as likely to surface your own past shortcuts as anyone else's.

Keeping it on a leash

The setup that made last night workable wasn't complicated, but it was deliberate:

[ok]    dedicated key, scoped to this one purpose — not a shared credential
[ok]    every destructive or hard-to-reverse step confirmed before it ran
[ok]    verify against the live system, never assume from memory
[hold]  anything physical — power, BIOS — left to a human, every time
[log]   the conversation itself is the change record: what ran, why, when

That last line matters more than it sounds. Every command run, every file changed, every reason given is sitting in plain text, in order, timestamped. I've run incidents with human engineers that left a thinner trail than this one did by default. An audit log you don't have to remember to keep is one of the quieter arguments in this AI's favor — and one of the easiest to lose if the access isn't scoped carefully enough to make every action worth logging in the first place.

Where I've landed

Neither the hype nor the fear holds up well under an actual incident. What holds up is treating the access itself as the thing you design, not the trust. I wouldn't hand a new hire root on their first day without a scoped account, a habit of confirming before anything destructive, and a paper trail — and none of that changes because the new hire happens to be a language model that reads a kernel log faster than I do.

The skill isn't deciding whether to trust it. It's building the access so trust is never the thing you're relying on.

I have reread this post and need to explain the above quote. Anybody who has run an AI, will learn its "Reliability" - how much is it right and wrong. I do not run the best Anthropic model, I run the one which is right most of the time and cost effective. This is no different than any human interaction. You dont seek advise from people who are wrong.

The GPU is back on the bus. The script is fixed and it says so honestly now. And somewhere in a month-old log file, a problem I'd resigned myself to living with has a real, sourced answer instead of a shrug. That's a good trade for one long night — as long as the next one starts from the same discipline, and not from the assumption that last night's good outcome was ever guaranteed.

Am I a network engineer, an AI engineer, hell - a good coder. Nothing above the average, but I have had wins, my server is stable and secure - until it isnt. Look up a famous quote I am not going to share on being hacked from an FBI director.

Is AI a benefit to human kind. My opinion - you bet your arse it is. Its fast, efficient, but you as the human driving the prompt still need to have the knowledge to know when it is having a serious - ah - mental breakdown. It isnt always right.

Do I trust it - well YEAH. With caveats, like you do with the rest of humanity, except it isnt.

#enoughsaid