MindSwarm
Case study

The hard part of an autonomous agent isn't the agent

Sergej June 17, 2026 ~10 min read

There is a retired phone on my desk — a six-year-old Android with 6 GB of RAM, the kind that gets traded in for €20 or thrown in a drawer. It has no SIM plan worth mentioning and no job anyone would pay for. For the last several weeks it has been running an autonomous AI agent that watches my rental bookings around the clock, with no computer attached, and pings me the moment a reservation lands. The agent works. That is not the interesting part. The interesting part is everything the phone's own operating system did to kill it, and what it actually took to keep it alive.

This is a case study from MindSwarm's own proving ground — the same three-apartment rental in Lithuania I run as a live testbed before any method is offered to a client. I wanted to know a specific thing: when people say "autonomous agent," how much of the work is the intelligence, and how much is everything else? The honest answer, measured on a phone I could afford to brick, is that the intelligence is the cheap part. The expensive part is the part nobody demos.

What the agent actually does

The scope is deliberately small, because a small job you can fully trust beats a big one you cannot. The agent pulls the iCal feeds that Booking.com and Airbnb publish for each apartment, every fifteen minutes. It reads them, works out which dates are now occupied, and at 08:00 sends me a single-line occupancy digest over Telegram. When a new reservation appears between polls, it fires an instant ping — both a Telegram message and a native Android notification with sound. That is the whole product. No dashboard, no app to open, no laptop that has to be awake.

The constraints are what make it real. It must keep running when the house PC is off, when the WiFi drops to cellular, and after the phone reboots itself at 3 a.m. for an update I did not ask for. An agent that needs me to babysit it is not autonomous — it is a demo with extra steps. So the test was never "can a model read a calendar." The test was "can this thing survive a month of normal life unattended." That is where the work lives.

An agent is mostly not the model

Here is the shape of what runs on the phone: Android, then a Linux environment running inside the Android app, then the agent inside that, reachable from the outside through a tunnel and an SSH key. Drawing that takes one sentence. Every layer of it is somewhere the model never touches and the agent can still die.

This is the point people miss when they picture an "AI agent": they picture the reasoning. But the reasoning is a commodity now — you rent it by the token and it is excellent out of the box. What you do not get out of the box is a process that stays alive on hardware actively trying to stop it, authenticates itself without a human present, and brings itself back after a power cut. None of that is intelligence. All of it is the difference between a clever transcript and something you can leave running while you sleep.

The core problem: an autonomous agent is not "a smart model in a loop." The model is maybe a tenth of it. The other nine tenths is keeping the loop alive in a hostile environment — and that is the engineering nobody puts in the demo.

Three ways the operating system kills your agent

A phone is built on the opposite assumption to a server. A server's whole purpose is to keep your process running. A phone's whole purpose is to kill background processes to save battery — and it is very good at it. Three separate mechanisms had to be defeated, each of which silently murders the agent in a different way.

The phantom-process killer

Modern Android ships a "phantom process killer" that hunts down background process trees and terminates them under load — exactly the trees a long-running agent spawns. Left on, it reaps the agent within minutes and nothing tells you why; the logs just stop. It has to be switched off at the system level and the per-process cap raised to effectively infinite. This is not a setting in any menu the owner sees; you reach it through the device's debug shell, and you have to know it exists to even look.

Doze and battery optimization

Android's Doze mode freezes apps the moment the screen is off and the phone is still — which, for a device sitting on a shelf doing one job, is always. The agent's host app has to be added to the battery whitelist, granted permission to run in the background, and made to hold a wake-lock so the OS treats it as a foreground service it must not suspend. Miss any one of these and the agent works perfectly while you are watching it and goes silent the instant you look away. That last detail — works while observed, dies when unobserved — is the signature of every hard production bug.

The reboot that un-installs your autostart

The worst one was specific to the phone's vendor skin. Its "autostart" permission — the thing that lets an app launch after boot — is silently reset to off on every single reboot. Granting it through the normal channel does not survive the next restart. So after any reboot, the boot hook never fired, and the agent was simply gone until I noticed and started it by hand. The only durable fix was to run a small script as root, through the device's own root layer, that launches the agent on boot before the vendor skin gets a chance to undo the permission. The agent now wins a race it did not know it was in, every time the phone restarts.

The agent that couldn't log itself in

Even alive, the agent hit a wall that has nothing to do with calendars: it could not authenticate. Signing it in requires opening a login URL in a browser where you are already signed in — and the phone's agent has no such browser, and no human in front of it. The login flow assumes a person at a screen, and there is no person. The working answer was almost comically physical: the agent emits the login URL, that URL is opened on a PC where I am logged in, the resulting code is read off the screen and typed back into the phone through its debug shell. A one-time human handshake to bootstrap a thing whose entire purpose is to need no human. Knowing where that seam is — where autonomy has to borrow a human for exactly one step — is half of designing an agent that is honest about what it can do alone.

The failure that looked like success: at one point the agent's session was started in the background "to be safe." It came up looking perfectly alive — process running, no errors — but the remote-control channel never attached. A live-looking zombie. The fix was counterintuitive (start it in the foreground, attached), and the lesson was the usual one: a green process is not a working one.

Autonomy needs a leash

There is a tempting next step: the phone agent can reach my PC through a bridge, which means it could drive the desktop, run commands, act on the wider system. The restraint is deliberate. The agent will read, summarize, and alert on its own all day — but it will not execute arbitrary instructions arriving through its message channel, and it will not take an outbound action that touches anyone but me without a human in the loop. That boundary is not a limitation I am apologizing for; it is the design. An autonomous agent with a network reach and no leash is not a feature, it is an incident waiting for a prompt. Deciding precisely what an agent may do unsupervised — and building the wall so it cannot quietly step past it — is as much the job as making it run at all.

The resilience loop, drawn out

Put together, the thing that keeps the agent alive is not a model call. It is a loop of unglamorous recovery machinery, most of it about losing and regaining ground against the operating system:

reboot ──► vendor skin resets "autostart" ──► normal boot hook never fires │ ▼ root-level script (runs past the vendor skin) wait for storage unlock ──► launch agent ──► reconnect tunnel + SSH │ ▼ agent polls Booking + Airbnb iCal (every 15 min) new reservation ──► Telegram ping + native Android notification ▲ │ tunnel down? PC keeper (every 6 min, over USB) restores the bridges │ process reaped? on-device watchdog respawns it └────────────────────────────────────────────────────────────────

Notice that none of the boxes on that diagram are "the AI." They are a root script that beats the vendor skin to the punch, a watchdog that respawns a reaped process, and a keeper on the PC that reaches over a USB cable to restart the bridges when the tunnel dies. The intelligence sits quietly in the middle, doing the easy part. The whole reason it can sit there at all is the boring scaffolding around it.

Why this is the job

The industry is about to deploy agents the way it deployed dashboards — confidently, at scale, and straight into the same wall. The demo always works, because the demo runs for ninety seconds on a clean machine with a human watching. Production is the opposite of every one of those: it runs for months, on infrastructure that fights you, with no one watching, through reboots and dropped connections and credentials that expire. The gap between a working demo and an agent you can actually leave running is filled entirely with the kind of work in this story — and it is exactly the gap where agent projects will die, the same way 88% of AI pilots already die in integration rather than in the model.

A retired phone and a fleet of enterprise agents have the same shape of problem: the model is a commodity, the environment is hostile, the authentication assumes a human who is not there, and the failures are quiet — the agent reports green while doing nothing. The difference is only how many of them there are and how much each silent failure costs. Solving it small first is the entire point of a proving ground: I learned where agents actually break by keeping one alive, unattended, for a month, on hardware I could afford to lose.

MindSwarm is Sergej's independent Forward Deployed AI Engineering practice. The method is the one in this story: treat the model as the easy part, embed close enough to the real environment to find the silent deaths, own the operational plumbing end to end, draw the leash deliberately, and hand back an agent the team can actually leave running. When everyone is selling you the intelligence, the work — and the risk — is everything around it.

Thinking about deploying agents?

The gap between a demo and a thing that runs unattended — in one brief.

Read the brief →