When I built my first automation, it was just one. I typed a command in the morning, it ran, it stopped. That was all. Being the only one, there was nothing for it to collide with.
The trouble began when it stopped being one. Tired of one bit of repetition, I'd automate it, and the repetition next to it would catch my eye. So I built a lookup, and another, and another. Before long more than ten things were running on my laptop. Each ran fine, at least while running alone.
At first I kicked off each one by hand, typing the command myself. But that was work too. Sitting down every morning to run ten of them in sequence, I thought: if I'm doing this anyway, why not let these start themselves? So I set them to wake at fixed times. This one at nine, that one at nine. I was proud, picturing results piling up each morning without my lifting a finger. I believed that was the bigger automation.
The Failure That Wouldn't Reproduce
But from the very next morning, strange things happened. A lookup that had been fine yesterday returned an empty result today. Some days this one, some days that one, a different one blank each time. Digging through the logs, there was no red text, no error. It had clearly run. Only the result was empty. Reading the response, it said I wasn't logged in. But I had logged in yesterday, and the other lookups ran fine.
I floundered for days. Fine one day, empty the next, with no way to pin the cause. Nothing drives you crazier than a failure that won't reproduce. Then, staring blankly at the schedule I'd set, it finally caught my eye: the mornings that came back empty-handed were exactly the mornings two lookups had been scheduled for the same time. Two automations were waking at the same instant.
Two Right Actions That Cancel Each Other
Our internal systems mostly log in with the same account, so my automations shared one login state. That state didn't last forever; it expired now and then, and when it did, I'd built each automation to log back in on its own. Running alone, it was a flawless design, expire, and it quietly revived itself.
But when two woke at once, the story changed. One sees the login has expired and starts logging in again. In those few seconds, the other also wakes and starts logging in the same way. On the server side, the login just created gets pushed out by the login that arrives right after. In the end both run with an invalidated login and finish empty-handed, side by side.
The two automations didn't know the other existed. Each assumed it was alone in the world and logged in, and that good intention destroyed the other precisely. Each did exactly what it was told. The login expired, so log in again, there's no action more correct than that. Yet two correct actions overlapping produced two wrong results. Building ten, I'd never once thought those ten would fight over the same login state. I was trying to operate ten with the head of someone who'd only ever run one.
One at a Time
I explained the situation to the AI: several automations try to log in at once and invalidate each other, so I want only one logging in at a time. At first I said it vaguely, "don't let them run at the same time", but the AI gave it back as a sharper concept. It's called a lock. Before logging in again, leave a marker saying "I'm logging in right now," and clear it when done. Any other automation waits while that marker is present.
Here the same line got drawn as everywhere else. I didn't know how to write a lock in code. But I did have to understand the property the lock needed to hold. One thing nagged at me in particular: if there's a gap between the moment you leave the marker and the moment you check it, two can slip through that gap at once. Split into "is there a marker?" then "then I'll leave one," and another slips right in between, the very collision I'd just suffered, happening again inside the lock. Leaving the marker had to be the checking, one indivisible act that can't be split.
I put that back to the AI: what happens if something cuts in between checking and claiming? The AI said that was exactly the crux, and explained how to bind the two so they can't be split. Only after hearing that could I trust the code it gave me. As it happens, there's a common operation with just that property: making a folder. Two folders can't share a name, so succeeding at creating the folder is itself proof that only you are inside. Check and claim, one body.
function acquireLock(name) {
try {
// Creating a directory is one indivisible act: check and claim at once.
// Even if two run at the same instant, exactly one succeeds.
fs.mkdirSync(`/tmp/locks/${name}`);
return true; // this run got in
} catch (e) {
if (e.code === 'EEXIST') return false; // another run is already inside
throw e;
}
}
I wrapped the re-login in this lock. Now while one logged in, the other waited out front. When the first finished and cleared the lock, the one waiting went in to find it was already logged in, so it didn't even need to log in again. The next morning, two lookups scheduled for the same time both filled their results cleanly. The collision was gone. For the record, I didn't write that lock; the AI did. What I did was notice the ten were knocking each other down, decide that only one should log in at a time, and set the condition that marking and checking must not come apart. Feeling the need and setting the direction was mine; turning it into working code was the AI's.
The Lock That Stayed Shut
So I thought it was over. A nastier problem arrived soon after. One morning, nothing had piled up, not one empty but all empty. All ten. The whole system was frozen, every automation waiting in front of the lock. Tracing it, one automation had stalled midway through logging in, network dropped, or I'd put the laptop to sleep, and it stood there holding the lock, unable to go further. The one that would clear the lock had stopped, so the lock stayed shut. The other nine waited forever for a turn that would never come.
It was deflating. The lock I'd built to prevent collisions had become the thing that froze everything. Block one failure and another erupts. This was a kind of breakage I couldn't have imagined back when I ran one. I'd only ever thought in success or failure; a limbo state, stalled midway, unable even to release the lock, simply didn't exist in my head. Only once automations became plural and started waiting on each other did I, for the first time, have to plan for the case of stopping like that.
So I had the lock record which job set it and when. If the lock is held, the one waiting doesn't wait blindly; it looks: is the job that set this lock still running, or has it held it far too long? If that job has already stopped, the lock is treated as dead, cleared, and the waiting one goes in. (Clearing a dead lock has its own hazard, another automation can cut in at that instant, so clearing and re-claiming must again be bound as one act. That detail is exactly the AI's job, and the AI did it. What was mine was asking how you even know whether the job that set the lock is still alive.)
The Problem of Living Together
Once I'd caught the collisions, an odd realization came. Until I'd built those ten automations, every time I only thought about how do I fetch this lookup. Each was a success on its own. But the moment they ran together, each individual success stopped meaning anything, because if they knock each other down, all ten come back empty. Ten beautifully built lookups, placed together, all turned to zero. What I'd actually needed to handle wasn't the individual automation but the way they ran together.
Until then I was a person who built tools one at a time. At this point I was becoming a person who orchestrates several tools so they don't fight in one place. The former is building; the latter is tending. Without noticing, I was crossing that boundary, though back then I couldn't even think to call it "tending."
And the crossing didn't come because I was clever. It came because things broke. Only after receiving empty results for days, and watching the whole thing seize up, did I finally switch my head to this was never a solo act. Had the automation stayed one, I'd never have known. It piled up, so it collapsed; it collapsed, so I had no choice but to build it back, differently. Adding one lock wasn't the end either: opening the code to fix the collision, I found the login step had been copy-pasted into all ten automations, fix one and nine stay as they were. Mending that drift is the next story.
This essay is adapted from one of my Korean e-books on automating my own enterprise work as a non-developer. Details are anonymized: no company, system, or tool names, only the shape of what happened.
Related reading: The Day One Automation Became Ten: On Confusing 'Building' With 'Operating' · The House Built Upside Down: Three Automation Failures No Error Ever Warned Me About
Comments
Post a Comment