The workflow that took everything else down with it

It is Monday, just after eight. A flow that summarises last week's case files starts on its schedule. This week the pile is twice as big. The flow takes more memory, then more, and it does not finish.

At twenty past eight, the intake form on the website stops getting replies. Nobody touched the intake flow. It only shared a server with the big one.

At nine, someone restarts the server. The schedule starts the big flow again, and the morning repeats.

No part was broken. The platform did what it was built to do. The team did not know where its limits were.

One team reported that steps which wait minutes for an outside AI job hit a gateway timeout. They moved the long work out of their workflow tool and into a queue.

One person runs a single instance for several paying clients. He reported that a long voice call for one client, with a loop that kept checking its status, stopped another client's webhook from firing. Later he gave each client its own instance, with a hard memory limit.

Another user reported that one workflow ran out of memory and took his whole hosted instance down for hours. A restart did not help, because the workflow started again. He found no way to switch off that one workflow while the platform could not be reached.

A third team reported that one large run filled a 120 GB disk and took the instance and its screen down. Nothing warned them first. After they deleted almost all old runs, the database file was still large. Only a manual clean-up, with the service stopped, gave the space back.

Four tools, four setups, one pattern. A limit existed. Nobody had written it down.

An AI agent does more than write text. It calls tools, waits for other systems, and holds data in memory while it works (see what an AI agent is). Each of these uses something the server has only so much of. Four limits matter most:

Time: what stops a job that runs too long, and what counts as too long. Shared capacity: what one heavy run takes away from the others. The off switch: how you stop one flow while everything else keeps going. The disk: what grows, and who removes it. These limits live in the harness, the runtime layer your agents run inside, not in the model (more in who audits the harness). Below is how each one works in EpicStaff.

In EpicStaff, each flow has a time limit, set per flow. It counts the time since the run's status last changed, not the total time of the run. The limit can also be switched off for a flow.

Inside a run, there are more limits. Python steps have one timeout for the whole system, which the operator sets. When a Python step reaches it, EpicStaff stops the step and every process that the step started. Each agent can have its own maximum execution time. You can also give a run a token budget.

So "how long may a job run?" has no single answer. Check each layer against your longest step: the flow, the Python step, the agent and the tokens.

In EpicStaff, one heavy run can slow the others. The operator sets how many runs go at once.

Python steps each run in their own process. There is one memory limit shared by all Python steps. There is no memory limit or CPU limit per step.

This is how you plan with it:

Set the number of runs at once to what your server can carry with your heaviest flow, not your average one. Schedule heavy batch work outside the hours when fast flows, such as webhook replies, are busy. If two clients must never affect each other, give each one its own installation. This holds for any tool.

A flow in EpicStaff has no on/off switch of its own. You can switch off its schedule, and you can stop a run that is going. Webhook and Telegram triggers cannot be switched off.

So for each flow that starts from a webhook, decide now where you will stop the calls: at the system that sends them, or at the gateway in front of EpicStaff. Write that next to the flow's name. During an incident, nobody wants to find it out.

EpicStaff has a size limit per file and per upload. It is a size limit the operator can change. Set it to the largest file your flows really need.

EpicStaff does not clean up old runs or files by itself. A run session is removed only when a user deletes it. So plan the clean-up: how long you keep runs, who deletes them, and how often. Then watch the disk on the server, with an alert well before it is full.

Use a test setup of the same size as production. Run these checks on any tool, ours included. EpicStaff's answer stands under each check.

Start your longest job. What stops it if it hangs, and at which layer? EpicStaff: there is a limit at each layer. Each flow has a time limit, set per flow, which counts the time since the status last changed. Python steps have a timeout the operator sets. Each agent can have its own maximum execution time, and a run can have a token budget. Run your heaviest flow, and call a small webhook flow at the same time. Does the small flow slow down? EpicStaff: it can. One heavy run can slow the others. The operator sets how many runs go at once. Measure the reply time with one, two and four heavy runs. Run a Python step that uses a lot of memory next to a normal one. Does the normal one feel it? EpicStaff: it can. There is one memory limit shared by all Python steps, and no memory or CPU limit per step. Each Python step does have a time limit. Switch off one flow while its triggers keep firing. Can you? EpicStaff: partly. You can switch off a schedule and stop a run that is going. Webhook and Telegram triggers cannot be switched off. Test the stop you planned at the sender or the gateway. Upload a file larger than you allow. Is it refused? EpicStaff: yes. There is a size limit per file and per upload, which the operator can change. Look for automatic clean-up of old runs and files. Is there one? EpicStaff: no. A run session is removed only when a user deletes it. Write down who deletes, and when. Fill a test disk to 80 percent. Who gets the alert? EpicStaff: this check is about your server monitoring, not the tool. Set the alert there, and confirm who receives it.

Put the seven answers on one page, and keep it with the runbook. Add the number of runs at once that your server carried in check 2.

Next Monday, the big flow will run again. This time, you will know what it can take with it, and where to stop it.