The run history filled the disk

One job, about 14,000 items, a few hours. That was enough to fill a 120 GB disk on a self-hosted workflow server and take the server down.

A team reported this on the project's issue tracker. They ran the tool in Docker, on its default database, SQLite, which keeps a database in one main file. Scheduled clean-up of old runs was on. A custom sync job processed about 14,000 items, and each item started more workflow runs. The stored run data filled the disk. The server stopped answering, and nobody could open the screen.

The team made the disk bigger, 160 GB. They deleted runs in the screen until only 17 were left. The database file was still 101 GB.

The space came back only when the team stopped the container and ran VACUUM by hand. In SQLite, VACUUM rewrites the database file without its empty space. Disk use went from 116 GB to 16 GB. The team reported that no error and no warning came before the disk was full.

A user with a long-running self-hosted server, on the same tool and the same default database, reported a different kind of growth. The database file was 1.8 GB. Next to it, the write-ahead log had grown to about 18.9 GB. The write-ahead log is a side file in which the database records changes before it copies them into the main file.

While the log was that large, the workflow list, the run history and new runs became very slow. A restart cleared it, because the user had set the tool to run VACUUM at every start. Within days or weeks the log grew back. It happened twice, with scheduled clean-up on.

In both cases, clean-up was on and old runs were deleted. The disk still filled, or the server still slowed down. Deleting a row and giving space back to the disk are two different things.

In both cases, nothing warned in advance. The first team found out when the server stopped. The second found out when everything became slow.

Run records are worth keeping: they show what an agent or a workflow did. The agent finished. Nobody can say what it did. describes what a good one holds. But every record takes space, and a large run takes a lot of it.

These checks apply to any self-hosted tool that runs agents or workflows, ours included. Where EpicStaff answers a check, the answer is under it.

1. Know where run records live

A run record holds what went into a run, what each step returned and what came out. Find out which database holds these records, on which disk, and what else shares that disk. In the first report, the database filled a disk that the whole server needed.

In EpicStaff: Run records live in your own PostgreSQL database.

2. Know who can delete old runs, and what a delete leaves behind

Find out which role may delete old runs, and who has the time to do it. Then find out what a delete removes. A run can write files and add to logs. If a delete removes only the record, the rest stays on the disk.

In EpicStaff: An Org Admin can delete old runs one by one or in bulk. Files the runs wrote stay in the file store.

3. Know whether anything deletes on a schedule

Find out whether your tool deletes old runs by itself, what the setting is and who chose it. Compare the schedule with your largest job, not only your daily ones: the 14,000-item job above filled the disk in a few hours while scheduled clean-up was on. If nothing deletes on a schedule, put the clean-up in a calendar and give it an owner. Last, decide how long your auditors and your support team need old run records.

In EpicStaff: Run records stay in your own database. They are deleted on a schedule only if your operator sets a retention period (30 to 365 days); 0 keeps them forever.

4. Know whether deleting gives space back to the disk

In the first report, deleting runs left a 101 GB file, and only VACUUM, run by hand with the tool stopped, gave the space back. In PostgreSQL, space freed by deleted rows is reused for new rows once vacuum has run. It is not given back to the disk unless you run VACUUM FULL.

Test what a delete does on your own server. Write down the size of the database, delete a batch of old runs, and measure again the next day. Then you know the answer before you need the space.

5. Name the person who watches disk and database size

A full disk is easy to see coming, if someone looks. Watch the free space on every disk the server uses, and the size of the database itself. Set a first warning well before the disk is full, and a second one closer to full. Send both to a person who can act on them. On a self-hosted server, this job is yours; for the rest of what that ownership means, see Self-hosted AI: how to own your AI stack.

6. Know what else grows

List every folder and volume on the server that grows with use: logs, files that runs write, and database side files like the write-ahead log in the second report. A third report, about a different tool, comes from reading its source code, not from a full disk: the user reported that each time a builder pushes changes to an agent's skills or files, the old file record and its stored copy stay behind, with no clean-up path. For each item on your list, note where it lives and what removes it.

In EpicStaff: Run records are not the only thing that grows: files, logs, Redis and saved Python environments do too.

7. Measure size, and test a restore, on a schedule

Once a week, write down the size of the database, the file store and the logs. The trend tells you more than any single number. Once a quarter, restore a backup on a spare machine and time it. A database full of old runs takes longer to back up and longer to restore. The test tells you how much longer, before an outage does.

Write down two numbers today: the free space on the disk that holds your run records, and the size of your database. Measure both again in a week. The difference is your weekly growth. Divide the free space by it, and you know how many weeks you have.

Then plan for the large run. Before a job with thousands of items starts, check the free space, and know who will delete its run records afterwards.