Schedulers & Invoke API Triggers

One record type fires both an Agent and a Multi-Agent Team. A single field says which, and it is the only thing that differs between the two — so one model, one dispatcher and one screen serve both, and neither the agent nor the team document is touched by being scheduled.

Five, and they are not interchangeable — the difference between them is who stopped it and why, which is exactly what someone looking at a silent schedule needs to know.

• active due to fire
• paused stopped by a person
• completed ran out of occurrences
• error auto-paused after failures
• disabled its target was deleted

One index carries the whole design: status together with the next-run instant. It is what the dispatcher's claim query hits, once per tenant per tick, and an idle tenant therefore costs one indexed miss and nothing else. The rest are listing support.

All the cadence maths lives in one pure module. No database, no framework, and — importantly — no implicit clock: every entry point takes the reference instant as an argument. That one decision is what makes the whole of scheduling testable without waiting for real time to pass, and it is why the interface can show you a live "next five runs" preview computed by exactly the code the dispatcher runs, rather than a second implementation that might disagree with it.

Five units, each with its own allowed interval range. The five-minute floor on minute cadences is a product rule, and it is enforced in two places — the API and the schema — on purpose, because the API is exposed to tenants and a limit that lives only in the interface is not a limit.

Sub-hourly cadences are rolling: the next instant is derived from the slot that just ran, never from the moment the run finished, so a slow run cannot make an every-fifteen-minutes schedule drift later and later. Calendar cadences instead hold wall-clock intent — "09:00 daily" is stored as an intent plus a zone, and the actual instant is worked out at compute time.

For intervals greater than one, the phase is anchored to the schedule's creation instant. "Every three days" means every third day counted from when you created it, not every third day of an arbitrary epoch — so the same recurrence created on different days produces different, predictable days.

Zones are resolved with the platform's own internationalisation data rather than a date library. A schedule stores wall-clock intent and an IANA zone; the instant is derived at compute time, so a daylight-saving shift is absorbed automatically and "09:00" stays 09:00 on both sides of it.

"The 31st" has no honest answer in February, so the schedule says which answer it wants: clamp moves it to the last day of the month — the default, and the way most people read it — or skip passes over short months entirely. A dedicated "last day of the month" setting tracks month length on its own and needs neither.

Both calendar walks are probe-limited — four hundred days, twenty-six months. A recurrence that can never be satisfied returns no occurrence rather than becoming an unbounded loop inside the dispatcher's tick.

A background loop that wakes every thirty seconds, walks the tenants, and claims whatever is due. It starts after the HTTP server is already listening, so a slow tenant handshake can never delay the application becoming ready, and it can be switched off entirely on instances that should only serve traffic.

Nothing is read, computed or fetched until a schedule is actually due. The claim is a single indexed query, the tenant list is cached for a minute, and an idle tenant costs one index miss per tick and zero tokens. This matters because the loop runs forever, in every deployment, whether or not anybody uses scheduling.

One atomic update both claims the schedule and stamps its lease. It has to be one write: reading the timeout first and extending the lease afterwards would leave a window in which a long run is protected by a short lease. The lease is therefore computed inside the database, from the schedule's own fields, against the database's clock — which also removes any assumption that two competing instances agree about what time it is.

A claimed schedule is still not certain to execute. It is dropped if the current instant falls outside its active window, and dropped if the slot is stale enough to have been missed while the service was unavailable. Either way the schedule is realigned and released, and a skipped run is written to the history — because a gap that is visible is a support question, and a gap that is silent is a bug report.

There is a small but telling detail in the realignment: when a schedule is claimed before its window has opened, the next instant is computed from the window's start rather than from now. Computing from now would re-claim and re-skip the same job on every single tick until the window finally opened.

Two independent mechanisms, because a crash leaves two different messes. The schedules come back on their own when their lock leases expire — nothing has to notice. The run records left marked as still-running are closed by a sweep that runs once per database connection per process, so the history does not accumulate rows that will never finish.

On an orderly shutdown the loop stops claiming, waits briefly for in-flight runs to finish and release their own locks, and hands whatever is left to the lease.

Schedules live in each tenant's own database. A standalone deployment keeps them on the master connection instead — but only when there are genuinely no tenants, because polling master in a multi-tenant install would create the collection and its indexes on a database that will never hold a schedule, and then poll it forever.

A claimed schedule holds its lock for as long as the run could possibly take — not for one attempt. Under-sizing the lease is precisely what lets a second instance claim a schedule that is still executing, so the budget has to cover the whole retry envelope: every attempt's timeout, plus every backoff between them, plus headroom.

Once in application code, for the heartbeat that renews the lease while a run is in flight. Once as a database expression, so the dispatcher can derive the lease from the schedule's own fields inside the atomic claim. They have to stay identical — so they live side by side in one small file, and a test asserts that the two agree across a matrix of specifications rather than trusting that a future edit will touch both.

While a run is executing, the lease is pushed out periodically — at a third of its own length. Without this, a run that retries, or that waits for a slot in the executor's concurrency limiter, can outlive its lease and be claimed a second time.

The renewal is guarded on ownership. If this runner's lease genuinely did lapse and another instance legitimately took the schedule, the renewal matches nothing and the lock is not stolen back. A heartbeat that renewed unconditionally would be worse than no heartbeat at all.

Executes one claimed schedule, and owns it from that moment on. Its contract with the dispatcher is absolute: whatever happens, it must advance the cadence and clear the lock. A leaked lock silently stops a customer's schedule until the lease expires, so the release lives in a block that cannot throw — and if even the final write fails, there is a last-resort attempt that does nothing but take the lock off.

1. Resolve the target. Gone? Disable the schedule rather than fail every slot forever.
2. Quota gate. Over budget is skipped, not failed — nothing is broken.
3. Open a run record marked running, and start the lease heartbeat.
4. Invoke under a hard timeout, retrying per the schedule's policy.
5. Close the run record with status, duration, usage, trace id and a response preview.
6. Advance the cadence and release the lock in one write.
7. Report telemetry — best effort, never allowed to affect the outcome.

A schedule either opens a fresh conversation per run — the default, and the right choice for a report — or appends every run to one fixed session, which is what you want when the agent should remember what it said last time.

The query supports a small set of substitutions, resolved against the slot rather than the wall clock, so a run that started late still asks about the moment it was scheduled for:

• now the slot, in full
• date the slot's calendar date
• lastRunAt the previous run
• runCount how many so far

Substitution is skipped entirely when the query contains no placeholder at all — the common case gets no scan and no allocation.

Five consecutive failures and the schedule stops itself, with status error and the last failure recorded. A broken schedule must not keep spending the tenant's model budget on a timer. Resuming clears the counter and realigns from now — a long pause does not produce a burst of catch-up runs the moment it is lifted.

Skipped slots are excluded from all of this. A skipped slot never reached a model, so it does not count toward the run total, does not move the failure streak, and cannot trip the auto-pause.

A multi-agent team is checked for graph validity before it is invoked. An unrunnable team fails with a diagnostic summary rather than an opaque executor error — which, on an unattended run nobody is watching, is the difference between a fixable report and a mystery.

Every execution writes an authoritative record: which slot it belonged to, when it actually started, how long it took, its outcome, the session and message it produced, a trace id that deep-links into the existing log viewer, token usage, and a truncated preview of the answer. The full answer lives in the session and the trace log — the preview exists so the runs list can show something useful without loading it.

The test my schedule button fires immediately without disturbing the cadence. It still claims the lock exactly as the dispatcher does — so a manual run can collide neither with a scheduled one nor with a second impatient click. Each of those would be a real, billable duplicate run. The request answers straight away with an accepted status and the client polls the run list, because a run can take minutes.

Editing a schedule while a run is in progress is refused for the same reason: mutating a job the dispatcher is already executing is not worth the race it creates, and the run finishes in seconds.

A scheduled run spends the tenant's budget exactly like an API call does, so it answers to exactly the same limits. Two gates run before any model is touched: the target's own daily cap, and — when the schedule names an invoking identity — that person's per-minute invoke budget.

Both gates fail open, consistent with the rate limiter itself: a counter outage must not silently stop every schedule in the tenant. And being over quota produces a skipped run, never a failure — so a busy day cannot push a perfectly healthy schedule toward auto-pause.

Scheduling brings its own permission check rather than reusing the application's general authorisation middleware, and the reason is stated plainly in the code: that middleware currently short-circuits before any check — a documented, project-wide temporary bypass — so applying it here would enforce nothing. Changing it would alter behaviour on every route in the application, which is not this feature's call to make. It matters more here than on an ordinary screen: a schedule spends the tenant's model budget on a timer, with nobody watching.

Validation is strict about one field in particular. The invoking email is the rate-limit subject, so an arbitrary string would hand out a private quota bucket, and a typo would silently detach the schedule from the limits of the person it is meant to run as.

The external trigger. Two endpoints — one for an agent, one for a team — each authenticated by an API key that is bound to exactly one target. Calling a team key on the agent endpoint gets an explicit message pointing at the right one, rather than a generic authentication failure.

The key itself says which database to talk to. A client key carries the client's identifier in the middle of it, so the tenant connection is resolved from the key before anything is validated against it; a system key goes to the master connection. This is what lets a single endpoint serve every tenant without the caller supplying routing information, and without a shared lookup table in front of it.

A key identifies an integration; it does not identify a person. Since rate limits are per-person, the caller must supply and the endpoint must validate an email — so budgets follow the human rather than the key they happen to be calling with. The same identity is then tracked as an API session, which is upserted rather than duplicated, and whose failure is non-fatal: losing the session record must not lose the run.

The per-minute invoke limit is consumed up front — and a rejected request has its increment rolled back, so a call that was refused never eats budget it was not served for. The daily model-call budget is only peeked up front, because the real number of model calls is not known until the run is over.

An invoke consults the same semantic cache as chat, scoped to the session and to the fingerprints of whatever assets the query references — but at a stricter similarity threshold than chat uses, which is the right trade for a machine-to-machine caller that cannot glance at an answer and notice it is about the wrong thing. A cache hit is still recorded as a full telemetry event, flagged as a hit and carrying the score, so cache effectiveness is measurable rather than invisible. A lookup failure is logged and skipped — never fatal.

If the HTTP connection drops, the run is cancelled rather than left to complete for a caller that has gone. Every invocation — cached, successful or failed — is attributed to the invoking identity, tagged with its source and its API key, so automated traffic can be told apart from people in the dashboards.