# Vigil: the fan-out stall watch

> crystl watches a fan-out for you. When the lead and every worker have gone quiet, it nudges the lead once or twice, and if nothing moves it raises one notification card and stops.

**An orchestrator watches its workers. Nothing watches the orchestrator.** A lead can end a turn waiting on a decision that was already made, its workers can sit waiting on the lead, and from outside a stopped fan-out and a working fan-out look identical. No error, nothing raising its hand.

Vigil is the part of crystl that notices. It watches a lead and the workers that lead spawned, and when the whole set has been quiet for long enough it nudges the lead. If the lead still does not move, vigil raises one [notification card](/docs/notifications/) and goes silent.

> **On by default.** See [turning it off](#turning-it-off) below.

## Turning it off

Settings → Agents → defaults has a VIGIL toggle, on by default, plus a picker for how many nudges it sends before carding you (0–10, default 2 — 0 means "card on the first settled quiet spell, never nudge"). Turning it off is live: every watch retires on its next tick, within a minute. Turning it back on from there arms every open gem immediately.

From a terminal, the same flag:

```bash
defaults write com.crystl.app crystlVigilEnabled -bool false
```

With the flag off nothing exists. No timer starts, no fan-out is watched, and nothing is sampled. A raw `defaults write … true` (bypassing the toggle) takes effect at the next gem open or app relaunch rather than live.

## What vigil watches

**A fan-out, not a gem.** Membership comes from spawn lineage: a lead, plus every shard spawned beneath it, and every shard beneath those. It is never "the shards in this window."

That distinction is the whole feature. An earlier prototype treated every shard in the gem as one set, so an unrelated worker typing away masked a fan-out that had been dead for an hour. Worse, the shard most often busy in that gem was the supervising agent itself, so the watch was blind exactly when it was needed.

What follows from lineage:

- **A shard you opened yourself is not watched.** It has no lead and no workers, so there is no fan-out.
- **A lead with no recorded workers is not watched.** Lineage records only spawns that went through `shard create`, so a real fan-out can be bigger than its recorded lineage. Nudging a lead whose unrecorded workers are busy would be a false alarm, so crystl does not watch that lead at all.
- **A spawn with no task is not delegation.** `crystl shard create` without `--prompt` opens a shard; it hands nobody work. That shard carries no edge up to whoever opened it.
- **A member must be an agent shard.** A plain shell is quiet by nature and has nobody to talk to, so it is in no fan-out. A shard that had an agent and lost it stays a member, and its death is reported to you.
- **A nested fan-out is watched on its own.** A worker that spawns its own workers is a member of the outer set and the lead of its own. Both are watched, so a stalled sub-tree is noticed even while the outer lead is working.
- **A dead lead dissolves the fan-out.** Its workers hold an edge to a shard that no longer exists, and no nudge can ever reach them. crystl tells you once and stops watching.

## What counts as stalled

**Every member is non-working, and the newest output across all of them is older than 15 minutes.** Both halves are required.

A lead idle while a worker runs is not a stall. That is what delegation looks like, and it is never nudged.

The numbers, as shipped:

| | |
|---|---|
| Quiet threshold | 15 minutes since the newest output from any member |
| Confirming observations | 3 consecutive quiet checks (checks run once a minute) |
| Nudges per quiet spell | 2 |
| Wait after a nudge | 10 minutes before judging whether anything moved |
| Grace after a gem opens | 5 minutes before any verdict |

So a fan-out that goes quiet and stays quiet gets its first nudge after roughly 18 minutes, its second about 10 minutes later, and produces one card at roughly 38 minutes, at the default of 2 nudges. The nudge count is a setting (Settings → Agents → defaults → VIGIL, 0–10); set it to 0 and vigil cards on the first settled quiet spell and never nudges at all.

Fifteen minutes is a starting value, not a derived one. A worker running a clean rebuild or a long silent test harness prints nothing and reads as idle the whole time, so crystl leans toward firing late. A missed stall costs you some latency. A false nudge costs an interruption and some trust.

Three safeguards keep the late direction honest:

- **A busy descendant vetoes the verdict.** An agent that launches a long command sits near zero CPU while its child process does the work. When crystl can see that child burning CPU, the fan-out is working, whatever the agent process says. The signal is a veto only: seeing it means working, not seeing it means nothing (it is never available on remote or SSH shards).
- **A hookless agent vetoes too.** Some agents (aider, opencode, goose, and others without crystl's hook integration) give crystl no way to see a tool call in flight. While that agent's process is alive, silent I/O and a finished turn look identical from outside, so crystl leans toward under-firing rather than guessing.
- **"I cannot tell" takes no action, ever.** A missing observation, an unreadable lineage graph, or a fan-out where nobody has printed anything at all produces an explicit indeterminate result. It is logged and nothing happens. A supervisor that answers "I cannot tell" by typing in someone's terminal is worse than no supervisor.

## The nudge

**Only the lead is nudged. Workers are never messaged.** The lead owns its workers, and going around it produces two supervisors disagreeing.

The nudge is a report, in crystl's own words:

> [crystl vigil] no output from the fan-out for 18 min, and no worker of yours is running. If you are waiting on something, check whether it already arrived (`crystl shards`, `crystl wait done`). If the fan-out is finished, run `crystl vigil done "<one line on what it did>"` — that ends this watch AND tells the user, and a reply here does neither.

It never says what the work should do next. Vigil detects and tells. It does not decide, and there is no field for custom nudge text, deliberately: free text turns an unattended watcher into a 3am dispatcher.

Some things stop a nudge before it is typed:

- **A crown is never nudged.** Agent messages into a [crown](/docs/court/) are refused by design, so a crown-led set nudges the hand instead, or nobody.
- **The target must have a live agent right now.** `crystl send` types into the terminal, so prose delivered to a shard whose agent has died back to a shell would sit on a command line one Enter from running. crystl requires a positively detected agent, not merely the absence of a death.
- **A nudge that would be held is not a nudge.** If you have unsent text sitting in the lead's input, or its agent is relaunching, the line would wait instead of arriving. crystl treats that as undeliverable and tells you instead. Unsent text also means a person is already at that shard.

An undeliverable nudge does not spend an attempt. It escalates.

## When crystl tells you instead

**Anything that needs a person outranks a nudge.** These raise a card and skip the nudge entirely:

- A member is parked on its agent's own approval prompt (`⏸ awaiting input`).
- A member's credentials have expired.
- A member's agent is gone.
- There is nobody the nudge can reach (a crown with no hand, a target with no live agent).

**A rate limit is a clock, not a person.** A time-blocked lead is never nudged, because a keypress cannot open a closed lane. Instead crystl informs you once, after a 30-minute dwell, and names the time the lane reopens. A worker that is merely rate-limited while the lead can re-staff around it stays an ordinary stall for the lead to handle.

**Escalation terminates.** One card per stall, then silence until the fan-out actually moves. A supervisor that re-cards every minute is a supervisor people mute.

What counts as moving is deliberately narrow: output from a member other than the one that was nudged, or that member going positively busy. The lead answering the nudge conversationally is not progress. Without that rule, a finished-but-open fan-out would answer every nudge forever, at one agent turn of tokens per cycle.

## Calling off a watch

A lead that knows its fan-out is finished can say so from inside its own shard:

```bash
crystl vigil done
```

Vigil stops watching that fan-out. Spawning another worker re-arms it, and closing the gem clears it as before. `done`, not `off`: it declares the work finished, not the feature disabled.

Only the lead of a roster may call off its own watch. A worker that tries is refused and told which shard may, the same shape as the crown refusal. A sub-lead retires its own fan-out without touching the outer one.

Within a single quiet spell the verb saves at most two nudges and one card. Its real value is for a long-lived lead that runs batch after batch: each batch is its own quiet spell, and without the verb such a lead cards once per batch, all day.

## Keeping a shard out

Some workers are meant to be quiet. Keep one out of every fan-out at spawn time:

```bash
crystl shard create --gem myapp --prompt "run the full suite" --no-vigil
```

`--no-vigil` is a property of that **shard**, in both roles: it counts toward no lead's fan-out, and it is never a lead itself. Exclude every worker in a set and there is no fan-out left to watch. Because it is the lineage edge that disappears, the branch beneath an excluded sub-lead leaves the outer set with it. The stamp survives a relaunch, so a deliberately quiet grinder does not quietly come back under the watch.

`--vigil` is the other half: include a shard where a default rule would leave it out, such as a spawn with no task. It does not override the agent rule, because vigil's one action is typing at a shard.

## What vigil does not do

- **It does not catch a stranded member inside a live fan-out.** If one worker sits idle for an hour while two siblings work, the fan-out reads as healthy, and that is correct rather than an oversight. Lineage says who exists; it never says who still owes work. An idle worker is either stranded or finished and waiting to be integrated, which is the normal end state of every healthy worker. crystl cannot tell those apart, and an alarm that fired on the second would be an alarm nobody reads. The stranded worker's supervisor is its lead, which is alive by definition (otherwise the whole-fan-out rule fires), backed by `crystl wait done` and the rule that quiet is not done.
- **It does not restart anything.** Detect and tell. Automatic recovery of a stalled agent is a much bigger promise and a much worse failure when it is wrong.
- **It does not nudge workers.** Only the lead.
- **It does not cross gems.** One gem, one fan-out at a time.

## Stalled and stale are different words

crystl already flags a **stale** shard: one that is *working* and has gone silent. That stays per shard.

**Stalled** is per fan-out, and requires every member to be *non-working*. The two cannot both be true of the same thing, since one working member makes the fan-out not stalled, and a working member is the only kind that can be stale. Both suppress on a time block.

## Tiers

Being told is free. Notification cards land on the desktop and the phone on every tier, because the paid capability is running a fan-out, never hearing about it.

`crystl vigil done` is a [Guild](/pricing/) control command, like `crystl shard create` and `crystl send`. That is consistent rather than restrictive: a CLI fan-out is Guild already, so a lead in a position to call off its own watch has a membership by construction.

---
Source: https://crystl.dev/docs/vigil/