Deep dive into Sibyl Research System internals. For contributors and advanced users.

The core of Sibyl is a 19-stage state machine in sibyl/orchestrate.py.

• _compute_action(stage) — Returns the Action for the current stage (not the next one)
• _get_next_stage(stage) — Computes the transition to the next stage (called by cli_record)
• cli_next(workspace) — Public API: returns the next action as JSON
• cli_record(workspace, stage) — Public API: marks a stage complete and advances state

When lark_enabled: true, Feishu sync is no longer a pipeline stage. Instead, cli_record() appends a trigger to lark_sync/pending_sync.jsonl and returns sync_requested: true. The main Claude session must launch sibyl-lark-sync in the background and continue the research loop without waiting. Sync status is written to lark_sync/sync_status.json.

The experiment supervisor owns local recovery, GPU refresh, dispatch, and lightweight intervention. When it either resolves something material or hits a blocker that needs main-loop judgment, it appends a structured wake event to exp/experiment_supervisor_main_wake.jsonl via experiment-supervisor-notify-main. The main control-plane loop must not sleep for the full experiment poll interval in one chunk; instead it checks wake_cmd every short interval (wake_check_interval_sec, default 90 seconds) and immediately collaborates when an event reports requires_main_system: true.

The orchestrator returns actions that the main session executes:

status.json now uses explicit paused / stop_requested booleans, with optional paused_at / stop_requested_at timestamps for diagnostics. Legacy numeric *_at markers are still read for backward compatibility, and transient paused state is auto-cleared by cli_next() so the control plane does not stall. Explicit /sibyl-research:stop remains the only supported manual halt path.

All agent roles are implemented as fork skills in .claude/skills/sibyl-*/SKILL.md.

1. Orchestrator returns action_type: "skill" with skill name and args
2. Main session invokes skill via Skill tool
3. Skill runs in an isolated subagent context (fork)
4. Skill uses !command`` to dynamically load the prompt template from sibyl/prompts/
5. Skill executes with the loaded prompt and writes outputs to workspace

Defined in .claude/agents/:

Used for debate stages (idea_debate, result_debate, writing_sections, writing_critique).

Requires: CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=1

1. TeamCreate(team_name) — Create team
2. TaskCreate(subject, description) — Create task per teammate
3. Agent(team_name, name) — Launch teammates in parallel
4. TaskUpdate(taskId, owner) — Lead assigns tasks (not self-claimed)
5. Wait for all teammates to idle
6. SendMessage(shutdown_request) — Close each teammate
7. Execute post_steps (e.g., synthesizer, codex review)

Four parallel stages support sub-step checkpoints for crash recovery:

Checkpoints use 4-way validation:

1. Status == completed
2. Output file exists
3. File mtime ≥ stage start time
4. File size matches recorded size

On resume, only incomplete steps are re-executed.

sibyl/gpu_scheduler.py handles parallel experiment execution:

1. Read task_plan.json with task dependencies
2. Topological sort into dependency layers
3. Greedy assignment based on available GPUs and gpu_count per task
4. Return batch of tasks that can run in parallel
5. After batch completes, compute next batch from remaining tasks

On shared servers, before each experiment batch:

1. Orchestrator returns action_type: "gpu_poll"
2. Main session SSH queries nvidia-smi on the server
3. parse_free_gpus() identifies available GPUs
4. Results written to a workspace-scoped marker such as /tmp/sibyl_<workspace_scope>_gpu_free.json
5. Next cli_next() reads the marker and assigns tasks

sibyl/evolution.py implements cross-project learning:

SYSTEM, EXPERIMENT, WRITING, ANALYSIS, PLANNING, PIPELINE, IDEATION

Each category maps to specific agents for targeted improvement.

build_digest() aggregates outcomes.jsonl into digest.json with mtime caching.

• Compare early vs late iteration scores
• Mark lessons as effective/ineffective/unverified
• Requires ≥ 4 occurrences for judgment
• Ineffective lessons get 0.3x weight reduction

Effective lessons are injected into agent prompts through the compiled prompt layer: render_skill_prompt(agent, workspace_path=...) builds the final prompt and appends contextual evolution lessons plus any project-specific overrides.

get_self_check_diagnostics() detects:

• Declining quality trend
• Recurring system errors
• Ineffective lesson accumulation

Written to logs/self_check_diagnostics.json after each reflection.

FarsOrchestrator.__init__ auto-loads project-level config.yaml:

This ensures per-project configuration works without explicit config passing.