Robot fleet mission supervisor

This page describes a simulated warehouse that we built and ran. LibraOS acts as the mission layer for three mobile robots. An operator gives an order in plain language. An agent turns the order into delivery missions. Every mission that moves a robot waits for a person to approve it. When a robot reports that it finished, a second agent checks that claim against an overhead camera before the delivery counts as done.

Scope

Everything below was run in simulation (Gazebo) on 2026-11-05. No physical robot was involved. LibraOS does not steer the robots. Motion control, obstacle avoidance and the hardware emergency stop stay on the robot's own controller. LibraOS works one level up, at the level of missions, approvals and evidence.

Overhead camera, 2× speed. All three robots were sent with a planted firmware fault, and all three reported "completed". In fact amr1 (left) stopped short of its dock while still carrying the tote. amr2 (centre) drove to its dock empty, and its tote is still on the shelf. amr3 (right) is blocked by a pallet. The verifier rejected all three.

What this shows

• Plain-language orders become exact missions. A local model on the edge computer turned each order into the expected three dispatches in 10 of 10 rounds.
• No robot moves without a person. Each dispatch became a pending action. No robot moved before a second person approved it. Rejected actions never ran. Emergency stop is deliberately exempt from approval, and it never waited.
• "Done" is checked, not trusted. A robot's own report is a claim. The verifier agent certified a delivery only when the camera showed the item at the destination. It caught all 18 planted false completions and did not reject any of the 12 real deliveries.

The setup

Part	What we ran
Simulator host	Gazebo Harmonic with ROS 2 Jazzy, on an x86 workstation with an RTX GPU. Three differential-drive robots, three shelves with totes, three docks
Fleet bridge	A small service that plays the robots' firmware: odometry navigation, a gripper, a status report per mission. It exposes the robot functions as a signed callback endpoint
Site camera	The simulator's ground-truth positions, exposed as a separate tool. This is the evidence that the firmware does not control
Edge computer	An NVIDIA GB10 (arm64), standing in for the robot fleet's onboard computer. It runs LibraOS Runtime v0.3.0 in Docker on a network with no internet route
Model	A Qwen model served locally by vLLM on the same edge computer. No hosted model was used
Approval	A second user account acts as the operator. The account that gave the order cannot approve its own actions

Two agents

Both agents are agent files with tool_scope: none, so they have no shell or file tools, only the robot tools declared below. Every tool call goes to the fleet bridge as an HMAC-signed callback.

fleet-ops plans missions. This is an excerpt of its tool declarations:

custom_tools:
  - name: fleet_status          # read: robot poses, idle/busy, gripper
    side_effects: false
  - name: dispatch_mission      # moves a robot: robot, item, source, dest
    side_effects: true
    risk_tier: high             # held as a pending action until a person approves
  - name: mission_report        # the firmware's own report: a claim, not proof
    side_effects: false
  - name: emergency_stop        # stop one robot now
    side_effects: true
    risk_tier: low              # runs immediately; must never wait for approval

mission-verifier checks claims. It has only two read tools, mission_report and site_camera. It certifies a mission only when the report says completed and the camera shows the item on the floor at the destination. It answers with one line of JSON: {"task_id": "...", "verdict": "certified" | "rejected", "reason": "..."}.

Both agents are called on native chat, with the agent route ID in the URL: POST /agents/v1/fleet-ops/chat and POST /agents/v1/mission-verifier/chat. Agent files load when Runtime starts. Our test harness reloads them (POST /api/admin/agents/reload) after every Runtime start, before sending any order. See Loading and reloading.

From order to verified outcome

1. Order. The operator writes, for example, "Please stage the outbound totes. tote_A goes to dock_1, tote_B to dock_2, tote_C to dock_3."
2. Plan. fleet-ops calls fleet_status, then calls dispatch_mission once per delivery.
3. Hold. Because dispatch_mission has side effects and a high risk tier, Runtime does not call the fleet bridge. It creates a pending action, and the agent tells the operator the mission and the pending action ID.
4. Approve. A second person reviews the action's input (robot, item, source, destination) and approves it. Only then does Runtime send the signed callback, and the robot starts. A rejected action is never sent.
5. Report. The firmware reports the mission as completed or failed.
6. Verify. mission-verifier reads the report and the camera, and certifies or rejects. The delivery counts as done only when it is certified.

The Runtime dashboard's Activity page shows every turn of both agents:

Faults we planted

To test the verifier, the simulated firmware can lie. Each fault makes the robot fail the delivery and still report completed:

Fault	What physically happens	What the firmware reports
Odometry drift	On the loaded leg, odometry over-reports distance, so the robot stops about 35% short of the dock	completed
Stuck, reported as success	A pallet blocks the aisle. Navigation times out, and a firmware bug treats the timeout as success	completed
Phantom pick	The gripper closes on air, but the gripper sensor reads "holding"	completed

For comparison, here is a run with no faults. All three totes reach their docks:

Results

30 missions in 10 rounds of three. 18 missions had a planted fault and 12 did not. Ground truth is the simulator's own record of where each tote ended up.

Who decides "done"	False completion (said done, was not)	Missed delivery (said not done, was)
Robot firmware self-report	18 / 18 (100%)	0 / 12
mission-verifier with camera evidence	0 / 18 (0%, 95% CI 0–18%)	0 / 12 (0%, 95% CI 0–24%)

Check	Result
Orders turned into exactly the expected three dispatches	10 / 10 rounds
Any robot moved before approval	0 / 10 rounds
Rejected dispatch	Not sent; the robot stayed idle
Emergency stop through fleet-ops	Robot stopped, mission marked failed, no approval wait; 6.0 s from request to answer
fleet-ops time per order (three dispatches)	median 21.3 s, max 25.8 s
mission-verifier time per check	median 6.3 s
Verifier replies that were not valid JSON	0 / 30

Limits

Read these before drawing conclusions from the numbers:

• The faults were designed to fool the self-report. The comparison shows that an independent check catches lies that a self-report cannot. It does not measure how often real firmware lies.
• The camera is perfect. It reads the simulator's ground truth. A real camera adds occlusion, misdetection and latency, and the verifier is only as good as its evidence.
• The orders were unambiguous. Item, shelf and dock names were explicit. We did not test vague orders, conflicting orders or a busy fleet.
• Emergency stop through an agent is not a safety system. It went through a model and took about 6 seconds. A real deployment keeps a hardware stop that does not depend on LibraOS, the network or a model.
• Small sample. 30 missions give wide confidence intervals. Treat these results as a working demonstration, not a reliability figure.

Where this fits

The pattern carries over to any fleet that can report a mission and can be observed independently: warehouse AMRs, inspection drones, lab automation. In that pattern, an agent plans, a person approves anything that moves, and a separate agent checks the result against evidence that the robot does not produce. LibraOS supplies the agents, the signed tool callbacks and the approval queue. Your robot stack still owns control and safety.

See also: Call your systems with tools · Employees and agents in YAML · Security & data boundaries