Robot arm work cell with a VLA policy
This page describes a simulated work cell that we built and ran. One robot arm runs a learned vision-language-action (VLA) policy. An operator gives an order in plain language. An agent on the robot's own computer turns it into one of the cell's tasks. The task waits until a person approves it. When the arm's controller reports that it finished, a second agent checks the camera frames before the order counts as done.
Everything below was run in simulation (LIBERO on MuJoCo) on 2026-11-06. No physical robot was involved. LibraOS does not steer the arm. The VLA policy and the arm's controller do that, and the hardware emergency stop stays outside LibraOS. LibraOS works one level up, at the level of tasks, approvals and evidence.
55 seconds, with sound. Three real runs from the 20-order test: a failed job the verifier caught, a good job it passed, and a near-miss it let through. The "without LibraOS" numbers at the end come from the same 20 runs: they show what a stack that trusts the controller's report would have booked.
Why this matters for VLA policies
A VLA policy stops when its action sequence ends. Most robot stacks report that stop as "completed", whether or not the job got done. In this test the arm's controller reported "completed" on 20 of 20 orders, and the job was actually done on 16. Without a check, the other four are booked as done and the next step starts from a wrong state.
What this shows
- No arm motion without a person. Every task became a pending action. The arm moved before approval in 0 of 20 orders.
- Plain-language orders map to the right task. An offline model on the robot's computer picked the right task from the cell's list in 20 of 20 orders.
- "Done" is checked, not trusted. The verifier stopped 2 of the 4 failed jobs from closing, and held back none of the 15 good jobs it checked. On a larger test of 300 episodes it had never seen, it let about 30% of failures through, almost all of them near-misses in placement. The limits section explains why.
The setup
| Part | What we ran |
|---|---|
| Policy | SmolVLA (lerobot/smolvla_libero) with LeRobot 0.7.0, on one RTX 3080 |
| Simulator | LIBERO kitchen scene (libero_goal, ten tasks), MuJoCo. Each run starts from a layout never used in earlier runs |
| Cell service | A small service that runs one policy episode per approved task and reports the controller's status. It exposes the cell as a signed callback endpoint |
| Robot computer | An NVIDIA GB10 (arm64) running LibraOS Runtime with a local Qwen model, no hosted model. It hosts the planning agent and the approval queue |
| Site server | A second LibraOS Runtime that hosts the verifier agent. It calls a vision model through the site's model gateway |
| Approval | A second user account acts as the operator. The account that gave the order cannot approve its own actions |
What "done" means. We wrote it down before any run: the goal state holds and the gripper has let go. Every episode ends with the same short retract, so release is visible on camera. The simulator scores the final state. That score is used only to grade the results. No agent sees it.
Two agents
Both are agent files with tool_scope: none,
so they have no shell or file tools.
cell-ops runs on the robot computer and plans tasks. Its tool declarations:
custom_tools:
- name: cell_status # read: arm idle/busy and the exact task list
side_effects: false
- name: run_task # moves the arm: one task, word for word from the list
side_effects: true
risk_tier: high # held as a pending action until a person approves
supports_dry_run: true
- name: task_report # the controller's own report: a claim, not proof
side_effects: false
- name: emergency_stop # stop the arm now
side_effects: true
risk_tier: low # runs immediately; must never wait for approval
Its instructions tell it to pick exactly one listed task, request it once, stop at the pending action, and never say a task is done.
cell-verifier runs on the site server with maxTurns: 1 and no tools.
It gets the task text and two frames from the cell cameras: the first, before
the arm moved, and the last, after the retract. Each frame shows the front
camera and the wrist camera side by side. It answers with one line of JSON:
{"verdict": "certified" | "rejected", "reason": "..."}. Its prompt was frozen
before any test episode existed.
From order to verified outcome
- Order. The operator writes, for example, "Set the bowl on the plate."
- Plan.
cell-opscallscell_status, picks put the bowl on the plate, and callsrun_task. - Hold.
run_taskhas side effects and a high risk tier, so Runtime does not call the cell. It creates a pending action. The arm stays idle. - Approve. A second person reviews the task and approves it. Only then does Runtime send the signed callback, and the arm starts.
- Report. The controller reports "completed" when the policy stops.
- Verify. The site integration sends the task and the first and last
frames to
cell-verifier. The order closes only when it is certified. A rejected order stays open for rework, with the frames and the reason.
Results
The 20-order run
Ten paraphrased orders, each run twice. Ground truth is the simulator's own check of the final state.
| Who decides "done" | Failed jobs closed as done | Good jobs held back |
|---|---|---|
| Arm controller's self-report | 4 / 4 | 0 / 16 |
cell-verifier with camera frames | 2 / 4 | 0 / 15 checked* |
* One more check failed because the model gateway returned an error. It was counted as not certified, so the order stayed open.
Both failures that got through were push the plate to the front of the stove: the plate stopped just short of the target area.
| Check | Result |
|---|---|
cell-ops picked the right task | 20 / 20 (median 10 s per order) |
| Arm moved before approval | 0 / 20 |
| Operator rejected the task | Not sent; the arm stayed idle |
| Order the cell cannot do ("Wash the plate in the sink") | No action requested; the agent listed what the cell can do |
| Emergency stop during a run | Arm stopped without waiting for approval, 11 s from message to stop |
cell-verifier time per check | median about 4 s |
How good is the verifier?
Four failures are too few to measure a verifier, so we also tested
cell-verifier's prompt on 300 episodes it had never seen, from two LIBERO
suites (spatial and goal). The prompt was frozen before these episodes were
recorded.
| Test set | Verifier passed a failure | Verifier rejected a success |
|---|---|---|
| 100 episodes | 17 / 42 (40%, 95% CI 27–56%) | 0 / 58 (0%, 95% CI 0–6%) |
| 200 more episodes | 19 / 63 (30%, 95% CI 20–42%) | 0 / 137 (0%, 95% CI 0–3%) |
It reliably catches gross failures: an object never moved or never picked up, dropped in the wrong place, the wrong drawer opened, or an object still in the gripper. It misses near-misses: a bowl resting on the rim of the plate, a plate pushed close to the target area, a drawer only partly open.
One task, three held-out episodes: a success certified, a failure caught, a near-miss passed.
We also gave the verifier each object's position and tilt, at about 1 cm resolution, alongside the frames. It barely helped (17 of 63 failures passed instead of 19), and it wrongly rejected 3 successes. The simulator decides "on the plate" by contact, and position alone cannot tell "resting on" from "propped on the rim".
Limits
Read these before drawing conclusions from the numbers:
- Simulation only. One arm, one kitchen scene, ten tasks. No physical hardware was used.
- Small chain sample. The 20-order run had 4 failures. Halving them shows the approach works on this cell. It is not a catch rate. Use the 300-episode test for that.
- Two frames are thin evidence. About 30% of failures still get through, nearly all near-misses in placement. Closing that gap needs evidence of contact or support at the target (a load cell, a tactile or depth reading, a scanner at the slot), or a placement tolerance that the task owner signs off.
- The verifier needs a strong vision model. We also tried the robot computer's local model as the verifier. It passed more failures and was much slower, so the verifier runs on the site server.
- Emergency stop through an agent is not a safety system. It went through a model and took about 11 seconds. A real cell keeps a hardware stop that does not depend on LibraOS, the network or a model.
- "Done" needs a written definition. A verifier can only be held to one. Each task owner has to supply it.
Where this fits
Learned policies are getting good at doing tasks, but they do not know whether they succeeded. The pattern here works for any cell that can report a job and can be observed independently. An agent plans, a person approves anything that moves, and a separate agent checks the result against evidence that the robot does not produce. LibraOS supplies the agents, the signed tool callbacks and the approval queue. Your robot stack still owns control and safety.
See also: Robot fleet mission supervisor · Call your systems with tools · Employees and agents in YAML · Security & data boundaries