40 solved tasks, three robots, original capture
ORA
Open Robot-Use Agents: an open-weight model operating robots through tools, with no VLA.
ORA: Open Robot‑Use Agents
An open-weight model operating robots through tools, with no VLA.
TL;DR: ORA lets DeepSeek-V4.1-Flash operate simulated robots through perception and motion tools. We use the released model without robot-specific fine-tuning or a VLA policy. It solves 69.2% of the RoboTwin cases and 81.4% of the LIBERO-Pro cases in our evaluation. The published RPent agents using closed frontier models score 58.4–62.4% and 75.1–92.6%, respectively, under different conditions. ORA supplies the robot documentation, tool feedback and context management around the model. These results compare complete systems; they do not isolate the effect of the harness.
RoboTwin C2R
50 tasks · 250 casesOurs: 173 of 250 cases.
LIBERO-Pro
8 Task and Swap suites · 800 casesOurs: 651 of 800 cases.
ORA rates use all evaluation cases. Models, observations, action budgets and access to task memory differ across systems. RPent rows are reproduced as published. [a] Astra uses frozen memory batches. [b] Flash Mode uses released memory and reports the best-performing seed from s0 to s9. Protocol, infrastructure retries and exact values: appendix.
Why robot-use agents
Language models solve competition mathematics[1], fix issues in codebases[2] and operate graphical desktops[3][4]. Coding and computer-use agents act through interfaces such as terminals, browsers and a mouse. CUA and OpenCUA also train their models for computer interaction.
Robot-use agents apply a similar approach to manipulation[5]. A language model receives observations and instructions, then calls tools to move the robot. The proposal illustrated below is to supply a general model with a robot and its manual[6]. In ORA, moving to another supported robot requires a body card and compatible perception tools, controllers and environment bindings. The language model itself receives no robot-specific fine-tuning. This separation also lets us test another base model through the same interface.
Recent robot-use demonstrations use frontier models[8][9][10]. On one reported robot-control task, GPT-6 Astra scores 95% against 40% for Fable 5.1, using 6.2 times fewer output tokens[7]. Evaluation frameworks provide adapters for testing language models and VLAs on simulated or real arms[11]. The action interface also matters: one study reports much better results with higher-level actions than with direct low-level control[8]. We test a released open-weight model with perception and motion tools, without a VLA policy.
Open weights let us preserve the exact model checkpoint and serve it ourselves. We can record the requests and responses without an intervening vendor harness, an issue also discussed in AHa-3D[12]. Together with the tool receipts and official verdict, these records help us investigate failures, although they do not by themselves establish a cause. Verified episodes may later provide training data. On earlier RoboTwin and LIBERO cohorts, estimated API-equivalent token spend was $0.29 to $0.35 per solved case, including failed attempts and excluding GPU and simulator costs.
We keep DeepSeek-V4.1-Flash fixed while developing the harness around it. Run audits found guessed robot constants, repeated refused calls and geometry errors. Some constants were missing from the manual; others had already appeared in tool results. These observations motivated changes to documentation and feedback. The method below distinguishes those audits from controlled tests of individual changes.
A taxonomy of robot-use agents
Robot systems divide planning and motion control in different ways. The categories below describe where decisions happen and can overlap within a system.
| Approach | How the work is split | Representative work | What it does well | What it asks for |
|---|---|---|---|---|
| End-to-end VLA | one model maps images and the instruction to motor actions | π0 and π0.5[13], OpenVLA[14] | motion policies learned from robot data | robot action training data and compatibility with the target embodiment |
| System 2 directing System 1 (slow planner, fast motor policy) | a vision-language planner directs a fast motor policy or skills | SayCan[15], Hi Robot[16], Helix[17], GR00T N1[18], RPent agents[19] | language understanding on top of learned skills | a planner-executor interface and a learned motor policy or skill library |
| Code as policy | the model writes a program over perception and control primitives | Code as Policies[20], CaP-Agent0[21], ASPIRE[22], OpenRUA[23] | programs that can be inspected and reused | perception and control primitives, with feedback checks in the generated program |
| Agent policy | the model chooses each bounded motion with a tool call; controllers run it | ORA | replans after bounded actions using tool feedback, with an open-weight model | handwritten tools and controllers; time spent waiting for the model |
The frequency and abstraction of model decisions affect how these systems operate. A generated program reacts through its programmed checks. A model can instead select actions throughout the episode. In one study, the best tested model reached 5.5% on LIBERO under direct low-level control[8]. ORA asks the model to choose bounded motions, executes them through controllers and returns feedback before the next motion decision.
How ORA works
The model reads the manual and recent context, then reasons and calls tools. A turn may contain several read-only calls or one call that changes the world. Motion calls go to the supervisor, which checks robot limits, runs the controllers and returns measured pose, gripper opening and holding evidence. The model can query the benchmark's official success check. It receives true or false for the current state without ending the episode, but cannot see the checker code, object ground-truth state or simulator contacts.
1ManualPrompt Bundle
Robot-specific cards and tools bind to compatible controllers.
2ToolsToolSurface
3ContextContext Management
Compacted in stages when the window fills.
ARX X5RoboDojo
Fixed-base ARX X5: left 6-DoF arm, right 6-DoF arm; parallel-jaw grippers. Cameras: head fixed RGB-D; left_wrist, right_wrist moving RGB-D.
ALOHARoboTwin
Fixed-base ALOHA-AgileX: left 6-DoF arm, right 6-DoF arm; parallel-jaw grippers. right_tcp is the controlled reference, 0.12 m from the flange; commands already target it.
FrankaLIBERO
Fixed-base Franka Panda: right 7-DoF arm; parallel-jaw grippers. right_grip_site is the controlled reference, 0.097 m from the end-effector root; commands already target it.
Body-card openings, verbatim. Shared rules are common; the task and environment contract vary with the benchmark.
A manual for each robot
The prompt combines shared rules with a robot body card, the benchmark's task instruction and tool definitions. The body card documents the robot's frames, sensors and motion conventions. Supporting another robot also requires compatible tools and controllers; a different benchmark supplies its own task and environment contract. The manual contains no task solutions.
Finding · run audit. In 28 of 46 audited runs, the model guessed the fingertip offset, using values from 0 to 0.103 m even though grasp_plan had returned it. The audit identifies failures to use available information.
What the manual contains
| Part | What it says | Changes with |
|---|---|---|
| Rules | How to turn evidence into action: label every claim as direct evidence, inferred, candidate, executed or verified; settle what is missing before acting on it; send arithmetic to the calculator; make one world-changing call per turn and compare the views before and after it. | nothing; shared by every robot and task |
| Environment boundary | What the benchmark exposes in public terms: the world frame and units, the cameras and proprioception, and the public completion contract, which restates what counts as done without saying how to get there. | the benchmark |
| Body card | The robot: arms and joints, grippers, the controlled reference point, cameras, frames and quaternion order. | the robot |
| Task | The benchmark's instruction, and what the task language requires, what the contract makes explicit and what neither specifies. | the task |
| Tool manual | Typed definitions of every tool the body supports: arguments, frames, units, results and limits, delivered as the callable schema. | the robot |
Manual of one RoboDojo run (core loop v8, ARX X5 card v5, tool schema v1). The cost estimator assumes 84% to 86% of input is cacheable across requests, including conversation history. It uses token counts rather than comparing prefix contents; this is not a measured cache-hit rate.
vision_segmentmask and measured centre
grasp_plancandidates, checked for reachTool outputs from one RoboDojo decision.
Tools that return measurements
Perception tools use SAM 3[24] to segment objects named in words or selected by pixels, then use depth to measure their position. Contact-GraspNet[25] proposes grasps, which are checked for reach. Other tools project points between camera and world coordinates or calculate derived geometry. Action receipts report the measured outcome and available holding evidence.
Finding · paired A/B. An overhead footprint centre replaced a mask median biased 2 to 3 cm toward an oblique camera. Of 59 paired states from two affected LIBERO task families, 29 succeeded with the mask median and 42 with the overhead estimate (McNemar p = 0.011). On 30 states from other families, it fell from 25/30 to 21/30 (p = 0.13). The improvement was specific to the affected families in this test.
Finding · run audit, offline replay. Audited runs showed the model abandoning poses after inverse-kinematics refusals. An offline replay found reachable tilted alternatives for 69 of 77 refusals. The updated feedback suggests a reachable orientation; the replay tests reachability, not whether the alternative completes the task.
Finding · corpus count. A corpus audit classified 12.2% of 35,270 responses as working out geometry by hand. In a separate prompt-only comparison on 11 previously failed RoboTwin cases, reasoning calls containing arithmetic were 76.2% with the original prompt and 76.3% with calculator-routing instructions. ORA provides vector and frame calculations and accepts poses specified by world axes. Three earlier runs had written quaternions pointing the gripper upward for “approach from above”.
The tools, with real calls
| Tool | What comes back | A call from the runs |
|---|---|---|
vision_ground | candidate masks and measured centres for an object named in words (SAM 3) | vision_ground(camera="head_camera", text="laptop", max_candidates=3) |
vision_segment | a mask for pixels the model picks, with measured centre, extent and support height | vision_segment(camera="head", points_uv=[[357, 248]]) |
mark_point | world points measured from depth at chosen pixels | mark_point(view="head_camera", point_ids=["h_farL", …], points_uv=[[160,112], …]) |
geometry_query | where world points fall in a camera image, and back | geometry_query(camera="wrist", world_points_m=[[0.048, 0.211, 0.940], …]) |
compute | vector and frame arithmetic on named values | compute(bindings=[{name:"E", value:[0.05,-0.043,0.786]}, …], expressions="u = geom.unit(E - H) …") |
grasp_plan | grasp candidates from Contact-GraspNet, each checked for reach, drawn on the image | grasp_plan(arm="right", target_ref="target-6d01…") |
move_pose | a receipt with the measured pose after the motion | move_pose(targets={right: {position_m:[0.04,-0.082,0.84], orientation_xyzw:[…]}}) |
set_gripper | a receipt with the measured opening and whether something is held | opens or closes one gripper |
grasp_execute | a receipt for a whole grasp: status, hold and lift | grasp_execute(candidate_id="grasp-2f16…") → completed · held · lifted 26 mm |
Tools that only look can run several in one turn. A call that changes the world must be the only call in its turn. The supervisor checks it against the robot's limits, admits or refuses it with a reason, runs the controllers and returns what it measured. A pose move holds the gripper as it is, and a gripper command holds the pose. The model never sees object ground-truth state, simulator contacts or ground-truth segmentation. It does see masks produced by the perception tools.
Context management
Recent observations and receipts remain inline. Older images and large results are replaced by references to files the model can reopen, while action summaries remain in context. In sessions with several rounds, notes from earlier rounds also remain. If the input window still fills, the oldest material is compacted in stages, as in Claude Code[26]. The benchmark runs reported here use one round per case.
Finding · same-commit A/B. One shortened observation format also rounded measurements and omitted limits. It scored 10/27 RoboDojo rounds against 17/27 for the full format. After exact values and six rules were restored, a second comparison scored 15/24 against 13/24 complete pairs. These tests compare delivery formats, not context management as a whole.
How the context is kept
- Recent steps in full. The current step and the last few completed steps keep their images and exact tool results inline: at most five completed steps for images and six for read-only results.
- Older steps as pointers. Earlier images, large or repeated results and bulky observation bodies become exact pointers to saved files. Sequence numbers, comparisons and the effect of the last action stay inline, and the model can reopen any file.
- Few rewrites. Older steps are reduced in groups of four. Between reductions each request extends the previous one exactly, so the provider can reuse its cache.
- Two backstops. Past 150,000 input tokens a finished episode is condensed into what was decided and measured. At 80% of the model's input window the oldest material is compacted down to 65%, in stages as in Claude Code.
- No task memory. Nothing carries from one task to the next. Within one session on one scene, the lesson the model writes at a reset stays in context for the next round.
Context schema v6, stride 4. The replay illustrates a session with several rounds; the reported benchmark scores use fresh single-round sessions.
Quick start
The code is being prepared for open release at hao-ai-lab/ora-agent. This section will show the install and a single LIBERO episode, from launching the model service to watching the run in the viewer.
Case studies
Each clip pairs the run's camera capture with the model's words, tool calls and results. Motion is sped up.
Look before actingLIBERO-Pro
“Pick the ketchup and place it in the basket.”
Zooms into crops to tell the ketchup from similar bottles, then grasps.
Trial and errorRoboDojo
“Remember the first object on the conveyor, then pick the matching object when it appears again.”
Misses twice on the moving belt, reads the cause from the receipt, then catches.
Learning across roundsRoboDojo
“Stack the blocks from bottom to top in the order of blue, yellow, and orange, then reset the robot arm.”
Round 1 knocks a block off the table. Round 2 names the cause and lifts straight up first.
Check before doneRoboTwin
“Use one arm to press the stapler.”
A trial outside the evaluation. After the success check returns false, the model observes that the stapler top has not moved, presses deeper and passes.
Two rounds of one task
Two rounds on the same RoboTwin scene, from a trial run outside the reported evaluation, chosen after seeing the outcome. Round 1 pushes the lid shut, and the model's note from round 1 stays in context. In round 2 the success check returns false, the model finds its finger 6 cm short of the lid's edge, and moves it under the edge.
Round 1 · failed23 decisions
“The approach pushed the lid down toward closed.”
“Key lesson from round 1: a top-down approach descends onto the lid and pushes it closed.”
Round 2 · succeeded6 decisions
“Official check is false … my finger was at x≈0.007 while the lid's free edge is at x≈0.07.”
More episodes: gallery.
Results
We evaluate the RPent[19] case lists for RoboTwin[27], LIBERO-Pro[28] and LIBERO[29], then compare with the published RPent leaderboard. The systems use different models, observations and action interfaces. ORA also exposes an official success query that we have not confirmed is available to the RPent agents. These are comparisons of complete systems.
RoboTwin and LIBERO-Pro are charted at the top of the page. Standard LIBERO:
LIBERO
4 suites · 40 tasks · 400 casesOurs: 334 of 400 cases.
All planned cases enter the denominator. The refreshed LIBERO and LIBERO-Pro cohorts have valid sealed outcomes for all 1,200 cases, including five infrastructure replacements. RoboTwin retains its 250-case cohort because its default was unchanged. [a] Astra uses separately frozen memory batches. [b] Flash Mode uses released memory and reports the best-performing seed from s0 to s9. The appendix records versions, replacements and protocol differences.
Beside code-writing agents
CaP-Agent0[21] and ASPIRE[22] generate robot programs from perception and control primitives without a VLA policy. Their reported LIBERO-Pro configurations use frontier models; CaP-X also evaluates open-weight models on other tasks. Neither reports RoboTwin. On LIBERO-Pro both report the six Spatial, Object and Goal suites, which puts them beside our runs:
| Success, % | Spatial Task | Spatial Swap | Object Task | Object Swap | Goal Task | Goal Swap | Mean |
|---|---|---|---|---|---|---|---|
| ORA · DeepSeek-V4.1-Flash | 98 | 91 | 93 | 98 | 77 | 78 | 89.2 |
| ASPIRE | 60 | 51 | 95 | 98 | 45 | 81 | 71.7 |
| CaP-Agent0 | 14 | 12 | 18 | 22 | 17 | 26 | 18.2 |
OpenRUA[23] reports 87.0% on LIBERO-PRO with Claude Opus 5 driving Claude Code. We have not yet compared its protocol with ours, so it is not in the table. Reported numbers side by side, not a matched comparison. ASPIRE evaluates 10 tasks on held-out seeds 1–50 per suite after learning skills on seeds 51–65, with Claude Opus 4.6 and SAM3. Its Long results, 38.3% (Task) and 22.6% (Swap), come from zero-shot transfer of a LIBERO-90 skill library, against our 52.0% and 64.0%. CaP-X reports 50 trials per task-condition. We retain the rounded percentages reproduced by RPent; the exact model configuration behind those results is not fully specified. Ours use RPent's seeds 1–10. Swap is the board's name for ASPIRE's position perturbation.
Cost
Estimated API-equivalent token spend per solved case on the earlier g003/g004 cohorts, including failed attempts, at the recorded September 17 off-peak V4.1-Flash rates:
The model was self-hosted. These estimates assume prefix-cache reuse and exclude simulator and GPU costs. LIBERO and LIBERO-Pro cost estimates refer to the earlier cohorts, not the refreshed scores above.
Limitations and what's next
ORA is slow. Median episode time is 14 to 25 minutes across the reported benchmark cohorts. In a separate 79-episode timing study, 63% of total time was spent waiting for the model. Simulation pauses between actions, so these results do not demonstrate real-time control.
Long-suite success ranges from 52% to 64%, and two published frontier-model agents score higher overall on LIBERO-Pro. An earlier RoboDojo scan solved 18 of 126 rounds. All results are from simulation with one language model. The tools and controllers are written by hand, and ORA cannot yet create its own tools.
Two findings above have controlled comparisons of specific changes. The others come from run audits, corpus counts or offline replay. Those observations help identify candidate fixes but do not establish that each fix improves task success. The appendix includes the unsuccessful tests as well.
We plan to reduce time spent between actions and test whether the system can revise its tools. A separate direction is to train a smaller model from verified episodes; neither capability is evaluated here.
Getting started
Code: ORA on GitHub (being prepared for open release). Episodes: the gallery has 30 episodes, solved and failed, with each run's camera capture. Evaluation: the appendix holds the protocol, exact values, cost, failure audit and provenance.
References
- OpenAI. Learning to reason with LLMs. 2024. openai.com/index/learning-to-reason-with-llms/ ↩
- Yang et al. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. NeurIPS 2024. arxiv.org/abs/2405.15793 ↩
- OpenAI. Computer-Using Agent. 2025. openai.com/index/computer-using-agent/ ↩
- Wang et al. OpenCUA: Open Foundations for Computer-Use Agents. 2025. arxiv.org/abs/2508.09123 ↩
- Phillip Isola. Robot-Use Agents. Essay, 2026. web.mit.edu/phillipi/www/writing/robot-use-agents.html ↩
- Jianyuan Wang. Slide at the Image Matching Workshop, June 2026 (photo shared on X). x.com/ducha_aiki/status/2062653228802515047 ↩
- Jay Chooi (Robocurve). GPT-6 Astra on a robot control task. Thread on X, September 5, 2026. x.com/chooi_jeq/status/2096064315115839904 ↩
- Anthropic. Claude plays robotics. July 9, 2026. www.anthropic.com/research/claude-plays-robotics ↩
- Robocurve. GPT-6 Astra on robotic manipulation. 2026. openai.robocurve.org/gpt-6-astra/ ↩
- Waddle Labs. Introducing Waddle: agents that control robots. 2026. www.waddlelabs.ai/research/introducing-waddle ↩
- Robocurve. Inspect Robots: open-source evaluations for physical AI. 2026. github.com/robocurve/inspect-robots ↩
- Congrong Xu, Siyuan Bian, Jiawei Wang, Haowei Sun and Jun Gao. AHa-3D: Agentic Tool Use for Real2Sim with GPT-6 Astra. Blog post, 2026. kevinxu02.github.io/real2sim-indoor-site/ ↩
- Physical Intelligence. openpi: π0 and π0.5 models. github.com/Physical-Intelligence/openpi ↩
- OpenVLA: an open-source vision-language-action model. github.com/openvla/openvla ↩
- Ahn et al. Do As I Can, Not As I Say: Grounding Language in Robotic Affordances. 2022. say-can.github.io/ ↩
- Shi et al. Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models. 2025. arxiv.org/abs/2502.19417 ↩
- Figure. Helix: A Vision-Language-Action Model for Generalist Humanoid Control. 2025. www.figure.ai/news/helix ↩
- NVIDIA. GR00T N1: An Open Foundation Model for Generalist Humanoid Robots. 2025. arxiv.org/abs/2503.14734 ↩
- RPent: paper, benchmark case lists and leaderboard (RLinf). arxiv.org/html/2607.08448v4 ↩
- Liang et al. Code as Policies: Language Model Programs for Embodied Control. 2022. code-as-policies.github.io/ ↩
- CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation. ICML 2026. arxiv.org/abs/2603.22435 ↩
- ASPIRE: Agentic Skills Discovery for Robotics. arxiv.org/html/2607.00272v1 ↩
- Chu et al. OpenRUA: Robot-Use Agents Are Zero-Shot Visuomotor Policies. arXiv 2610.02459, 2026. arxiv.org/abs/2610.02459 ↩
- Meta. SAM 3: Segment Anything with Concepts. github.com/facebookresearch/sam3 ↩
- NVIDIA. Contact-GraspNet: grasp generation in cluttered scenes. github.com/NVlabs/contact_graspnet ↩
- Anthropic. Claude Code. github.com/anthropics/claude-code ↩
- RoboTwin 2.0 benchmark (RoboTwin-Platform). github.com/RoboTwin-Platform/RoboTwin ↩
- LIBERO-PRO benchmark. github.com/Zxy-MLlab/LIBERO-PRO ↩
- LIBERO: benchmarking knowledge transfer for lifelong robot learning. github.com/Lifelong-Robot-Learning/LIBERO ↩
Citation
@misc{ora2026,
title = {ORA: Open Robot-Use Agents},
author = {Yang, Yuanbo},
year = {2026},
url = {https://github.com/freemty/robot-harness}
}