Authored by: Franco Cipollone, Sebastian Castro, Yadunund Vijay, Michel Hidalgo, Tully Foote, Katherine Scott, Vicky Brasseur, Michael Gentner, and Anh Vu Vu
As AI rapidly reshapes robotics, software tools specialized for Physical AI are appearing at a rapid pace. This has led the community to ask questions about the future of existing tools designed for more traditional robotics applications, including the Robot Operating System (ROS).
On one hand, ROS remains the world’s most widely deployed open-source robotics framework (2025 ROS Metrics Report). ROS supplies the production tested tools, standard abstractions, OEM-managed hardware drivers, and capabilities needed to turn machine learning models into reliable physical systems. On the other hand, there are gaps to be addressed for ROS to evolve in a way that supports and strengthens the next era of AI-driven robotics.
In this post, we discuss why we believe the most effective path forward is to leverage the strengths of ROS along with those of complementary tools designed for Physical AI. We present our findings, ongoing work, and long-term roadmap to achieve this. We will also share ways for you to get involved in these efforts.
Why Physical AI, and why ROS
We can summarize machine learning (ML) as a collection of methods that can take data and learn a model. Physical AI then describes the application of ML to systems that can interact with the real world, for example via sensors and actuators, which robots have in abundance.
In practice, this spans several key techniques: imitation learning driven by demonstration datasets, reinforcement learning built on environment exploration (frequently aided by simulation), and using pretrained foundation models for high-level decision-making or even fully agent-driven execution.
These techniques are different ways to approach the traditional sense-think-act pipeline, but what has not changed is how we interact with these systems. We still need to describe a robot’s morphology and embodiment, and communicate with it in some way, whether it lives in simulation or on real hardware. We also want to reuse those same interfaces to collect data and run model inference. This is exactly where ROS shines: it already runs on a large share of the world’s robots, in small corners of a codebase or across the whole stack, and it is a shared language that companies and research labs have in common. It provides the middleware, the tooling, and the ecosystem to build on instead of reinventing from scratch.
The fit goes beyond interfaces: learned policies often coexist with components of the classical stack. For example, a model that outputs end effector position and orientation might benefit from an inverse kinematics solver or task-space controller to execute on a 7 degree-of-freedom arm; or, an agent can orchestrate several skills such as waypoint navigation or object picking given high-level goals. Ken Goldberg argues that the next leap in robotics will come from anchoring AI advances atop “good old-fashioned engineering” (The art of grasping, Berkeley Engineer, 2026). ROS, through its vast ecosystem of open-source libraries, is extremely well-positioned for exploring these ideas.
That said, being a good home is not the same as being ready. Working with the community, we have identified a set of gaps that stand between ROS as it is today and the Physical AI platform it can become: moving large tensors between processes with minimal overhead, describing robots and scenes in formats that AI tooling understands, collecting training data consistently, running low-latency inference within control loops, unifying online planning and real-time control to safely track and validate actions coming from learned policies, and making ROS and its dependencies easy to install alongside ML stacks (for example, via Pixi and RoboStack). Closing these gaps is already underway today.
Meet the Physical AI SIG
That work has a home. The OSRA TGC Special Interest Group for Physical AI, known informally as the SIG PAI, exists to make ROS the platform of choice for the next generation of AI-powered robots. Driven directly by the needs of the Open Robotics community, the group works to identify and eliminate the specific friction points that slow down Physical AI development in ROS. By closing these functional gaps, the group creates standard representations and interfaces for training, simulation, and deployment, together with reference applications ensuring the resulting tools support diverse robot embodiments across the entire ecosystem.
The SIG PAI contains several working groups, each taking on a slice of the problem. It is completely open: anyone interested can follow along and get involved in whichever group fits them. The effort happens in public, too. The proposals live as public REPs (REP-0157, REP-0158), the code lives in public repositories, and members come from across the industry: Intel, NVIDIA, BMW, Intrinsic, Robotec.ai, Ekumen, FieldAI, and others.
You can find the group at https://physical-ai.ros.org and on GitHub at https://github.com/ros-physical-ai
We at the SIG PAI are interested in strengthening the capabilities of ROS in two major areas:
- Collecting data for training models
- Running models on real robots
Moving and describing data for Physical AI
Physical AI is data-hungry in a way classical control is not, whether you are doing reinforcement or imitation learning on simulated or real robot data. In simulation, you can “cheat” (this is good cheating!) by bypassing a lot of the middleware like ROS. Therefore, at the SIG PAI we mostly care about on-robot data collection using real hardware drivers.
On the data side, we are looking at three areas: data collection, performance, and compatibility with existing learning frameworks.
Data collection: To collect demonstrations, you can either directly teleoperate a robot, or use standalone devices such as UMI, later retargeting the data to the robot’s kinematics. Teleoperation is more relevant to us at the SIG PAI as it involves moving robots. Teleoperation devices fall into two camps. Some are joint-space: a leader arm (for example GELLO, or a kinematically-matched twin of the follower) streams joint targets you can feed almost directly into ros2_controllers. Others are Cartesian: a VR handset, a phone, or a gamepad gives you end effector pose targets, which then can go through inverse kinematics solvers such as Pink to output joint targets.
Either way, we rarely want to command a stiff position setpoint against a human operator or a contact-rich task. Instead, we want compliant control. Controllers like crisp_controllers provide Cartesian impedance and operational-space control on top of ros2_control. By the way, the payoff is double: the same compliant controllers that make teleoperation safe are what later track and validate the trajectories output by your learned policies.
Performance: Camera frames, tensors, and tactile signals move at high rates, often between processes on the same machine, and on that single-host path the copies and serialization steps can quietly come to dominate the cost of moving a message.
Native buffer support in rosidl_buffer_backends, available now in ROS 2 Lyrical, tackles this directly. By introducing dedicated buffer abstractions, it enables efficient, zero-copy sharing of hardware-accelerated buffers, like GPU tensors and image frames, between processes without requiring host CPU copies or serialization.
Taking this concept further, a broader messaging redesign for zero-copy transport is captured in REP-0157, “Minimal Overhead Messaging Support Using Runtime Agnostic Memory Layouts“. This effort rethinks how messages are laid out in memory so they can be passed across process and language boundaries with minimal overhead, accompanied by a work-in-progress reference implementation at zero_copy_rosidl.
Additionally, there is ongoing standardization of messages for tensors, tactile data, and other types that Physical AI treats as first-class.
Compatibility: Training frameworks expect data in their own shapes. The de-facto format in imitation learning today is Hugging Face’s LeRobotDataset (synchronized video plus action and state streams). The native recording format in ROS 2 is the rosbag, so something has to bridge the two. For example, Rosetta interfaces ROS 2 topic streams directly with ML frameworks like LeRobot, turning live demonstration recordings into structured training datasets without bespoke export scripts. It is built framework-agnostic on purpose, so ROS data can target a variety of formats.
Aside: Coming back to Simulation
We mentioned “cheating” in simulation by avoiding middleware altogether, but that’s not always the case. Simulation can help with large-scale data collection, but it is also great for testing and validation. For some tasks, you may want your simulation to more faithfully reproduce on-robot behavior and all its distributed-system goodness and latency, though it may not be exactly the same one you used for model training.
For example, simulators such as Gazebo and MuJoCo (through mujoco_ros2_control) provide a good way to simulate on-robot behavior. On the other hand, large-scale training frameworks include NVIDIA Isaac Lab (built on Isaac Sim) and mjlab (built on MuJoCo). Realistically, you may be looking at using a handful of simulators in your Physical AI journey.
Simulators pull assets (meshes, robot descriptions, etc.) across many sources, and today those assets rarely survive the trip between simulators intact. Work on scene and embodiment interoperability aims to fix that: OpenUSD conventions in REP-0158 and an evaluation of the HCDF format. The goal is a single asset that behaves consistently across Gazebo, MuJoCo, Isaac Sim, and the rest, with its ROS interfaces (topics, frames, joints) described declaratively in the asset rather than reimplemented per simulator.
Running Policies On Your Robots
The other half of the picture is figuring out what to do once you have a trained model. This is going to heavily depend on which type of models you are looking at for your application. Broadly, we can divide models into a few categories.
- Low-level control policies (~100-1000 Hz): Relatively small models trained from mostly proprioceptive data (e.g., robot positions, velocities, forces), but are increasingly including sensor data as we figure out how to make these models run fast. They are usually trained using reinforcement learning, and learned from large-scale simulation data with the goal of sim-to-real transfer on hardware.
- Mid-level action policies (~0.1-100 Hz): Usually trained from a mix of proprioception data and sensor data (images, point clouds, tactile sensors, etc.) Because of their rate, they often return action chunks, or trajectories the robot must follow with some low-level controller or policy. These models are usually trained from real data via imitation learning, either by directly teleoperating the robot or by retargeting other data (often cheaper to collect) to the robot’s kinematics.
- High-level decision-making models (<1 Hz): These models typically do not return direct actions for the robot to take, but rather take in vast amounts of information (human language prompts, semantic maps, etc.) and can direct the behavior or mid- or low-level policies, for example by steering them towards a specific goal or mode or operation. You will often see large, pre-trained models at this level, most notably for agentic workflows.
In the SIG PAI, we are exploring two workflows for running models: prototyping and performance.
For prototyping, we are prioritizing the ease of testing out a trained model on a robot. Conceptually, the easiest path is to stay in Python, where most of the popular ML frameworks reside today (for example, PyTorch or LeRobot). We are working on providing guidance for how to best create Python inference in a way that makes the official ROS Python client library (rclpy) interact well with the CPU-GPU transfer mechanics of frameworks like PyTorch.
For performance, which may be driven by some combination of production-grade and simply needing to run policies at high rates, there is another path. This is where running inference as a controller inside the ros2_control stack comes in, together with a standalone inference node, in progress at ros2_policy_execution_pipeline. The idea is to execute policies through portable runtimes such as ONNX rather than tying deployment to a training framework, and to put inference where the real-time control loop already lives, so the policy participates in the same update cycle as the rest of the controllers rather than being bolted on from the outside.
An alternative, more decoupled path for inference living in ROS is taking advantage of the native-buffer support we mentioned earlier. In the case of a ROS 2 controller, this means the C++ controller implementation would simply be in charge of passing tensors around, zero-copy, and directly talking to the hardware drivers at a fast rate. On the other side, there will be a specialized inference node running the models on GPU.
Higher up the stack, SIG PAI addresses agentic reasoning: how autonomous AI agents deliberate and act through embodied systems running ROS. A key effort driving this is the ros-mcp-server project, which uses the Model Context Protocol (MCP) to bring agentic AI capabilities into ROS ecosystems. This approach bridges modern AI with traditional robotics: rather than expecting an LLM to reinvent low-level motion planning, the agent orchestrates skills from trusted ROS packages that have proven themselves for years, if not decades, on real robots.
Tying it together: a reference stack you can run
Individual pieces are useful, but it is important to see them working together, end to end, on an actual robot. That is what the ros-physical-ai/demos repository is for. It is an integrated, reproducible reference stack that pulls these efforts together, and it doubles as the place where they get integrated in practice, whether that is an inference approach or a transport improvement.
The current showcase runs on the SO-ARM101, a low-cost educational arm, using a combined ROS and LeRobot approach through the Rosetta bridge. We’ve collaborated with Jafar Uruç (@JafarAbdi) to adopt his implementation under the ros-physical-ai organization as the canonical ROS driver stack for the hardware. It walks the whole loop: leader-arm teleoperation to record demonstrations, data collection over rosetta, policy training, and on-robot inference. This single workflow is runnable in simulation with Gazebo or MuJoCo, or on real hardware, which is a testament to the modularity offered by ROS.
If you want to see the pipeline work without recording and training anything first, there is a genuinely low-friction path: a pre-trained ACT policy you can run in Gazebo in minutes, using pre-recorded rosbags, a converted LeRobot dataset, and a trained checkpoint hosted on the Hugging Face Hub. Clone the repo, follow the quick start, and watch the arm run a policy on your own machine. It is the fastest way to get a feel for what “Physical AI on ROS” actually means in practice, and it is a good on-ramp if you are thinking about building something similar.
The SO-101ARM reference application is designed for both physical and simulated robots (supporting MuJoCo and Gazebo) while utilizing the exact same pipeline across both environments.
What’s happening now, and what’s next
None of this is standing still. The zero-copy transport work is moving from proposal toward a maturing reference implementation, and the scene-description standards are advancing through their draft stages toward ratification. The data collection pipeline is evolving to support workflows beyond LeRobot, with flexible, plug-and-play modules for online/offline data collection and processing. The reference stack is growing too: the SO-ARM101 demo is the first of many reference applications.
On the near-term roadmap are:
- Influencing the core ROS roadmap: Part of the SIG’s job is to feed concrete guidance to the ROS PMC on where the core itself needs to move for Physical AI.
- Meeting researchers where they are: Lowering the barrier to entry with easy-to-configure projects (for example, via Pixi and RoboStack) and first-class interoperability with the simulators the AI community already lives in, such as MuJoCo and Isaac Sim.
- More production-grade reference applications: Beyond the SO-ARM101, an industrial-grade example inspired by the AI for Industry Challenge hosted by Intrinsic and Open Robotics, and expansion to other embodiments like mobile manipulators, legged robots, and humanoids.
- Reinforcement learning and higher-level reasoning: Extending past imitation learning into RL workflows, and into models for higher-level reasoning, with the ROS MCP server as one direction.
- Continued integration of model inference and data transport work as it lands.
- … your own idea(s)!
If any of this is close to what you care about, here is where to look:
- The SIG and its working groups: physical-ai.ros.org
- Meetings are public and posted on the OSRF Calendar, most groups meet biweekly.
- The code and the organization: github.com/ros-physical-ai
- Public channels on Open Robotic’s Zulip:
- The reference stack you can run today: ros-physical-ai/demos, including the pre-trained policy you can try in Gazebo in minutes.
- The standards being drafted, open for comment: REP-0157 and REP-0158
ROS needs you!
None of this is finished, and that is the point. The REPs are drafts, the reference stack is a work in progress with an open roadmap, and the gaps are written down in public for anyone to pick up. The improvements land wherever ROS needs them: in the transport, the message types, the scene and embodiment descriptions, the data pipeline, the inference path, and the reference stacks, as drafts you can read and repositories you can clone.
We also want to hear from organizations that aren’t in the SIG yet. Whether or not you build on ROS today, your take on where Physical AI workflows actually break down would genuinely help shape this work. Come tell us what works and what does not.
ROS has evolved through every era of robotics so far, and this wave is the most exciting yet: robotics is taking a massive step towards solving more challenging real-world tasks by applying data-driven Physical AI methods. ROS can be an integral part of that journey, as it has been in the past. What made ROS matter was never a single feature, it was a community that kept building the things they needed, in the open, together. That is still how this works, and the working groups are simply the current shape of it. Come build it with us.
Join us in Toronto on September 22-24, 2026 for ROSCon Global to learn even more about SIG PAI and physical AI in robotics and the ROS ecosystem. Speak directly with the SIG PAI members at their birds of a feather session on the 22nd, then continue your physical AI learning with sessions on the subsequent two days, including a deep dive into industrial-grade AI with lessons from the AI for Industry Challenge. We look forward to seeing you there!
