Separate the stack
Robotics papers often compress a full system into one success number. Unpack it into perception, state estimation, planning or policy, low-level control, and hardware execution. A vision-language model may choose a high-level action while a classical controller handles motion. A learned policy may output joint targets while safety software clips them. Knowing which layer changed is essential for understanding the contribution.
Mental model: robotics performance is a chain. Better semantic reasoning cannot rescue bad calibration, and precise control cannot rescue a mistaken scene representation.
Demonstration learning is not deployment
Behavior cloning learns to imitate actions observed in demonstrations. It can work well near the training distribution, but small deviations compound: once the robot enters a state absent from the demonstrations, the policy may have no recovery behavior. Interactive data collection, corrective demonstrations, reinforcement learning, or explicit recovery policies address different parts of this problem.
Ask how demonstrations were collected, how diverse operators and scenes were, whether failures remain in the data, and whether train and test setups share objects, backgrounds, cameras, or trajectories. Dataset hours are a poor proxy for coverage.
Contact changes the problem
Tasks such as insertion, folding, opening, and tool use depend on forces that images may not reveal. Position error that is harmless in free space can break an object during contact. Force or torque sensing, compliant control, tactile feedback, and carefully limited velocities are not implementation details; they determine which tasks are feasible and safe.
Simulation is an instrument, not the world
Simulation enables repeatable training and large-scale evaluation, but every simulator chooses approximations for friction, deformable objects, sensor noise, and dynamics. Domain randomization can improve robustness only across the variations it actually samples. “Zero-shot transfer” is meaningful when the real setup was not used for training or tuning, but hardware choices and calibration procedures still matter.
A strong paper reports both simulated and physical results, explains discrepancies, and describes the real evaluation protocol: number of trials, reset rules, object placements, human interventions, and failure definitions.
Success rate needs a denominator
Ten successful videos do not establish a 100% policy. Look for repeated trials across initial conditions. Long-horizon task success multiplies stage reliability; a policy that succeeds at each of five stages 90% of the time has a much lower chance of completing the whole sequence without recovery. Progress metrics can be useful, but they should not be mistaken for full completion.
Latency and control frequency matter
A policy that takes seconds to produce an action may be acceptable for deliberative planning but unsuitable for dynamic contact. Distinguish model inference rate from the lower-level controller rate. Check whether timing was measured on the claimed hardware and whether communication, image encoding, and safety checks were included.
Questions to ask when reading a paper
- Which layer changed? Perception, representation, planning, policy learning, controller, sensor, or data?
- What was held constant? Hardware, demonstrations, model size, compute, and evaluation scenes?
- How many physical trials? Were resets and human interventions counted consistently?
- What is truly novel at test time? New objects, positions, tasks, rooms, robots, or only new episodes?
- How does it fail? Perception error, unreachable plan, unstable contact, timeout, or no recovery?
A small evaluation you can trust
For a minimal tabletop task, define a fixed workspace, observable success condition, and safe stop. Record calibration, object placements, policy latency, intervention count, and every failure video. Run enough randomized trials to expose variation, then repeat after changing one factor such as lighting or object pose. Compare with a simple scripted or teleoperated baseline before claiming the learned policy adds value.
Bottom line
Robotics research becomes convincing when the system survives contact with variance. Look beyond the best demo to the interfaces between learned components, control, hardware, protocol, and recovery.