Jesse Robo

Loading MuJoCo (about 10 MB)…

Robot

Jesse runs a 110-servo humanoid

A 0.9 m humanoid with 110 servos, in the proportions of a human body. Each leg has 10: a three-axis hip, a knee with a little twist, a two-axis ankle, a midfoot hinge, and big and lesser toes. The spine has four three-axis segments and the neck two, and a jaw and two moving eyes sit in the head (her camera is her right eye). Each arm has 30: a three-axis shoulder girdle, a three-axis shoulder, an elbow, forearm twist, a two-axis wrist, and a 23-joint hand whose fingers spread and flex, whose thumb opposes, and whose palm cups. Nothing passes through anything: every part collides with the ground, other robots and itself. Jesse learns to stand through shoves, walk, walk to a marker she finds with her head camera, cross rough floor, and pass oncoming walkers by deciding from what she sees whether to proceed, adjust or stop. The world runs on real-life physics time; the achieved rate is shown top left (one robot runs about 20–25× real time; the 20-robot crowd about real time). Every point on the success curve is scored on held-out layouts she never trains on. A task passes at 17 of 20.

Model

Loading ssf001…

Curriculum

Held-out success vs training attempts

Walking speed on held-out tests

Median forward speed in each held-out walking test round, in metres per second, over total practice attempts.

Practice attempts needed to pass

Walking skills carry forward, so later tasks can need fewer attempts. A pass on the first test with zero practice means pure transfer.

Live training

–

body tilt

–

forward speed

–

camera looks this attempt

–

episodes run

–

real-life time practised

–

running at

What she has worked out

Orange tick: current mean. Gray band: how widely she is still exploring.

Controls

How it works

What is learned, and what is given

Physics: MuJoCo 3.14 (DeepMind's official WebAssembly build) at 250 Hz. There are 110 position servos with torque limits (30 N·m in the legs, 30–45 N·m per spine joint, 12 N·m at the shoulder girdle, 5–8 N·m in the shoulders and elbows, 3 N·m at the wrists and toes, 0.3 N·m per finger), realistic link masses, joint damping and foot friction. Every body collides with the floor, rocks, other robots and its own other parts. The only exclusions are shapes that overlap by design at a joint. A 6 mm contact margin and stiff contact impedance keep contacts from sinking in (fingers: 2 mm, under half their 5 mm gap). The measured worst penetration over single-robot walks and falls is 0.00 mm, and 0.00 mm over the crowd's first 10 s. In a later 20-robot pile-up, two robots' little fingers meeting at arm-swing speed overlapped 3.8 mm for a single 4 ms step before separating. That is a time-step limit: a 2 ms step removes it (0.17 mm) but halves the speed. Floor grip and shoves vary per layout.
Given: the structure of the controller. A gait clock drives leg inverse kinematics. Balance feedback uses her tilt sensor, and foot placement depends on her body velocity. Steering turns the hips. That is the shape of the controller, not its numbers.
Learned by practice: all of its numbers. That means standing and walking balance gains, hip height and lean, step period, length and lift, hip sway, arm swing, foot-placement gains, steering gain, how often she looks, and rough-floor lift, step, ankle stiffness and pace. Every skill starts at the middle of its allowed range, and nothing is hand-tuned. She samples variations, keeps what scores well (cross-entropy method), and carries shared walking skills into later tasks.
Seen, not given: the marker's position comes only from her head camera. Each look ray-casts a 64×40 image through MuJoCo and finds the marker-coloured pixels. It back-projects them with depth and averages the looks. Once she has a fix, she only re-casts a window around where she expects it (foveation). The bottom-right view is that camera, with her detection outlined.
Seeing to pass: in the pass task and the crowd, each robot scans a depth strip through its own head camera, ray-cast against the other robots. It proceeds if the forward corridor is clear beyond a learned distance, and adjusts if something is nearer: it slows and steers into the free gap nearest its goal. It stops and steps in place if something is too close. The distances, gains and look rate are learned by practice. Oncoming walkers are copies of her running the same skill. The POV inset shows the scan and her decision.
Whole-body control: beyond the legs, the spine counter-rotates the chest against the legs and bends against pitch and roll (hip strategy), spread over its four segments. The shoulder girdle follows the arm swing and lifts against sideways tilt, and the arms swing and reach out against tilt. The neck steadies the head, and a vestibulo-ocular reflex counter-rotates the eyes (and so her camera) against head pitch. Feet roll heel to toe, knees soften as they take load, and the toes grip through stance. Fingers stay in a relaxed curl that follows the arm swing. Each of these reactions has a signed gain that starts at zero: she begins with the plain gait and learns which reactions help and in which direction. Wrists, forearm twist, finger spread, palm cupping, thumb opposition and jaw are held neutral for now: they are there for manipulation and expression, not walking.
Scoring: after each round of 16 practice attempts, the current skill is frozen and tested on 20 held-out layouts. A task passes at 17 of 20. Each practice candidate from task 2 on also rehearses an earlier task. After the last task, consolidation checks everything on separate validation layouts. The final exam runs once, on 20 fresh layouts per task.