Jesse Learn 3D

Loading physics…

Jesse runs a gantry arm

Jesse starts knowing nothing about the arm's masses, friction or lag, or how to grip, place, or insert a peg. She learns from her own sensors and attempts. Every second of the world runs on real-life physics time: at 100× you watch 100 seconds of her practice every second. Every point on the success curve is scored on held-out layouts she never trains on. A task passes at 17 of 20.

Curriculum

Held-out success vs training attempts

Motor precision: fingertip tracking error

Median error between where Jesse meant the fingertip to be and where it went, measured in each held-out test round (log scale). Her motor model is frozen during tests, so this shows what it had learned about gravity, inertia and friction at that point.

Practice attempts needed to pass

Shared reach, grasp and place skills carry forward, so later tasks should need fewer attempts.

Live training

–

fingertip error

–

peak torque use

–

wrist force

–

grip force

–

payload estimate

–

episodes run

–

real-life time practised

–

running at

What she has worked out

Orange tick: current mean. Gray band: how widely she is still exploring.

Controls

How it works

What is learned, and what is given

Physics (hidden from Jesse): the arm has link masses, gravity, joint friction, motor lag and torque limits. Objects are cannon-es rigid bodies at 120 Hz, the same engine as the body lab. Cannon-es cannot keep a tall stack of boxes still on its own, so blocks that have rested in contact for 0.3 s are held by a breakable static-friction joint (4 N). It breaks at 3 mm of slip, so a knock or a bad placement still topples the tower.
Learned online: a model for each joint of the torque needed for any motion. She fits it by recursive least squares from joint encoders, motor torque and a wrist force/torque sensor. It starts at zero.
Learned by practice: skill parameters such as close-look height, travel speed, approach height, grip depth, grip force, release gap, and the peg-search spiral, contact force and compliance. She samples variations, keeps what scores well (cross-entropy method), and carries shared skills into the next task.
Seen, not given: every object pose comes from her wrist camera. The page ray-casts a 96×72 depth and segmentation image from the real scene, and she finds the objects in it, back-projects them with depth, and fits position and rotation. She has to move the arm to look: a wide survey, then a close look whose height she learns. Her camera's hand-eye calibration is slightly off, as a real one would be. The bottom-right view is exactly that camera, with what she detected outlined.
Given: arm geometry (inverse kinematics), the step order of each skill (look, reach, grasp, lift, place, search), feedback gains, and the sizes of the object types.
Scoring: after each round of 16 practice attempts, the current skill is frozen and tested on 20 held-out layouts. Nothing is learned during tests, not even the motor model. A task passes at 17 of 20. From task 2 on, each practice candidate also runs one earlier task (rehearsal), so shared skills are not forgotten. After the last task comes consolidation: every candidate runs all four tasks and is checked on separate validation layouts. The final exam then runs once, on 20 fresh layouts per task.

Measured headless on this exact code (seeds 1, 3, 5, 7 so far): all four learn every task, including the ten-cube tower (16–224 practice attempts). Seed 1 then passes the final exam on everything (tower 18 of 20). On seeds 3, 5 and 7 the tower slips to 10–13 of 20 by the final exam, because later peg practice shifts grasp skills the tower shares. Her trained tower success is about 80%, so the final bar is close to her ceiling. This page starts on seed 1. Browser runs can differ in detail from Node, because tiny floating-point differences between JavaScript engines grow over a chaotic simulation.