Ultrasound imaging turns a robot hand into a skillful mimic
Why this matters: a development in AI with implications for how people work, create, and decide.
Our hands are the nimblest parts of our bodies, coordinating 34 muscles, 27 joints, and over 100 tendons and ligaments to perform countless nuanced movements and gestures. So far, robots have been notoriously bad at mimicking that dexterity, in part because researchers struggle to capture what is actually going on under our skin in order to reproduce it. Now MIT researchers are pioneering a promising new approach. Mechanical engineering professor Xuanhe Zhao and colleagues at the Institute and the University of Southern California have designed a wristband equipped with an ultrasound “sticker”—a miniaturized version of the transducers used in medical offices, paired with a hydrogel that can safely adhere to the skin. As the wearer’s hand moves, the device produces ultrasound images of the wrist’s muscles, tendons, and ligaments. Then an artificial-intelligence algorithm, trained on ultrasound images meticulously labeled by humans, continuously translates the images into the corresponding positions of the five fingers and the palm. “The tendons and muscles in your wrist are like strings pulling on puppets, which are your fingers,” says Gengxi Lu, a former MIT postdoc and one of the lead authors of a paper on the work. “So the idea is: Each time you take a picture of the state of the strings, you’ll know the state of the hand.” The wristband precisely tracks a wearer’s hand movements in real time.MELANIE GONICK In demonstrations, the team has shown that a person wearing the wristband can wirelessly control a robotic hand. As the person gestures or points, the robot does the same. In a sort of wireless marionette interaction, the wearer can manipulate the robot to play a simple tune on the piano and shoot a mini basketball into a desktop hoop. With the same wristband, a wearer can also manipulate objects on a computer screen—for instance, pinching the fingers together to enlarge and minimize a virtual object. The researchers are planning to further miniaturize the wris