One Step Closer to Human Intelligence — MIT CSAIL Combine Sight And Touch in AI

Yunzhu Li, a PhD student at MIT CSAIL who led the project MIT CSAIL

Sense and sensibility

With a simple webcam, the team headed by Yunzhu Li, CSAIL PhD student and lead author of on the system, built up a dataset of over 200 everyday objects being touched more than 12,000 times. They then reduced those 12,000 video clips into static frames, and used those to compile “VisGel,” a dataset of more than 3 million visual/tactile-paired images. Using that dataset, the team trained an AI model to predict what an object would feel like based on visual data of the surrounding area, and used a KUKA robotic arm paired with a GelSight tactile sensor to acquire the corresponding tactile information. For instance, the team would feed the system images of a certain point on a computer mouse, and the AI would use a generative adversarial network (GAN) to build a tactile map of the area.

Dexterous digits

Connecting an image feed and a tactile sensor within an AI model represents a fascinating step in the progress of AI systems and robotic arms that experience the world more like us. Giving a dual-insight to a digital intelligence effectively compounds the information and the “knowledge” that this system has access to.

This would theoretically allow an AI system to learn things about its environment and process information much faster and more effectively than a single-input system.

In surgery, for example, robotic arms can currently handle incredibly delicate procedures such as prostatectomies using minimally invasive, or keyhole, surgery, which make up around 86% of all robotic surgeries in the U.S.

Robots that can connect senses together and infer much more about their context and their environment could lead to groundbreaking advances in any number of industries that currently utilize robotic limbs.

This project is still in the research phase, but by showing that this kind of intelligence is possible, the MIT CSAIL team have proved that the field of robotics is still only in its infancy, and there is far more to come before we reach the limit of what robots are capable of.

CEO of Pod Group (@PodGroup_IoT) Author of “The WEIRD CEO”. Advocate of Employee Self Responsibility, #FutureOfWork and #AI. Contributor:

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store