A lightweight bracelet that can continuously track 3D hand movements could bring activity tracking to the next level, believe researchers from Cornell University and the University of Wisconsin-Madison.
The bracelet, which is called FingerTrak and incorporates four miniature, low-resolution thermal cameras, has the potential for a variety of uses including automated sign-language translation.
The research is still at an early stage, but the researchers think it has a lot of potential. For example, if synced to a smartwatch it could allow hand-gesture initiated commands in a similar way to voice-activated commands from Siri on an iphone.
It could also have health implications for people trying to control habits such as overeating, smoking and drinking, if the device can learn to warn someone when they are about to do something unhealthy, or be linked to another device that could alert the user. It also has the potential to warn users about symptoms of diseases such as Parkinson’s or Alzheimer’s that impact fine motor skills and can be detected through abnormal hand movements.
Cheng Zhang is an assistant professor at Cornell University and runs the aptly named SciFi (Smart Computer Interfaces for Future Interactions) lab. He led the research to develop the FingerTrak device. He explained that while hand tracking has been attempted before, it has been difficult to make devices portable or convenient enough for easy use.
“Most of the previous hand tracking technology uses a camera in the environment, or a glove, which may not work well in a mobile setting. Because the camera is not always available and people may not like wearing gloves all day long.”
The FingerTrak device combines thermal imaging from the four different angles with machine learning. Multiple still images from the cameras are then merged together using a type of artificial intelligence network to create a continual 3D outline of the hand in real time — both when empty and when holding a variety of different objects.
Although there are a lot of activity trackers, smart watches and bracelets on the market right now, many of them are limited in the accuracy and kind of information they can collect, particularly related to how we move.
“Right now, activity recognition is running into a bottleneck… we have a lot of technology trackers that can sense human activity, but many of them do not work well in daily activities,” says Zhang. “The information they get on people is not accurate enough or the resolution is super low. So, you can get how many steps you do, but not really more than that.”
He thinks one of the key things the FingerTrak device has the potential to do is to help computers to learn about how humans use their hands during the day and to help link certain movements with regular activities such as eating, drinking, smoking and other health-related behaviors such as cleaning your teeth.
Another potential use could be to help people who have lost a hand or fingers to control a prosthetic limb. However, Zhang says there are some limitations with this, as the device needs to be able to visualize the hand to give effective readings.
“Our technology is built on the assumption that the really small cameras can see the contours of your hand. If the contour does not change our device may not work.”
One use that has real potential is as a wearable translation device for deaf people who use sign language to communicate.
“Sign language recognition requires information not only on hand posture but also on arm posture, and facial expression,” explains Zhang. “FingerTrak offers a minimal-intrusive wearable for hand tracking. If paired with other sensing methods, it can potentially build a minimal-intrusive wearable sign language translator.”
While the hand tracker is the most advanced device they have developed to date, Zhang says his lab is working on other sensing devices that could help make automated sign-language translation a reality in the near future.
Outside of health and communication, the hand tracking device also has potential to improve a user’s experience of virtual or augmented reality.
“The current hand tracking methods in virtual reality usually require a camera on the headset or chest, limiting the range of movement of the hand to in front of the chest,” notes Zhang. As the new device has integrated cameras, this opens up a much wider range of possible hand movements.
Learning or evaluating fine manual skills such as surgery, playing an instrument, making jewellery or painting a picture could also be enhanced by using a device such as FingerTrak if paired with the right software and computers.
Zhang and his colleagues think their device could be part of a new generation of tracking and sensing technology. Indeed, they are in early talks at the moment to commercialize the device.
“One of the biggest challenges for wearable health trackers to moving forward is to acquire accurate and continuous information about the user. Right now, most of the trackers can only record your steps, or heart rate, which is great, but not enough for fine-grained activity recognition in free-living conditions,” says Zhang.
He cautions that as devices like FingerTrak become more accurate and can track more activities we need to be more careful about who can access our data.
“As more of these new sensing technologies are being invented, computers can also get more potential privacy information on a user… How to protect the users’ privacy is a challenge that academia, industry, and government have to work on together to address in the future, before these technologies can be widely deployed.”