Augmenting Touchless Gesture Recognition with Haptic Feedback

Touchless gesture recognition is widely acknowledged as the next frontier in the development of human machine interfaces and the technology is now beginning to move beyond computer and gaming markets to include consumer and domestic appliances as well as automotive dashboards. However, it is commonly found that touchless controls can lack the precision and tactile feedback that is so valuable to humans to confirm they are properly controlling the system. Vicky Messer, Technical Marketing Manager at Ultrahaptics looks at the market and advocates touchless or mid-air haptic feedback as the next key step for the development of the technology.
Ever since the development of the trackball in the 1940s and the consumer release of the computer mouse in the 1980s, engineers have sought to push the limitations of computer-human interaction in an attempt to achieve a more intuitive and natural user experience. An obvious avenue to explore has been the gestures used in face-to-face communications, and the market has come a long way especially when considering developments in sophisticated audio and visual capabilities. The importance of the human sense of touch has also been championed with haptic feedback added to many consumer devices. Traditionally, however, the two markets have been distinct entities.
According to market analyst firm GIA, the global market for gesture recognition will exceed $12.7 billion in 2020, driven by expanding applications in gaming, healthcare, automation, consumer electronics and automotive sectors. Consumer electronics (including gaming) currently accounts for approximately 99% of the gesture recognition market. A second report predicts the market for both gesture recognition and touchless sensing technologies will have a CAGR of 28% over the next few years and is predicted to be worth $24 billion worldwide by 2020.

Touching and Touchless
Controlling devices with human gestures is, of course, now commonplace: the ability to swipe, pinch, push virtual buttons and type on touchscreens is provided by almost every type of electronic equipment. However, almost all require the physical touching of controls or surfaces. Clearly, this has limitations such as wear and tear of electrical switches and vulnerability to contamination – the classic example being spilt coffee on a computer keyboard. Some less obvious examples include the possibility of cross-infection from medical equipment controls or public toilets, or possible injury from hot or cold surfaces or sharp or electrically charged objects.

Touchless haptic hob control
Controlling equipment via human gestures without touching has become a growing area of interest to major technology companies over the past few years. The enormous global gaming market in particular has driven demand for the detection of body movements at a distance. However, a fundamental weakness of touchless controls is that they do not offer the desired level of precision and tactile feedback necessary to provide the best user experience. Users will prefer to feel a control or different textures and surfaces, and ideally also have audible confirmation. Effective simulation of tactile feedback – haptics – is therefore being seen as the key to unlocking the potential of touchless gesture recognition.

Tactile Feedback
The accuracy and usefulness of human gesture recognition is advancing rapidly and will continue to do so for decades to come. The issues associated with image-based systems are a focus of development efforts, including adaptability to varying lighting conditions and backgrounds and managing unwanted objects in the field of view. However, only seeing and hearing limits human perception and the ability to effectively control computers, which can be a significant problem particularly in safety critical applications. Simply recognising human gestures does not necessarily provide the tactile feedback experienced from physical controls such as switches and dials.
For example, even with the use of predictive text, there is the need for the user to check the screen to ensure the correct key has been pressed when using the virtual keyboard of a tablet, phone or touchscreen. Adding the third sense of touch provides feedback that a key, button or switch has been activated correctly and substantially enhances perception and control. Tactile feedback is even more important in some other applications such as vibrating the steering wheel to simulate the effect of driving over a rumble strip, which warns a distracted driver that their vehicle is drifting from its road lane. In a number of scenarios, tactile feedback will be an absolute necessity for the public acceptance of touchless gesture recognition.

Haptics
Haptic feedback is the sensation of touch and is already well known in computer gaming to provide a more immersive environment, and especially in various applications such as aircraft control where user attention is critical. Even flat-panel touchscreens can use programmed interference patterns between several tiny vibrators around the edge of the glass display to produce haptic feedback at the fingertips and simulate different surface textures.
Many new consumer products now feature haptic feedback – for example, major smart phone makers are now implementing vibrational haptic feedback in some of their latest models – generating a great deal of interest for numerous other applications worldwide. A number of standard APIs (Application Programming Interfaces) are now available for research in the field of haptics, and several of these are being used to produce tactile sensations in simulations and training for remote and keyhole surgery such as laparoscopy and other medical applications.
Although tactile feedback provides a much more complete user experience – it closes the loop to provide helpful non-visual cues – almost all these approaches are limited by the need to be in physical contact with the haptic system. Therefore, they cannot be used in applications based on touchless gesture recognition at a distance.

Evaluation Kit
Providing touchless haptic feedback has been impossible until very recently. Ultrahaptics has developed a range of focussed ultrasound techniques to augment recognition and control systems for hand gestures with touchless or mid-air haptic feedback. The company’s haptics evaluation kit – the only solution for touchless gesture recognition on the market today – includes a ‘leap motion’ camera and gesture recognition system and is allowing developers to experiment and generate novel haptic sensations in touchless gesture control interfaces for a range of devices including computers, consumer electronics, domestic appliances and automotive dashboards.

Summary
Recognising human gestures and using them to control computers and equipment is well proven given the existing market of billions of smart consumer device and other electronic equipment. The advantages of not touching equipment are valuable, if not vital, in many use cases. Touchless gesture recognition has therefore become a well-established market sector and an area of academic and intense commercial research by major global corporations.
There is little doubt that many challenges still remain in perfecting the user experience, including maximizing the range and field of view, and particularly in the accuracy of detection – not only of position and movement – but perhaps more importantly in the correct understanding of a user’s intention. In addition to incremental improvements from enhanced software algorithms, plus more accurate lower-cost and lower-power camera and sensor hardware, tactile feedback adds another new dimension to the accuracy of touchless gesture recognition in interpreting a user’s commands and confirming the correct action.

1. Global Industry Analysts Inc. – Gesture Recognition, June 2015
2. Markets and Markets – Gesture Recognition & Touchless Sensing Market, June 2015