Sending a friend a virtual birthday present, or quickly beaming a new product over to a customer in America to try out – it sounds like science fiction, but this is what researchers at the Computer Vision Lab want to make possible, with the aid of new technology. Their first step was to successfully transmit a virtual object to a spatially remote person, who could not only see the object, but also feel it and move it.

Incorporating all the senses

The more senses are stimulated, the greater the degree of immersion in the virtual reality. While visual and acoustic simulation of virtual reality has become increasingly realistic in recent years, development in the haptic area, in other words the sense of touch, lags far behind. Up to now, it has not been possible to touch the virtual copy of an object, or to move it. The aim of the EU project “Immersence”, in which ETH Zurich has also been involved, was to develop new methods for haptic interaction. Matthias Harders, together with other scientists from the Computer Vision Lab, led the sub-project which dealt with interaction between people and virtual objects.

3D in real time

The researchers developed a method for combining visual and haptic impressions with one another. Whilst a 3D scanner records an image of the object, which in one experiment was a soft toy frog, a user simultaneously senses the object using a haptic device. The sensor arm, which can be moved in any direction and is equipped with force, acceleration, and slip sensors, collects information about shape and solidity. With the aid of an algorithm, a virtual copy is created on the computer from the measurements – even while the toy frog is still being scanned and probed.

Combining vision and touch

The virtual copy can be sent to another person over the Internet if desired. In order for this other person to be able to see and feel the virtual frog, special equipment is needed: data goggles with a monitor onto which the virtual object is projected, and a sensor rod which is equipped with small motors. A computer program calculates when the virtual object and the sensor rod meet, and then sends a signal to the motors in the rod. These brake the movement that is being made by the user, thereby simulating resistance. The user has the sensation of touching the frog, whilst from the outside it appears that he is touching air.

In order to intensify the impression of reality even further, the virtual frog can be projected into the actual environment, where, for example, it appears to be sitting on the table in front of the viewer. The researchers have already achieved this superimposition of the real and virtual worlds, which is termed “augmented reality”, in an earlier project. There, they developed a ping-pong game in which only the handle of the bat exists in reality. With a virtual striking surface, two players can bat the virtual ball to one another. “It feels quite similar to a real game,” says Matthias Harders.

Technology of the future

Whereas earlier attempts simulated the virtual object largely on the basis of assumptions, the method developed by the ETH researchers is based more heavily on measured data. “Our approach can be viewed as an extension of photography,” explains Harders. The method is particularly advantageous in the case of complex objects which would be difficult to describe with a model.

Up to now, it has been possible to touch a virtual object, but not to grasp it. For this, special sensor gloves are needed, with which the user can both “scan in” a real object and touch a virtual object. The ETH researchers are still working on the development of these. However, Harders believes that this technology could be as widely used in twenty years’ time as the Internet is today.

Attending conferences by teleporter

In their next EU project, whose name, “Beaming”, is reminiscent of the Starship Enterprise, the scientists are going a step further. They are planning a method of teleportation, in other words for transporting people. “Only virtually, of course,” Harders assures us. This would, for example, make it possible to take part in a conference without having to be present in person, which would save time and travel costs. At the same time, interaction with other participants would be more intensive than is possible in video conferences, for example. “The idea is to generate the feeling of sitting together with other people at a table”.