An Innovative Mixed Light Field Technique for Immersive Projection Mapping

A novel mixed light field technique that utilizes a mix of ray-controlled ambient lighting with projection mapping (PM) to obtain PM in bright surroundings has been developed by scientists at Tokyo Institute of Technology. This innovative technology utilizes a novel kaleidoscope array to achieve ray-controlled lighting and a binary search algorithm for removing ambient lighting from PM targets. It provides an immersive augmented reality experience with applications in various fields.

Projection mapping (PM) is a fascinating technology that provides an immersive visual experience by projecting computer-generated images onto physical surfaces, smoothly merging real and virtual worlds. It allows us to experience augmented reality without the need for special glasses. As a result, PM is in high demand in various fields including enhanced stage productions, trying on clothing and make-up, and educational demonstrations.

Despite its potential, current PM methods face challenges in bright environments with ambient lighting. Ambient lighting drowns the entire scene in light, reducing the contrast of PM targets. This is why conventional PM solutions mainly function in dark environments. However, even in dark environments, PM fails to provide a natural scene, as in a dimly lit scene, only the PM target is well-lit while the rest of the surroundings remain dark, causing it to appear overly bright. Additionally, non-PM objects appear too dark, breaking the immersion.

To address these issues, a team of researchers from Japan, led by Associate Professor Yoshihiro Watanabe from the Department of Information and Communications Engineering at Tokyo Institute of Technology, has recently developed an innovative new mixed light field approach for achieving PM in brightly lit environments. “In this approach, instead of using normal ambient light, we introduced a mixed light field in which a ray-controllable light avoids the PM target while adequately lighting other areas within the scene while the PM projector exclusively illuminates the target,” explains Dr. Watanabe. Their findings were published in the journal IEEE Transactions on Visualization and Computer Graphics and was presented at the 31st IEEE Conference on Virtual Reality and 3D User Interfaces in Orlando, Florida USA, with the presentation scheduled for March 19th at 1:30 PM local time (UTC-4).

At the center of this novel approach lies the ray-controllable lighting unit. This unit reproduces a range of ambient lighting scenarios while also avoiding illuminating the PM target. To achieve this, the researchers developed a novel kaleidoscopic array comprising an array of mirrors positioned behind a lens array, which, in turn, was placed in front of a projector. This setup allowed the projector to produce a high-density light field, crucial for ray-controllable lighting.

Furthermore, to avoid illuminating the PM target, the researchers deployed a camera to capture images of the scene and identify the pixels from the projector that illuminated the PM targets, subsequently turning them off. To identify these pixels, they employed a simple binary-search-based method, resulting in effective mixed light field.

Figure 1.

The entire configuration of the system (left) : from left to right, the projection target, mirror-lens array, projector for projection mapping, and projector for illumination. A close-up of the mirror-lens array. This configuration reflects rays emerging from the lens array, leading to an increased ray density (right).

This innovative approach allowed them to achieve high-contrast PM presentations in brightly lit surroundings. Notably, it preserved the natural appearance and shadows of ordinary non-PM objects, addressing a key challenge in PM technology. Through several captivating augmented scenes, the researchers showcased the seamless coexistence of PM targets and ordinary objects, providing an immersive visual experience.

Figure 2.

The high-contrast projection mapping achieved by the proposed method, even under ambient light conditions (left). The conventional projection mapping in a bright environment where the model on the left is overexposed (top right). An unnatural situation where only the mapping target is brightly lit in a dark environment (center right). The projection mapping with ray-controllable ambient light, ensuring light does not cast on the projection mapping target (bottom right).

While the researchers identified some limitations, such as artefacts and the low efficiency of binary search algorithms with large PM targets, they have already identified potential solutions and are actively working to expand this approach in the future.

“Our experiments prove the effectiveness of using this technique for achieving natural PM presentations with accurate lighting for all objects. Mixed light field has the potential to usher PM for various practical day-to-day applications, such as attractions, for support in manufacturing and trying on make-up,” says Dr. Watanabe, highlighting the applications of their technology.

Overall, this approach marks a significant step for PM technology, paving the way for immersive augmented experiences in future.