Immersive Lab

Tuesday 1 June 2021

The Immersive lab is an immersive projection space, set up at our office in Amsterdam Noord. In it, we want to create and experiment with new things which are fun, immersive, interactive and visually beautiful.

Why we are dedicating this much office space to it

We love to experiment with new technologies, to play around, invent, fail, try again and most importantly have the space to immediately test out new ideas.

We have more and more clients ask for immersive and interactive projections, so it is very useful to have a dedicated set-up at our very own office.

We like sharing and collaborating. This spring, students from Amsterdam University of Applied Science’s minor Immersive Environments are using our space for a study project. Let’s see who we will host next month?

Experiments with new technologies in our Immersive lab

Students from Amsterdam University of Applied Science using our space for a study project

How we set it up

First, we tidied up our R&D room and bought lots of white paint.

Secondly, we bought a whole lot of hardware.
The Immersive lab consists of:
- 6 x Projectors (5x Canon WUX 450ST; 1x Canon WUX 500ST)
- 6 x realsense cameras (D455)
- 2 x high end computers
- Speakers (2.1 Sound System)

Thirdly, we did some prototyping and testing on the layout and dimensions before finally installing the hardware.

When the physical set up was ready, we developed two pieces of in-house software to set up the space!
YIPP developers trained a neural network with tens of thousands of top view depth images of people in all sorts of poses and people of all sorts of heights. As a result, the Immersive Lab has a tracking system that can correctly detect a “person” from above.

Next to the tracking, we wrote a piece of software that pieces all the images from the 6 cameras into one single 3D image of the room

3d Layout Test

Physical Layout test

From Neural net hand detection tests to reallife tests

What can it do?

Key features of the tracking system:

- Detect the location of a user’s head, as well as left and right hand.
- Tracking happens from above and is not disrupted by users walking in front of one another. Due to this, the space can be enjoyed by several people at the same time.
- Consistently track individual users in a very large area as that one user, for as long as they stay visible in the system and do not step out of the space.

We can use this information to:

- Show hidden content in the projection triggered by user’s hands (by using the detect hand function and have this trigger any type of interaction).
- Store and display info for unique user e.g. score for a user, text in language choice of user (by using the individual tracking of users).
- Make an educated guess to viewing direction (by using the detected left vs right hand).
- Roughly detect a user’s height (by using the head position).

The space can easily be used to create interactive, as well as non-interactive immersive projection experiences. Due to the setup of the space, all development projects need to be made in Unity. We have also set up a feature, that images and videos can be “uploaded” and projected for immersive, non-interactive projects without any code involved to make it happen.

Zoom

4 fun projects we did

Necessity is the mother of invention?
What game do you make at the very beginning when only head tracking works? Exactly, a multiplayer chicken game where you need to mama bird feed your little chicken with worms that you head butt on the floor. Some colleagues had sore muscles for days...

Feed the bird

Always wanted to take a stroll in your own art?
In this non-interactive projection, our colleague put her daughter's painting in the space so she could play in her own world. For this, she simply altered and uploaded the image into our system and voila!

non-interactive projection

You think Tetris is hard?
Wait until you've tried our Tetris. Step onto the control on the floor to move your blocks around...

Tetris

Hello Fish!
For a client in Belgium we've created a beautiful underwater world. Our favourite detail? The little fish on the floor projection that follow you around everywhere you go.

Following Fish

The future

In the future we want to tweak and train the neural net more, realise beautiful and fun projects together with clients and share the space with more enthusiastic people.

Are you interested in using the space, or you wanna talk to us about immersive projection projects?
Get in touch with Wouter van der Zouwe via info@yipp.nl

Testing with 3d in the immersive lab