Tag: unity

To make Spectator View work in our own HoloLens project, we actually have to understand how it’s working, what it is doing and how it is related to the HoloLens Sharing Experience. Turns out that there is a lot to do in our app (including transmitting custom messages) to prepare it for the full Spectator View experience!

Synchronizing Objects: Anchor & SceneManager

The Spectator View is based on the Sharing experience of the HoloToolkit, but it contains its own “fork” of the code. Instead of us having to manually decide & code which objects and interactions to send via network messages (as in the normal Sharing use case), the fork of the Spectator View transmits some data by default to our DSLR-mounted HoloLens.

However, it’s important to understand that if you use Spectator View, you also need to integrate most of the Sharing code. Essentially, a lot of code is duplicated between the two frameworks, with only slight differences in order to give the DSLR-mounted HoloLens a special role, plus the PC running Unity also needs to connect in order to correctly render the scene.

In the previous part, we’ve re-compiled Spectator View using the latest HoloToolkit sources. At the time of writing, this resulted in a compile error, as the HoloToolkit for Unity has seen a breaking change since Spectator View was released.

In this part, we’ll first fix the Spectator View code, and then set it up correctly in our own Unity scene. Then, we’ll need to check several other things: how to launch the Sharing Service, adding Internet Connectivity and the Holographic Camera prefab.

Fixing the Spectator View Code

From the two choices at the end of part 4, of course, we go with the adventurous road to fix the code 🙂

Turns out that the AddSurfaceObject()  method from the base class SpatialMappingSource  was changed. Instead of directly creating a Surface Object from a mesh, this process was now split up into two parts. There is one method to create the surface object, and another one to add it.

Getting Spectator View to run with your own app is obviously the most important step. How to include the required tools – including Sharing – into your app using the latest versions?

More specifically, I’ll add Spectator View to my playground project, which is mainly for placing a hologram in the room (with a few other useful scripts). Currently, this hologram is a nice, life-size skeleton from the Unity asset store.

HoloToolkit-Unity & Holographic Academy

The Holographic Academy course 240 explains how to set up your project to include Shared Holograms. It’s a good idea to work through the tutorial, as it will show you some of the basics of what is needed for Spectator View.

However, as with most of the Academy tutorials, it’s quite old. The HoloToolkit has evolved since then, and the demos often have little resemblance to what your project would look like if you start from scratch.

Therefore, for your own project, first clone the HoloToolkit-Unity GitHub repository to your local PC. Open the project in Unity and export the Assets folder to a new package according to the instructions.

Once this is finished, import your newly made, fresh HoloToolkit-Unity package into your own Unity project. Alternatively, you can of course use one of the pre-packaged HoloToolkit-Unity releases, which is slightly older, but is usually a good compromise.

HoloLens Spectator View allows capturing high quality photos including the mixed reality contents as seen by other HoloLens users. To test the complete system setup, the next step is running the supplied sample app in Unity.

Based on my previous two blog posts, calibration of the HoloLens in relation to the DSLR camera has been successfully completed. If you need to complete these steps, check out Part 1 and Part 2 first.

Spectator View Calibration Data

The calibration app saved its calculated data file to your Documents\CalibrationFiles  folder. Copy the CalibrationData.txt  file to the Assets -folder of the Sample project that is part of the HoloLens Companion Kit / Spectator View GitHub repository.

The following screenshot shows the contents of the CalibrationFiles  directory, where the app saved both the screenshots of the Canon EOS camera, as well as from the HoloLens. The txt file in the directory is the data file you actually need for your app.

The next step to getting the HoloLens Spectator View to run is the calibration. This needs to be done once after mounting the HoloLens to the DSLR camera. The output is a calibration info file that lists how the position, field of view and other parameters differ between the HoloLens and the DSLR.

Step 4: Camera Setup

By default, the Canon EOS 5D Mark III outputs some information on top of the live HDMI video, including a big white focus rectangle. That needs to be deactivated in order to get a full-screen, clear image.

Switch the camera to movie mode, and in the camera settings menu, page “SHOOT5: Movie”, set “HDMI output + LCD” to “Mirroring”.

We’re developing several HoloLens apps at the St. Pölten University of Applied Sciences, for example in the area of Healthcare and Radiology, as well as for Smart Engineering. For better demonstrations, the HoloLens Spectator View is crucial: it allows taking 3rd person photos and videos of people using the HoloLens app. That helps in understanding the overall scene setup and the interactions between users and the otherwise invisible holograms.

Step 1: 3D Printing the HoloLens Mount

Microsoft has released 3D printing files for the HoloLens mount, which allows fastening the HoloLens on a digital camera. They recommend using Aluminium for better stability. However, we do not have such a metal-based 3D printer, and using an external company would cost several hundred Euros. Therefore, we decided to go with the standard plastic 3D printing instead.

One issue with the current design is that it’s a bit too large for most 3D printers – the mount is a little bit larger than A4. Luckily, our old 3D printer was able to produce the correct size (3DTouch by Bits from Bytes).