HoloLens–The Path to 60fps (Normal Maps)
Posted on April 10, 2017
Since composing the last post: HoloLens–The Path to 60fps I have carried out some more work on HoloLens apps with an important requirement for as high quality rendering as possible but of course the goal is always to maintain 60 frames per second. This time around we were looking to render glass containers with associated labelling which needed to be clear and legible and the glass should look as realistic as possible in the available time.
I’m going to use Blender for demonstration which, if you haven’t come across it before is a free, open source 3D creation software package
As with a lot of project scenarios we don’t always have a greenfield where models and textures, etc. can be created from scratch; often the assets will already exist but not necessarily in a form we can instantly consume easily on a HoloLens. In this case there were pre-existing assets that were highly dense in terms of faces and vertices and also somewhat wasteful in terms of topology with geometry inside other geometry. These assets were used for high-quality renders of products for marketing materials so performance was not a consideration at all. To give a feel for the details, one glass container consisted of nearly one million polygons including embossed writing on the glass constructed of triangles.
Check out the underside of the bottle – this is all polygons!
We also had a requirement to be able to read and examine the labels applied to the container with high fidelity. Based on these requirements we knew that we needed to:
Decimate the polygon count of the mesh
Experiment with shaders to give a glass look
Ensure that the experience all ran with as high a frame rate as possible given the other requirements
At the outset we used the HoloLens device portal to record a baseline for the performance – this turned out to run at below 20fps for one container – we hoped to be able to create a series of containers in the final experience so clearly this needed improvement.
Shaders for Glass
We replaced all of the existing shaders which came across from the initial FBX files as standard Unity shaders with optimized shaders from the HoloToolkit for Unity. We used the BlinnPhong Configurable Transparent shader so we could have specular highlights and could also experiment with the opacity of the glass.
The shader gives control over the highlights and also the alpha blending modes:
There was a huge amount of details in the original model which we wanted to capture for some added realism. This included embossed writing on the surface of the glass and also very fine serrations on the cap of the containers. Enter normal maps which define how the lighting interacts with the surface based not on the geometry of the mesh but the directions of the normals specified by the normal map. Since we had a very dense model with full detail it is possible to generate the normal map from that model and apply it to a model which retains the overall shape of the original but is specified with a vastly reduced triangle count. Here are the steps we carried out using Blender to achieve this on the cap but can be applied to the model as a whole:
Original bottle – this was not the 1M polygon model but one that had been derived from it but still retaining most of the details (not the embossing) which has 30k + polygons and comes in a few different pieces (the labels and the cap are separate meshes)
Let’s see how we can apply a normal map to a cylinder in Blender to create a low polygon model of the cap shown in the screenshot above. The cap itself has approximately 28k triangles.
The mesh we are starting with here is not particularly ‘clean’ so I first ran a tool in Blender to remove duplicate vertices called Remove Doubles.
This took the number of faces down to about 11k
Next I carried out the following steps:
- Created a cylinder which was roughly the same size as the cap
- With the cylinder selected – create a smart uv unwrap – shortcut key u will bring up the menu item to run this
- Then create a new image in the UV editor and save it
Select the original high-res cap model and then shift select the cylinder
The order of this selection is important as it defines the source and target for rendering the normal map
and then carry out the bake operation with these settings:
This should give the result shown below – note the normal map with shading which reflects the details in the high definition cap.
First, save the image and then:
- Create a new material on the cylinder
- Add the previously saved normal map to the material as a texture
- Set preview render mode to material (so we can see the effect of our changes).
- The texture will be there on the surface of the cap as an RGB texture wrapped onto cylinder
- Set the texture to be a normal map
Change the influence setting for the texture from diffuse rgb to normal
This shows the caps side by side and our new cap is down to approximately 100 triangles and also responds as expected to dynamic lighting changes.
So you can see this is a useful technique to use when detailed models are required but the cost of processing a large number of vertices is prohibitive such as on HoloLens. We can apply this technique to the bottle as a whole and retain the embossing on the surface of the glass.