Here’s a short video of the sample app that goes with this post:
The code is all found on Github here
One of the coolest aspects of HoloLens2 is the iris login as it removes the need for typing on a virtual keyboard. So a cool feature which avoids the need for the coolest feature! Although hand-tracked typing is arguably the coolest feature it can become tiring when needed over and over…
So set up your HoloLens2 device using AAD credentials and then configure the device to use Iris login as shown below:
An app to configure will run and guide you to watch a moving animation.
You can now add up to 64 other users logging in via AAD and have the device be shared by signing in an out. We’ll see how this can be extended into an app later in the article.
I have previously written a little about the topic of retrieving Azure Active Directory (AAD) OAuth2 identity and access tokens on a HoloLens device.
If this topic is new to you I would suggest reading through the identity platform overview here
I focused on specific topics such as how to retrieve a token to access the Microsoft Graph API using Microsoft Authentication Library for .NET (MSAL) using the OAuth delegated flow and device code flows. You can see code and read about those here and here. There are some complications and potential blockers for those trying to set this up for the first time and, in my experience there is quite a lot of misunderstanding around the topics of auth in general. In my role at Microsoft I have worked through some of these issues for customers and I get asked a lot about how to set up auth correctly to access Graph APIs, your own APIs and Mixed Reality services. Often samples for services will skirt the issue by using secrets embedded into a client side application which you can configure with your own set of secrets for your service instances. This is fine for a demo and a first-try of the services in question but as soon as you turn your thoughts to developing production code falls down immediately and the first task you will be faced with is how to secure access to your services. I am going to cover the OAuth 2.0 Authorization Code Grant which you can read about here OAuth Grant Types if you want to understand the details of the flow.
In brief, the scenario I am talking about is when an end user provides permissions for an app to access services that the user has access to on their behalf. The end user does this by authenticating and consenting to a set of permissions known as scopes. The consented scopes are encapsulated within the access token itself.
I won’t cover, for instance Client Credentials flow which is used in an app to service scenario not involving a user.
So, it seems pretty simple so far so why the need for this post? These are the complications that I associate with this:
-
Which APIs or framework should I use?
-
The acquisition of an OAuth token requires a browser doesn’t it? So, how can I do this in Unity without buying a browser plugin?
-
How do I configure my Azure Active Directory to access services and use role-based access control?
-
I’m already logged into the HoloLens can I not just access the token already on the device?
I won’t focus on some of the other topics like IL2CPP stripping code from .NET libraries as I have covered that in a previous post IL2CPP + HoloLens, so you may want to be on the lookout for those kind of issues. I will leave you with a code repository that you can borrow code from to set up the particular scenario that you are interested in. The code and the rest of this post will be concerned with the different ways that access and id tokens can be retrieved, back-end configuration and an illustration of how you might get an AAD token for some of the new Mixed Reality services such as Azure Spatial Anchors and Azure Remote Rendering.
Sample Walkthrough
This just describes some aspects of how I created the demo application. If this isn’t of interest just skip these sections as I will make my way towards the various frameworks and APIs used.
For the sample I have used Unity 2019.4.0f1. First add Nuget 2.0.0 to your project. Then, add MRTK v2.4.0 to your project using nuget. It looks like nuget support has been dropped for MRTK v2.4.0 so download the packages and import into your project.
Set app package name and capabilities to allow debugging on device via the unity debugging tools so we can connect using the managed debugger
I have trained myself to set the package name to avoid clashes with other apps I am developing
This also involves removing any legacy XR packages using the Unity package manager, if necessary..
add MRTK to the scene
Turn off spatial awareness by cloning the MRTK configuration and turning Enable Spatial Awareness off. Add a near menu prefab to your scene, empty it’s contents (you will need to unpack the prefab) and resize the remaining items so that the backplate is bigger, the visual cues are re-positioned and the pin is re-positioned.
Find the RadialSet prefab in the MRTK and drag an instance under the near menu game object, add a GridObjectCollection component to the RadialSet and add a toggle button for each option we want to represent (one for each login method).
Search the project panel for toggle to find the PressableButtonHoloLens2Toggle and drag one under the RadialSet.
Layout the toggle buttons
Adjust the backplate and grip visuals to arrive at:
Rename and configure toggle buttons in the InteractableToggleCollection component on the RadialSet
Then we are going to make this into a tabbed dialog by adding a parent node to the RadialSet (I called this TabbedDialog) and then adding a script to that node. Here’s a snippet from TabbedDialog.cs showing how the content visibility is synchronised with the tab selection.
private void OnSelectedChanged()
{
// Retrieve the current Index here...
var newId = SyncPanelToCurrentIndex();
if (!string.IsNullOrEmpty(newId))
SelectedChanged.Invoke(newId);
}
private void OnEnable()
{
if (_selectionChanged == null)
_selectionChanged = OnSelectedChanged;
tabs.OnSelectionEvents.AddListener(_selectionChanged);
SyncPanelToCurrentIndex();
}
string SyncPanelToCurrentIndex()
{
int idx = tabs.CurrentIndex;
if (idx < 0 && idx >= Panels.Length)
return string.Empty;
foreach (var panel in Panels)
{
panel.SetActive(false);
}
Panels[idx].SetActive(true);
return Panels[idx].GetComponent<ProviderId>().Id;
}The content pages are arbitrary game objects and are given an ID by adding the ProviderId script. This way the ID can be passed around to switch the content page and also the Identity provider.
The video below shows the tabbed dialog being used to switch the content pages in the Unity Game preview.
We can use this UI to plug in different auth providers and switch them at runtime.
APIs and Frameworks
Web Authentication Manager
Here we can access the APIs which enable you to retrieve the current user’s AAD token and also to invoke a biometric check against the current physical user. this would guard against another user picking up a device and accessing sensitive information. You can switch the iris check on/off in the sample with a toggle switch.
When this is selected and you sign in your iris will be re-scanned and you will see this dialog.
With the switch toggled to the ‘off’ position you go through a flow that acquires the currently logged in user’s AAD token so you won’t get a prompt to login. This way you can share a HL2 device amongst multiple people and ensure that if one user forgets to sign out they will not leak sensitive data. More information about setting up a device for multiple users can be found here.
Windows Account Provider
I have experimented here as I found a scenario where I wanted to switch between different known accounts but I couldn’t get the account settings pane to show so haven’t completed this part but may return to it later.
Microsoft Authentication Library
Aside from the built-in APIs this would be the next favourite method of gaining an AAD token and that would be the suggestion of the Microsoft Identity team. Plus it supports device code flow which is sometimes handy depending on your scenario. Also it allows you to continue to develop your app against real APIs from the Unity editor.
Active Directory Authentication Library
When we first try to use ADAL we get hit with the following error:
Error on deserializing read-only members in the class: No set method for property ‘TenantDiscoveryEndpoint’ in type ‘Microsoft.IdentityModel.Clients.ActiveDirectory.InstanceDiscoveryResponse’.
This looks suspiciously like an IL2CPP code stripping issue. This occurs as byproduct of the optimisation used when the .NET binary is converted to C++. Unused code is stripped out in that process and sometimes code that isn’t detected as reachable gets removed. Look out for code that is only referenced in a reflection scenario. See here for further details. The short answer is to add a link.xml stating which types to not strip.
You can see the link.xml for the app here
Although ADAL is not the recommended way forward for legacy and cross-platform reasons you may need to use it and so I have included it in my sample.
If you are using ADAL but would like to migrate to MSAL then the differences are distilled in this guide.
Web Authentication Broker
WebAuthenticationBroker is a broker component that manages the http redirects retrieving the resulting AAD token that would usually be managed by a browser for a web application. As a result there is a bit more work to do to construct the original url than the other libraries.
Settings
The settings for auth are stored in a json file in the Assets folder
So edit this file with your own settings and rename it to app-settings.json and the sample code will pick it up.
Configure the Backend
It is worth scanning the docs here and although we can see from those docs that you can login to your HL2 with a local account or an MSA things get more interesting when we consider Azure Active Directory because at the time of writing it is possible to have 64 AAD user accounts with the following options:
- Azure web credential provider
- Azure Authenticator App
- Biometric (Iris) – HoloLens 2 only
- PIN – Optional for HoloLens (1st gen)required for HoloLens 2
- Password
Of most interest to me being iris login with multiple accounts.
Just getting an OAuth token and displaying it isn’t too interesting as really the aim in a real app would most-likely be to access a custom API or some other Azure resource. So, to show that things are working as expected we’ll take the AAD token and exchange it for and access token for Azure Mixed Reality services. We’ll use that access token to make a call to the Azure Remote Rendering Service to enumerate rendering sessions just to check that the call succeeds. This access token could be used to access any MR services, e.g. Azure Spatial Anchors, that the logged in user has access to. We’ll look into how to configure the application and also provide access to a particular user next.
Test
I added a button to run the test which calls the ARR service and shows the results of whether the call succeeded or not.
After getting an AAD token you can press the button which will exchange the AAD token for an MR access token using the Mixed Reality Secure Token Service (STS). This access token will subsequently be in a call to the ARR service. The test UI will show green if the http status is successful and red otherwise.
Register the App
I’ve covered this a few times on my blog so will just include the basic steps here:
In the Azure portal if you don’t have an AAD set up there is a quickstart here to help get you up and running. Once created you can register a new application:
Select ‘New Registration’ and just name it without worrying about the other settings. You will be shown a page which has settings that you will need for the app-settings.json file we mentioned above (maybe fill those now). The other important thing is to register a redirect uri for your app.
Redirect URI
The easiest way to discover the redirect URI for the sample app is to run it and the app will write the uri to th unity log.
Note, that the running instance of your app needs to be terminated before the log is flushed. You can terminate the app and download the unity log using the HoloLens device portal.
This code:
string URI = string.Format("ms-appx-web://Microsoft.AAD.BrokerPlugIn/{0}", WebAuthenticationBroker.GetCurrentApplicationCallbackUri().Host.ToUpper());
Logger.Log("Redirect URI: " + URI);will log out a redirect URI for a UWP app.
If I have this right the ADAL implementation required a slightly different format so I registered both:
string URI = string.Format("ms-app://{0}", WebAuthenticationBroker.GetCurrentApplicationCallbackUri().Host.ToUpper());
Logger.Log("Redirect URI: " + URI);To configure access for a user navigate to the Mixed Reality service you want to grant access to
More info here along with instructions on how to set up an Azure Remote Rendering account. I’ve set up my test user as an owner.
Along with this I also need to give my app the necessary API permissions to access the Mixed Reality services. I can navigate to my app registration page on the Azure portal.
and I can request a permission by searching for Microsoft Mixed…
Selecting that and choosing ‘Delegated Permission’…
and choose mixedreality.signin permission.
This will also be the scope that you will request for your AAD token.
And finally I granted access to my tenant:
Diagnosing Token Issues
If at some point you get stuck on some AAD error then having an understanding of the OAuth flow that you are using is vital and you can also use https://jwt.ms/ to paste in a token and have a look at it’s contents/claims to ensure they match with what you are expecting.
Hi Pete,
First of all thanks for this awesome tutorial, this is a topic rarely covered properly and you have done it very clearly.
Could you please do a tutorial on using MSAL with Unity iOS or Android app?
I have used both Amazon SDK and Firebase which integrate with AR/VR apps very easily.
I don’t understand why Microsoft doesn’t provide the same level of flexibility in the MR space.
I agree, and I think a sample/tutorial showing a VR/AR app accessing Azure services across platforms; iOS/Android/Windows MR/HoloLens/Windows Desktop would be really useful. If I get chance (I don’t think I will unfortunately) but would like to extend this sample (or a better scenario) to include those platforms and maybe discuss design decisions across those platforms. I wonder if the Azure Spatial Anchors samples might help as they are x-plat https://github.com/Azure/azure-spatial-anchors-samples.
HI Pete, thank you for the detailed explanations on your tutorial.
It helped me get through all the steps of deploying your sample app to HL2.
I got an issue though when attempting AD authentication using MSAL.
The app fails to show the interactive login dialog, and shows these errors instead (displayed on the main panel + UnityPlayer.log):
[1]
An error occurred while sending the request
UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
UnityEngine.Logger:Log(LogType, Object)
UnityEngine.Debug:Log(Object)
AuthManager:Log(String, Boolean)
System.Threading.ThreadStart:Invoke()
d__4:MoveNext()
UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)
UnityEngine.MonoBehaviour:StartCoroutine_Auto(IEnumerator)
System.Threading.ThreadStart:Invoke()
UnityMainThreadDispatcher:Update()
(Filename: C:\buildslave\unity\build\Runtime/Export/Debug/Debug.bindings.h Line: 35)
[2]
Object reference not set to an instance of an object.
UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
UnityEngine.Logger:Log(LogType, Object)
UnityEngine.Debug:Log(Object)
AuthManager:Log(String, Boolean)
System.Threading.ThreadStart:Invoke()
d__4:MoveNext()
UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)
UnityEngine.MonoBehaviour:StartCoroutine_Auto(IEnumerator)
System.Threading.ThreadStart:Invoke()
UnityMainThreadDispatcher:Update()
I contacted MS Support, they asked to upgrade MSAL from 4.9.0 to 4.29.0, which I did and I also tried other things like changing platform settings (under inspector / plugin). Did you run into these same issues?
Hi Houssam,
I don’t recall seeing similar issues, are you using the same MSAL version that I used in my sample? I just tried an old build of that sample with the latest insider OS and I see the dialog as expected.
Could you find the full unity log from your HoloLens device and send to my email address in the about page of this blog?
You can find the log using the HoloLens developer portal File Explorer with the path:
User Folders\LocalAppData\\TempState\UnityPlayer.log
(Terminate the app if it is running otherwise the log file won’t get flushed to disk).
Pete
I am using the same MSAL version that’s in your sample, which is 4.9.0.
It is worth noting that MS Support asked me to upgrade it to 4.27+ because of a known issue in MSAL prior to 4.27. So I updated the MSAL under Plugins/MSAL/Placeholder to 4.29.0 and updated the platform settings to match your sample’s (Editor / Any CPU / Any OS). And I removed the one under Plugins/MSAL. Redeployed to HL2 and got the same exception.
I will send the log shortly.
Regards
Houssam
Hey! I’m till trying to get this working on my own project. Everything works in the editor, but when built to the device, Device Code Flow fails at the callback point with no error or exception thrown. (We’ve talked in the slack about it.)
I went through several cycles of excluding everything I could from the code stripper as discussed above with no success in UWP (although that made the difference in a standalone build).
Now I’m wondering about the two versions of the microsoft.identity.client you have in this project. What’s the difference and could that be causing my issue?
Thanks
Hi Pete,
Great article. Thank you.
When I try the example code in the github and test the WAM using Iris toggle on. When click Sign in => I don’t see the Credential Dialogue window show up? It stuck there and show these logs in the windows:
Logging in with WebAuthenticationCoreManager…
I did check Hololens Sign-in options in the settings and see it is as the picture shown in the article?
Am I missing anything?
Thanks for this article, I was referred to it by someone on the Hololens Developer group on facebook. I am trying to get authentication working such that users do not need to login again to get access to an internal web API so I am hoping that I find the solution here. I have hit a problem though with the WAM login provider. The app appears to hang on the following line in the WAMLoginProvider.
var consentResult = await Windows.Security.Credentials.UI.UserConsentVerifier.RequestVerificationAsync(“Please verify your credentials”);
In the device portal I can see that Credential Dialog process/app appears but I do not see anything in the device display. Has anyone else come across this problem?
I am using Iris authentication on the device.
Hi Phil,
Yes, this has been noted in an earlier comment. I haven’t had chance to fully investigate this but here is one observation I have made that might help. I had a build of the sample that I used previously for testing deployed to a HoloLens 2. I opened that app and tried to auth using the WindowsAuthenticationManager and I saw the same behaviour that you mentioned. The call to UserConsentVerifier.CheckAvailabilityAsync() didn’t result in the dialog being shown – it just silently failed somehow. When I checked the HoloLens privacy settings I noticed that the Eye Tracker capability was not switched on. I switched it on and I could invoke the dialog again.
Hi Pete,
Thanks for responding. I have checked my privacy settings and Eye Tracker capability is defintely switched on.
Phil
Hi Phil,
See my other answer here referring to
UnityEngine.WSA.Application.InvokeOnUIThread(async () =>
{), false);
Same issue here. Eye tracking is On in privacy settings, but no popup.
Hi Tom,
I have replicated this issue and I believe the fix is to schedule the code that triggers the dialog to run on the UI thread. Something like:
UnityEngine.WSA.Application.InvokeOnUIThread(async () =>
{
// code here…
}, false);
I’m not sure what changed as this worked in an earlier unity build but I will update the repo with this fix within the next week.
Thanks Pete, think that indeed solves it. By now we actually managed to get the access token in Unity. So that’s great. Thanks.
Hey! Good Post, but my HoloLens 2 is already AAD joined, so can I just take the O Auth 2.0 token from the device to SSO into my app?
I can’t find anything about that in documentations.
greetings!
Yes, that is what the iris login part of the sample does.
Thanks for your reply! But how can I rewrite it so that I can use for instance Microsoft Dataverse API? 🙂