Kinect

Kinect 4 Windows V2 – Unity 3D

unityparticles

After installing the Kinect v2 SDK from here http://www.microsoft.com/en-us/download/details.aspx?id=44561 you can also download the supporting Unity 3D plugins here http://go.microsoft.com/fwlink/?LinkID=513177. Note that the plugins require Unity 3D Pro and they expose APIs for Kinect for Windows core functionality, visual gesture builder and face to Unity apps. The zip file containing the Unity packages also contains two sample scenes Green Screen and Kinect View. Lets take a look at KinectView first:

 

So, I stumbled around a bit on the next step as initially I tried opening the KinectView scene from it’s existing location and this doesn’t seem to work very well. The result I got was that when I examined the game objects there was an error on each of the scripts. After some head-scratching I watched the video here http://channel9.msdn.com/Series/Programming-Kinect-for-Windows-v2/04 and at about 10:54 the presenter shows copying the scene locally to the current project. That fixed the issue for me so I was back up and running. You can plug in your Kinect sensor and run the game and then explore the scripts to see how the data is retrieved from the sensor and applied to GameObjects in your scene.

 

kinectview 

My idea was to create a very simple Kinect sample which will move particle systems around following the positions of the users hands. So lets step through what I did.

First, I chose File > New Project to bring up the Unity project wizard:

projwiz

I added the Visual Studio Tools as I’m more comfortable editing and debugging in Visual Studio. You can download the tools from here https://visualstudiogallery.msdn.microsoft.com/20b80b8c-659b-45ef-96c1-437828fe7cf2.

Once the project is created I imported the Kinect Unity package; Assets > Import Package > Custom Package… I navigated to where I installed the Kinect package and chose Kinect.2.0.1410.19000.unitypackage.

import

I left everything selected and imported the various libraries and scripts. The editor shows that I have an empty scene just containing a Camera.

emptyscene

The first thing to do was to try to get the Kinect sensor data into the scene, so to do this I created a new c# script in the project and copied the code from the KinectView sample in the file BodySourceManager.cs. The code retrieves the sensor and opens a reader on it for the body data and also reads the current frame’s data in the scripts update() method. The update() method is called as part of the game loop so we are ‘pulling’ the Kinect skeleton data each time we are called. The class also has a method GetData() to allow the data to be accessed by other scripts.

lefthand

I created an empty GameObject called BodySourceManager and associated the BodySourceManager.cs script with it so the code to set up the Kinect would be executed. Next, I created another empty GameObject and associated a new script which would get the body data and position the GameObject depending on the position of one of the joints in the skeleton. I made the joint type a variable so I could set up the positions of various GameObjects via the Unity editor. The intention was that if I then add a particle system as a child of this new GameObject then it would follow the tracked position of say, the left hand and I could add others for each joint that I wanted to track.

hierarchyThe code for the update loop showing how the game object position is set from the joint. 

  1. void Update ()
  2. {
  3.     if (_bodySourceManager == null)
  4.     {
  5.         return;
  6.     }
  7.  
  8.     _bodyManager = _bodySourceManager.GetComponent<BodyManager>();
  9.     if (_bodyManager == null)
  10.     {
  11.         return;
  12.     }
  13.  
  14.     Body[] data = _bodyManager.GetData();
  15.     if (data == null)
  16.     {
  17.         return;
  18.     }
  19.  
  20.     // get the first tracked body…
  21.     foreach (var body in data)
  22.     {
  23.         if (body == null)
  24.         {
  25.             continue;
  26.         }
  27.  
  28.         if (body.IsTracked)
  29.         {
  30.             var pos = body.Joints[_jointType].Position;
  31.            this.gameObject.transform.position = new Vector3(pos.X, pos.Y, pos.Z);
  32.             break;
  33.         }
  34.     }
  35. }

The next step was to add two new GameObjects, one for each hand and set their joint types to HandLeft and HandRight and then add particle system’s to each as a child. Then the particle systems had their parameters tweaked until they looked how I wanted. It seems to me that the combination of Unity and Kinect is a pretty powerful one and I’m looking forward to seeing the results in the Windows Store. Please find the sample project here http://1drv.ms/1tyKlIT.

7 thoughts on “Kinect 4 Windows V2 – Unity 3D

  1. Hi Pete,

    I am trying to accomplish a similar functionality for a project I am trying to develop. Could you explain to me how did you do the jointPosition script? I do net get this step: The next step was to add two new GameObjects, one for each hand and set their joint types to HandLeft and HandRight I suppose is based on the functionality of the JointPosition script.

    Thank you for your help!

    1. I created game objects from the Unity menu and added particle systems as children. You can associate a script with a game object by dragging and dropping it onto it’s node in the hierarchy view. Once the script is associated you should see it’s variables in the inspector window.
      Hope that helps.

  2. Hey Pete,

    I’ve tried your example, and was trying to include JointOrientation as well by assigning the Kinect’s JointOrientation to my GameObject’s Transform Orientation. But I seem to be getting the wrong Orientation values.

    Much like this:
    UnityEngine.Quaternion jointRotation = UnityEngine.Quaternion.identity;
    var pos = body.Joints[_jointType].Position;
    var rot = body.JointOrientations[_jointType].Orientation;

    jointRotation.Set(rot.X, rot.Y, rot.Z, rot.W);

    this.gameObject.transform.position = new Vector3(pos.X, pos.Y, pos.Z);
    this.gameObject.transform.rotation = Quaternion.(jointRotation);

    1. Hey Sean,

      Can you post the implementation (guide) for the Orientation on the plugin’s side?
      Or any progress for the rotation…?

      Thank you

Leave a Reply

Your email address will not be published. Required fields are marked *