Download source code: KinectSpaceToWindowCoords.zip

Part of the effort of developing a new application for our new front display in the Avtex lobby is nailing down how users will control a "mouse pointer" on the screen with their hand in mid-air. To keep things simple, we basically want to show and move a hand-shaped cursor on the screen as the user moves their hand – similar to what you see in XBox 360 Kinect games like Adventures and Sports.

The new 60" LCD Avtex display

Temporary placeholder welcome screen.
Ignore the taskbar!

Before I start to dig in – the open source Kinect community and the tools it produces are always changing. Please note the date of this post – the code and examples shown here were made with tools from late 2010.

Fun with NITE

The lack of documentation and examples with managed .NET code for the Kinect makes beginning this task a challenge. Assuming the OpenNI and PrimeSense drivers and binaries have all been installed on your Windows computer, you can find the managed NITE library at this location (or a similar one depending on your specific operating system):

C:Program Files (x86)Prime SenseNITEWrappersC#BinManagedNite.dll

NITE is basically the library that sits on top of the Kinect device drivers and allows you to do things such as hand tracking. If you aren't familiar with the NITE object model then the managed NITE wrapper will look a bit confusing. In addition, the managed NITE wrapper also doesn't contain the full capabilities of the unmanaged NITE libraries – so if you are familiar with the NITE object model then you may be disappointed with the managed library's limited abilities.

The best example of NITE hand tracking I found was posted by Joshua Blake on this OpenNI-dev Google group post: http://groups.google.com/group/openni-dev/browse_thread/thread/707ab3ba1dd37231/a8964c4a652bc58e?lnk=gst&q=niteconsole#a8964c4a652bc58e. In the fifth message in that thread Joshua points to a C# app sample that uses ManagedNite.dll and performs hand tracking. The download link he posted for that sample app is here: http://dl.dropbox.com/u/7911268/managed%20OpenNI.zip 

An Example WPF App

That sample app is a simple console app that writes out your hand's coordinates to the screen. I built on that example and dealt with some Kinect-to-WPF translation challenges in a new WPF example app. 

WPF Example App controlled by Kinect sensor

A screen cast demo of this app is available here: http://www.youtube.com/watch?v=bJjUpfvnMmM

The sample console app uses the XnMOpenNIContext, XnMSessionManager, and XnMPointFilter in the NITE library to take care of all the hand tracking. Using that sample app's approach, I wanted to fit it into a MVVM pattern where I could bind my WPF view to data such as the current NITE hand location. Since the NITE stuff runs in an infinite loop in its own thread, I decided to use a timer on the ViewModel to inspect the Kinect sensor data on a periodic basis (every 10 milliseconds provides more than enough resolution). I wrap all of the Kinect operations and data into a single class called CameraSession.

I set up my ViewModel like so:

void Initialize()
{
	this.Scale = 2d;
	var start = new ThreadStart(CameraSession.Run);
	var cameraThread = new Thread(start);
	cameraThread.Start();

	// assumes only single/first monitor
	var screen = Screen.AllScreens[0];
	this.workingArea = screen.WorkingArea;

	this.cameraTimer = new DispatcherTimer()
	{
		Interval = TimeSpan.FromMilliseconds(10)
	};

	this.cameraTimer.Tick += new EventHandler(cameraTimer_Tick);
	this.cameraTimer.Start();
}

As you can see, the CameraSession (which holds the NITE references) spins up on its own thread and then I kick off the DispatcherTimer. You may notice I'm using something called workingArea. workingArea is a Rectangle representation of my screen size, which I will reference later.

The Run method of the CameraSession initializes all of the NITE activity:

public static void Run()
{
	XnMOpenNIContext context = new XnMOpenNIContext();

	try
	{
		context.Init();
		KinectExists = true;
	}
	catch (XnMException)
	{
		Trace.WriteLine("Kinect device was not found.");
		Active = false;
		KinectExists = false;
		return;
	}

	Trace.WriteLine("Kinect device found.");

	sessionManager = new XnMSessionManager(context, "Wave", "RaiseHand");
	sessionManager.SessionStarted +=
		new EventHandler(sessionManager_SessionStarted);
	sessionManager.SessionEnded +=
		new EventHandler(sessionManager_SessionEnded);

	XnMPointFilter filter = new XnMPointFilter();
	filter.PointCreate += new EventHandler(control_PointCreate);
	filter.PointUpdate += new EventHandler(control_PointUpdate);
	sessionManager.AddListener(filter);

	while (!ShutDown)
	{
		context.Update();
		sessionManager.Update(context);
	}
}

The session manager and point filter objects fire events from which we can determine when a hand tracking session starts and when the user's hand moves.

When the timer ticks every 10 milliseconds, a little bit of UI math magic happens:

void cameraTimer_Tick(object sender, EventArgs e)
{
	if (CameraSession.KinectExists)
	{
		if (CameraSession.Active)
		{
			this.CursorX = this.workingArea.Width / 2 + CameraSession.PositionX * this.Scale;
			this.CursorY = this.workingArea.Height / 2 - CameraSession.PositionY * this.Scale;
			this.OnPropertyChanged("SensorX");
			this.OnPropertyChanged("SensorY");
			SetCursorPos((int)this.cursorX, (int)this.cursorY);
		}
	}
}

The key parts are the workingArea and Scale. workingArea is a Rectangle that represents the window size. In my app's case, I'm assuming that the window is maximized. In your app you will need to change the working area to represent your window size and also account for the Top and Left properties of your window – because we are going to automate the mouse cursor position using SetCursorPos. If you don't account for the top-left position of your window then your cursor position will be placed too far up and to the left.

The code above finds the midpoint of the window and adds the Kinect sensor hand position to it (multiplied by a Scale factor). The Kinect sensor space has its origin in the "middle" of the space in front of the sensor, so you will need to take the Kinect's X and Y deltas from its origin and add them to the window's midpoint to calculate the new cursor position. Technically, we need to subtract the Y delta because Kinect's Y direction is opposite of Windows/WPF.

Scale is a value that makes the cursor "feel right" when the user moves their hand. In the code above, if Scale equals 1 then the cursor seems to move very slowly. A Scale of 2 or 3 feels right, and a Scale of 10 is (probably) too much. You need to find the right balance based on how far away the user is from the Kinect and what the user needs to do in the app.

After the new cursor position is calculated, I automate the cursor movement by calling SetCursorPos, which is an imported call from the Win32 API. This call is made from within the ViewModel for now, which might not be best from a pattern standpoint but it works just fine for my example.

Wrap Up

That is pretty much all there is to automating a mouse cursor from Kinect hand-tracking:

1. Run NITE in its own thread
2. Query NITE data
3. Calculate new cursor position from Kinect deltas and a Scale factor
4. Move the cursor from a Win32 call