How exactly does a Kinect work?

[Glass]

A guy known as GIL posted on his blog GIL-O-TOPIA about the internal workings of Kinect.

If put in simple terms the Kinect basically contains a VGA camera, an IR camera, an IR projector, a motor, 4 mics and a fan and some control chips. The cameras are structured light cameras and provide distance measurements for each pixel in space. These parts make it possible for the Kinect to register every movement of every part of the body of the person standing in front of it. It sends this data over USB.

However, the real magic of Kinect lies in how all that data is interpreted and used by applications. The software uses exceptionally complex algorithms which Microsoft has perfected for the Xbox 360 console and that the openKinect community is trying to do now for the computer.

This is what Johnny Lee Chung (creator of Wiimote project) had to say about the software used in Kinect…“The sophistication and performance of the algorithms rival or exceed anything that I’ve seen in academic research, never mind a consumer product.We would all love to one day have our own personal holodeck. This is a pretty measurable step in that direction.”

The whole elaborated version is available on GIL-O-TOPIA:
http://gilotopia.blogspot.com/2010/11/how-does-kinect-really-work.html

[Mammie27]

i don't know exactly how dies a kinect work. But what i heard somewhere to which i am sharing with you. Kinect use depth sensor produced by PrimeSense, but how exactly it works is not obvious from the first glance. Some person here, claimed to be specialist, assured me that PrimeSense sensor is using time-of-flight depth camera. Well, he was wrong. In fact PrimeSense explicitly saying they are not using time-of-flight, but something they call “light coding” and use standard off-the shelf CMOS sensor which is not capable extract time of return from modulated light.
Daniel Reetz made excellent works of making IR photos of Kinect laser emitter and analyzing it’s characteristics. He confirm PrimeSense statement – IR laser is not modulated. All that laser do is project static pseudorandom pattern of specs on the environment. PrimeSense use only one IR sensor. How it possible to extract depth information from the single IR image of the spec pattern? Stereo triangulation require two images to get depth of each point(spec). Here is the trick: actually there not one, but two images. One image is what we see on the photo – image of the specs captured by IR sensor. The second image is invisible – it’s a hardwired pattern of specs which laser project. That second image should be hardcoded into chip logic. Those images are not equivalent – there is some distance between laser and sensor, so images correspond to different camera positions, and that allow to use stereo triangulation to calculate each spec depth.
      The difference here is that the second image is “virtual” – position of the second point y_2 is already hardcoded into memory. Because laser and sensor are aligned that make task even more easy: all one have to do is to measure horizontal offset of the spec on the first image relative to hardcoded position(after correcting lens distortion of cause).
That also explain pseudorandom pattern of the specs. Pseudorandom patten make matching of specs in two images more easy, as each spec have locally different neighborhood. Can it be called “structured light” sensor? With some stretch of definition. Structured light usually project grid of regular lines instead of pseudorandom points. At least PrimeSense object to calling their method “structured light”.