It's the FUTURE!

Things have been busy over here. This past week had me spend an entire day working with the lower Cohort and their first client meetings. Nothing that was super taxing but it did eat up a day of work. However I still managed to make some headway in doing......something.... with 3D.

First off is some amazing help that I have been receiving from Kimberly Voll, one of the great professors here at the CDM. She has in her possession a 3D printer that she has generously been allowing me to run stuff off on. And here is the first thing!

So what is that piece of black plastic you ask? It is a camera mount. Frist some background, in order fo the RGBD toolkit to properly and reliably take 3D footage, it needs to have a locked spatial relationship between a DSLR and a Kinect. To achieve this one needs a special camera mount. On the RGBD website they recommend constructing your own camera mount using aluminium, drill presses and other traditional fabrication materials. This is not an option that I wanted to pursue because it would be hard to find those materials and IT'S THE FUTURE. So in place of losing a finger in the process of fabricating a metal camera mount, Kim has allowed me to use her 3D printer to print out a plan for a RGBD camera mount that I found on the 3D warehouse. 

It's not perfect, in fact this first model doesn't work at all. The problem is that the arms with the hooks have too small an angle and overall the print is a little small. No big surprise there since this was my first ever 3D printing attempt. So I took the model into Maya, installed some special plugins to allow me to export in an .stl format and got to modifying the model. I extended the arms, increased the angle and made those hooks on the arms less intense. I've sent the plan back to Kim for printing and I hope to have another version by next week.

While this is great progress, it is still not the last thing I need to do in order just to get the system working as intended. I also needed to order a quick release plate mounting system from Amazon.ca which will plug into the mount and allow the DSLR to sit securely. This particular piece should be in my greedy hands in two weeks.  =[ 

But that's not good enough!! While the above is progress in terms of gathering the materials needed for the toolkit, it was not progress on the software end. So I spent a lot of time this week trying to get the system working in some manner.

First, more background. RGBD calls for a DSLR with live mode, which allows for the camera to output high quality .mov files. These files are used for both the calibration and the filming portions of the toolkit. Such a DSLR is not available at the CDM, since they are extra expensive. In place of this camera I managed to get my hands on a digital video camera that outputs files in .mkv files, or something else that RGBD doesn't know what to do with. Not to be deterred, I figured out how to get those .mkv files into Adobe Media Encoder and exported as a .mov, thus allowing RGBD to pick up and continue it's process of calibration and recording.

This is not a perfect fix and in fact it is less that ideal. Firstly, the viewing area on the video camera is smaller than that of a proper DSLR with live mode, which means that overall the image will look closer to the subject that how the RGBD toolkit is expecting it to. To combat this the kinect and video camera needed to be much further apart than intended in order to capture enough footage to be useful for calibration. What this means is that I had a video camera on a stack of books at one of the my table, and the Kinect perched on the edge on the other end of the same table. The distance between the two cameras was just over a foot, way more than the system was designed for.

But there is more difficulty. In order to get the footage converted and into the system I needed to remove the memory card from the video camera without disturbing the placement of either camera. This required surgical (ok maybe not) steady hands so as not to disturb the relationship between the cameras.  Somehow, with all this converting and boot strapping I managed to get the thing calibrated and capturing video.

BREAKTHROUGH!! This was a very exciting momement when the calibration worked and I was able to start capturing images. Below are some images of the calibration screens and editing process.

This one above shows the initial recording screens without any color maps applied. This is the raw data from the Kinect being interpreted by the RGBD toolkit. 

This one above shows the two footage streams coming together. You can see how the color image is off center from the depth map images, making for a strange scene. You can see the two streams separately in the upper right corner. 

Finally this above image is when the two streams have been somewhat lined up. Due to the imperfect setup of my hardware I was unable to achieve a closer match. However, just getting everything to this point is a big success in my book. 

Lastly this is the video that I was able to create using my makeshift setup. Again it's not perfect but it is a working proof of concept! 

So where to go from here? Well now that I know I can get the whole system working even with non specific parts my next steps are to refine this approach with the correct hardware as well as further experimentation with the editing software. Originally I wanted to be able to pull out the 3D mesh and take it into Maya, or go the other direction and begin placing my own 3D meshes into the RGBD timeline. Such an application does not seem to be ready built into the system, but maybe I'll be able to find the data files somewhere and start playing with them.

Exciting times! Much progress was made but there is still a lot to do and many questions to answer in terms of application and direction.