Failures and Success

Greetings dear readers. This last week has been quite busy for me on a number of different fronts, but the only one I assume you are interested in is my RGBD and 123D projects. So! here we go:

I am still waiting on a 3D print for my camera mount which would allow me to finalize the calibration of the RGBD setup and turn the whole system into a more mobile iteration. Since I don't yet have said camera mount in my greedy little hands I can only make temporary camera setups and shoot stuff in a stationary way. So that's what I did in order to explore what kind of interesting images I could create using my own fine art skills and this new RGBD toolkit.

In the above you can see how I am working things out. The Kinect is in front, the video camera behind and sitting on some books to raise it up enough, the calibration grid and my trusty macbook ready to process all the information. This setup is not ideal, as I have said before, but now I understand even more so why it's not so good. A major problem is that the video camera has a different aspect ratio than that which is recorded by the Kinect, making alignment extra difficult during calibration. Not to be deterred I pressed onward and got the best calibration images that I could. Below is a screen shot of the process.

Here you can see my using the calibration gird and recording images in RGBDcapture, the open source program I am using for this. Problems start to become apparent when comparing these images with the ones captured from the video camera because one can notice that the frame of recording is quite different between the two devices. Still, not to be dissuaded I pressed forward and created as many good quality calibration images as possible.

From there it was a matter of bringing the two different sets of images together and matching them up so RGBD can figure out the relationship between the two cameras. In order for that to happen RGBD needs to run a python script on collect of .mov files that were generated from the non Kinect camera. That's all well and good but my files needed to be converted to .mov from .mkv. Fine again, but in order to get at tose files I needed to gingerly remove the memory card from the video camera without nudging or otherwise changing the position of said camera. Moving it too much would result in my calibration efforts being wasted as I would have just changed the relationship between the two cameras. Anyway, I got the card out and started converting all my files. Below is a screen shot of the process.

Great! So that went well and then it was on to matching up those two sets of calibration images. Pretty straight forward as you can see by the screen shot below. If you look closely you can see that there are two rows of images, each relating to each other, which help RGBD to determine the relationship between the two cameras. You may also be able to see that there is quite a bit of difference between the two images since I was using a non recommended video camera.

None the less I still got what I thought to be enough calibration images to inform the system of how the cameras were related to each other in real space. From here it was onto recording.

Recording was probably the easiest part of the whole process since most of the work happened outside the computer as I was trying to make interesting things happen in front of the camera. So I completed a number of actions in front of the camera setup to allow for me to experiment with the footage. Here is a screen shot of what that looked like on the screen during recording.

Looks pretty interesting eh? Well kind of I suppose. This is just half of the footage however, this being the depth. In addition to this one needs to have the color information of the footage in order to overlay the two together and create compelling 3D images. So from here it was a simple matter of converting the files from the video camera into a format that RGBD could understand. Here is a screen shot of that process.

So great! At this point I had both the color and the depth recorded and was ready to start the editing process. For the next two hours this is what I kept getting to appear on my screen.

What the crap is that?! At this point I had spent 6 hours setting up, calibrating, shooting and converting everything. It was extremely disheartening to see my work go nowhere due to a software crash that I had no idea how to fix.

With a heavy heart I broke down my camera setup and archived my files for later contemplation. Maybe I could use the files generated in some other way? I had no idea what to do next except go get some comfort food and watch Dexter (Yes I was working on Sunday).

Then something amazing happened just as I was trying to fall asleep that night. I was replaying my process over and over in my head (I know I should think of other things before bed) and I was trying to pinpoint where the problem might have been. I knew that the new files crashed upon opening, but my old files still opened just fine and allowed for me to make new compositions. So the problem could not be in the application being corrupt itself, but rather somewhere in the files that I was supplying. I had looked over the directory structure of the two separate recordings and they seemed the same in terms of the files that I had captured. It was about this time that I thought, "what if the calibration is so wrong that it is crashing?"

That was it!! The next day I took the old calibration that I was using for the successful recording that I did weeks ago and put that into the new recordings directory structure. This mean that the calibration would be way off since the relationship between the two cameras was different from the two different shooting days, but I didn't care I just wanted to see if I could get the darn thing to open. And it did open!!!

So here is another example of why I need to get that 3D printed mount and a correct DSLR camera with Live Video mode. The video camera I was using had a different aspect ratio from the Kinect and the frame was smaller which made it impossible to get an ideal set of calibration images. Apparently it was so bad that RGBD just refused to open. Once I had placed working (if not the correct) calibration files into the directory structure, RGBD was able to keep going! So it was not ideal but it opened up my work again and I was able to proceed with editing. THANK GOODNESS!

So I edited some of those videos that I shot and experimented with what RGBD can do. I edited 3 videos, 1 person (myself), one performance with practice juggling flags, and one brief experiment with colored foam core cutouts. The results informed me a great deal of what works, what is interesting and how I could make it better. Needless to say, with a better registration setup and the correct camera I would have been able to achieve a better alignment of depth and color. Anyway, ENOUGH TALK! Here are those videos for your viewing pleasure:

So whats next? Well that sliding head camera mount showed up in my mailbox the other day so I am a step closer to assembling a completed RGBD camera mount. I'm still no closer to the $800 it would cost to get a proper DSLR, but I am friends with a guy who has one.... So next up is to check out how the 3D print is going and see how I can assist the process. If the mount needs more time then I have more experiments with 123D which I can do in the meantime.

Thanks for reading! More to come next week!

Nuts and bolts

And we're back!

Sorry dear readers, I missed a week back there. Truth is that I am waiting on parts for my camera mount and the pitch project that I am working on here at the CDM needed an increase in work hours to make our milestone. Things are more balanced now and I have some updates.

On the RGBD front I have been slowly solving hardware problems. Firstly is that I've ordered a number of parts that will go into the 3D printed camera mount that I am working on (more on that later) and I expect them to arrive in the next few days. With any luck these parts will be the easiest part of the assembly and snap into my properly scaled 3D printed camera mount.

About that camera mount. I think that I may have solved the scale problem. Somewhere between sketchup, Maya and an exported .stl file the scale was lost. This led to imperfect sizes for the prototype of the 3D printed camera mount which mean that my Kinect device just wouldn't fit. To combat this I made a calibration cube. This cube was based on the grid size that appears in the ReplicatorG 3D printing program that Kimberly Voll is using to print this stuff. A little research online found that the grid was likely in centimeters, so then it was a just a matter of making a centimeter cube in Maya and watching the scale as the file was put through ReplicatorG. What I ended up with was a cube that was dang close to 1cm, thereby confirming that now I can measure and export my 3D models with predictable results.

Once the scaling was figured out I needed to modify the camera mount itself. I felt that there were a few details that made the placement of the Kinect extra difficult, so I changed them in Maya. Primarily I was concerned with the severity of a arm hook which holds the Kinect in place, as well as the angle in which the arms were arranged. I lessened that hook with some soft selection tools and modified the angle of the holding arms so that they were more straight and less pinchy. Of course I was paying close attention to measurements of the Kinect itself, the scale in Maya and the grid placement of the model. My new model has been sent to Kim and I expect to have a new version of the camera mount in a couple of days. Exciting!

"Well that's great," you may be saying. "But what are you going to do with it once it is all together?" Good question dear reader! I have used the past week to brainstorm what I could do with this alternative film recording setup and I have come up with a few ideas. What I would like to do is make a series of short experimental captures to see how I can capture interesting events and present them in new ways. Here is a list of some of the performances I had in mind:

• Interviews about the CDM that blend the 2D and 3D components.
• Flag dancing.
• Meteor performance (a type of juggling).
• Perspective play with figure cutouts made from foam core and glue. 
• My dog

It is my intention that these experiments will inform me on some techniques that can produce interesting and never before seen 3D visuals. My final product would be something of a fine art piece, but how it is displayed or what it actually is has yet to be determined. First I have to get the setup mobile and make a bunch of experiments.

That's all well and good, but I have a plan B too. The above plan still needs a DSLR camera which has the capability of Live Video and that is something that I will have to either purchase myself or barrow from a friend. If this, or any of the other integral pieces fall through, I have my backup plan, 123D Catch. Autodesk's experimental 3D capture application is an easy way to quickly create 3D meshes based on photos. I have been working with this application in my downtime so that I can understand how it works and see if I can get any of those meshes into the game that I am developing for my pitch project.

These attempts have been met with limited success. It would seem that there are a few things that can really aid in getting 123D catch to generate a successful mesh and I'm still figuring them out. First off, the web app sucks and one should only use the downloaded application of 123D Catch. This is because the desktop application allows for custom stitching of each photo that was taken, allowing for much greater control of which points are really descriptive of the 3D shape of the object being processed. Secondly, there is a sweet spot of in terms of number of photos that the system likes as well. Less than 20 seems to be too little but more than 35 seems to confuse the system, not to mention trying to get elevated photos of the top of the object which really confuses the program. Thirdly, the size of the photos is also important. Since they are being transferred to Autodesk and being put through some automated magic, I assume that overly large photos (<3mb) simply don't get processed. This is evidenced from my early attempts and they're ever processing status.

So I thought that I had the number, resolution and placement of the source photos figured out. I had limited success with a toy Lama, seen below, and I was ready to try real people again. However, something is still not right with my capture process as my models are still in processing limbo. I did have less than 20 images in that submission, so perhaps that was the problem. Regardless, more experiments with capturing and stitching need to be done before I have a reliable workflow for generating these 3D meshes. At this point, the end of this exploration would see these meshes appear in my pitch project game for Easter Egg purposes.

Alright so that's what has happened these last two weeks. My 3D printed RGBD camera mount version 2 is almost done and I have more direction on the 123D front as well. If the camera mount needs another week worth of work then I am prepared to setup another temporary RGBD configuration and start testing out my ideas in a more limited manner.

It's the FUTURE!

Things have been busy over here. This past week had me spend an entire day working with the lower Cohort and their first client meetings. Nothing that was super taxing but it did eat up a day of work. However I still managed to make some headway in doing......something.... with 3D.

First off is some amazing help that I have been receiving from Kimberly Voll, one of the great professors here at the CDM. She has in her possession a 3D printer that she has generously been allowing me to run stuff off on. And here is the first thing!

So what is that piece of black plastic you ask? It is a camera mount. Frist some background, in order fo the RGBD toolkit to properly and reliably take 3D footage, it needs to have a locked spatial relationship between a DSLR and a Kinect. To achieve this one needs a special camera mount. On the RGBD website they recommend constructing your own camera mount using aluminium, drill presses and other traditional fabrication materials. This is not an option that I wanted to pursue because it would be hard to find those materials and IT'S THE FUTURE. So in place of losing a finger in the process of fabricating a metal camera mount, Kim has allowed me to use her 3D printer to print out a plan for a RGBD camera mount that I found on the 3D warehouse. 

It's not perfect, in fact this first model doesn't work at all. The problem is that the arms with the hooks have too small an angle and overall the print is a little small. No big surprise there since this was my first ever 3D printing attempt. So I took the model into Maya, installed some special plugins to allow me to export in an .stl format and got to modifying the model. I extended the arms, increased the angle and made those hooks on the arms less intense. I've sent the plan back to Kim for printing and I hope to have another version by next week.

While this is great progress, it is still not the last thing I need to do in order just to get the system working as intended. I also needed to order a quick release plate mounting system from Amazon.ca which will plug into the mount and allow the DSLR to sit securely. This particular piece should be in my greedy hands in two weeks.  =[ 

But that's not good enough!! While the above is progress in terms of gathering the materials needed for the toolkit, it was not progress on the software end. So I spent a lot of time this week trying to get the system working in some manner.

First, more background. RGBD calls for a DSLR with live mode, which allows for the camera to output high quality .mov files. These files are used for both the calibration and the filming portions of the toolkit. Such a DSLR is not available at the CDM, since they are extra expensive. In place of this camera I managed to get my hands on a digital video camera that outputs files in .mkv files, or something else that RGBD doesn't know what to do with. Not to be deterred, I figured out how to get those .mkv files into Adobe Media Encoder and exported as a .mov, thus allowing RGBD to pick up and continue it's process of calibration and recording.

This is not a perfect fix and in fact it is less that ideal. Firstly, the viewing area on the video camera is smaller than that of a proper DSLR with live mode, which means that overall the image will look closer to the subject that how the RGBD toolkit is expecting it to. To combat this the kinect and video camera needed to be much further apart than intended in order to capture enough footage to be useful for calibration. What this means is that I had a video camera on a stack of books at one of the my table, and the Kinect perched on the edge on the other end of the same table. The distance between the two cameras was just over a foot, way more than the system was designed for.

But there is more difficulty. In order to get the footage converted and into the system I needed to remove the memory card from the video camera without disturbing the placement of either camera. This required surgical (ok maybe not) steady hands so as not to disturb the relationship between the cameras.  Somehow, with all this converting and boot strapping I managed to get the thing calibrated and capturing video.

BREAKTHROUGH!! This was a very exciting momement when the calibration worked and I was able to start capturing images. Below are some images of the calibration screens and editing process.

This one above shows the initial recording screens without any color maps applied. This is the raw data from the Kinect being interpreted by the RGBD toolkit. 

This one above shows the two footage streams coming together. You can see how the color image is off center from the depth map images, making for a strange scene. You can see the two streams separately in the upper right corner. 

Finally this above image is when the two streams have been somewhat lined up. Due to the imperfect setup of my hardware I was unable to achieve a closer match. However, just getting everything to this point is a big success in my book. 

Lastly this is the video that I was able to create using my makeshift setup. Again it's not perfect but it is a working proof of concept! 

So where to go from here? Well now that I know I can get the whole system working even with non specific parts my next steps are to refine this approach with the correct hardware as well as further experimentation with the editing software. Originally I wanted to be able to pull out the 3D mesh and take it into Maya, or go the other direction and begin placing my own 3D meshes into the RGBD timeline. Such an application does not seem to be ready built into the system, but maybe I'll be able to find the data files somewhere and start playing with them.

Exciting times! Much progress was made but there is still a lot to do and many questions to answer in terms of application and direction.