I am still waiting on a 3D print for my camera mount which would allow me to finalize the calibration of the RGBD setup and turn the whole system into a more mobile iteration. Since I don't yet have said camera mount in my greedy little hands I can only make temporary camera setups and shoot stuff in a stationary way. So that's what I did in order to explore what kind of interesting images I could create using my own fine art skills and this new RGBD toolkit.
In the above you can see how I am working things out. The Kinect is in front, the video camera behind and sitting on some books to raise it up enough, the calibration grid and my trusty macbook ready to process all the information. This setup is not ideal, as I have said before, but now I understand even more so why it's not so good. A major problem is that the video camera has a different aspect ratio than that which is recorded by the Kinect, making alignment extra difficult during calibration. Not to be deterred I pressed onward and got the best calibration images that I could. Below is a screen shot of the process.
Here you can see my using the calibration gird and recording images in RGBDcapture, the open source program I am using for this. Problems start to become apparent when comparing these images with the ones captured from the video camera because one can notice that the frame of recording is quite different between the two devices. Still, not to be dissuaded I pressed forward and created as many good quality calibration images as possible.
From there it was a matter of bringing the two different sets of images together and matching them up so RGBD can figure out the relationship between the two cameras. In order for that to happen RGBD needs to run a python script on collect of .mov files that were generated from the non Kinect camera. That's all well and good but my files needed to be converted to .mov from .mkv. Fine again, but in order to get at tose files I needed to gingerly remove the memory card from the video camera without nudging or otherwise changing the position of said camera. Moving it too much would result in my calibration efforts being wasted as I would have just changed the relationship between the two cameras. Anyway, I got the card out and started converting all my files. Below is a screen shot of the process.
Great! So that went well and then it was on to matching up those two sets of calibration images. Pretty straight forward as you can see by the screen shot below. If you look closely you can see that there are two rows of images, each relating to each other, which help RGBD to determine the relationship between the two cameras. You may also be able to see that there is quite a bit of difference between the two images since I was using a non recommended video camera.
None the less I still got what I thought to be enough calibration images to inform the system of how the cameras were related to each other in real space. From here it was onto recording.
Recording was probably the easiest part of the whole process since most of the work happened outside the computer as I was trying to make interesting things happen in front of the camera. So I completed a number of actions in front of the camera setup to allow for me to experiment with the footage. Here is a screen shot of what that looked like on the screen during recording.
Looks pretty interesting eh? Well kind of I suppose. This is just half of the footage however, this being the depth. In addition to this one needs to have the color information of the footage in order to overlay the two together and create compelling 3D images. So from here it was a simple matter of converting the files from the video camera into a format that RGBD could understand. Here is a screen shot of that process.
So great! At this point I had both the color and the depth recorded and was ready to start the editing process. For the next two hours this is what I kept getting to appear on my screen.
What the crap is that?! At this point I had spent 6 hours setting up, calibrating, shooting and converting everything. It was extremely disheartening to see my work go nowhere due to a software crash that I had no idea how to fix.
With a heavy heart I broke down my camera setup and archived my files for later contemplation. Maybe I could use the files generated in some other way? I had no idea what to do next except go get some comfort food and watch Dexter (Yes I was working on Sunday).
Then something amazing happened just as I was trying to fall asleep that night. I was replaying my process over and over in my head (I know I should think of other things before bed) and I was trying to pinpoint where the problem might have been. I knew that the new files crashed upon opening, but my old files still opened just fine and allowed for me to make new compositions. So the problem could not be in the application being corrupt itself, but rather somewhere in the files that I was supplying. I had looked over the directory structure of the two separate recordings and they seemed the same in terms of the files that I had captured. It was about this time that I thought, "what if the calibration is so wrong that it is crashing?"
That was it!! The next day I took the old calibration that I was using for the successful recording that I did weeks ago and put that into the new recordings directory structure. This mean that the calibration would be way off since the relationship between the two cameras was different from the two different shooting days, but I didn't care I just wanted to see if I could get the darn thing to open. And it did open!!!
So here is another example of why I need to get that 3D printed mount and a correct DSLR camera with Live Video mode. The video camera I was using had a different aspect ratio from the Kinect and the frame was smaller which made it impossible to get an ideal set of calibration images. Apparently it was so bad that RGBD just refused to open. Once I had placed working (if not the correct) calibration files into the directory structure, RGBD was able to keep going! So it was not ideal but it opened up my work again and I was able to proceed with editing. THANK GOODNESS!
So I edited some of those videos that I shot and experimented with what RGBD can do. I edited 3 videos, 1 person (myself), one performance with practice juggling flags, and one brief experiment with colored foam core cutouts. The results informed me a great deal of what works, what is interesting and how I could make it better. Needless to say, with a better registration setup and the correct camera I would have been able to achieve a better alignment of depth and color. Anyway, ENOUGH TALK! Here are those videos for your viewing pleasure:
So whats next? Well that sliding head camera mount showed up in my mailbox the other day so I am a step closer to assembling a completed RGBD camera mount. I'm still no closer to the $800 it would cost to get a proper DSLR, but I am friends with a guy who has one.... So next up is to check out how the 3D print is going and see how I can assist the process. If the mount needs more time then I have more experiments with 123D which I can do in the meantime.
Thanks for reading! More to come next week!
No comments:
Post a Comment