First thing's first: why am I bothering with trying to connect an 8 year old depth sensor to UE4? Well it's mostly because I'm too stubborn and lazy to go to the mocap studio on campus when I have a perfectly usable (?) Kinect right at home! Is that a good reason? No! But nonetheless this whole thing proved to be a fun little exercise in learning Unreal and refactoring old C++, so I have no real regrets about doing it. You can find my work on my github repo for this project.
Naturally this all started with some research. What were the current prospects from markerless mocap like for Unreal 4? I already knew about LiveLink and Rokoko thanks to LSU's onsite mocap studio, but what if I can't be on campus for some reason and need a quick and cheap alternative? Yes I know the data isn't nearly as clean, but that's not really what this is about for me. I'm more interested in what's possible, even in its earliest, roughest stages. As far as markerless setups go, I'm somewhat familiar with iPi Soft from my earlier forays into mocap. The price tag, however, is not doable for me right now (and my school can't cover it either). I should note that my goal for this is live motion capture, and simply recording it from another software (like mesh-online's mocap recorder) is certainly doable. It should also be noted that a lot of these solutions require a much newer Kinect than the one I'm using (which was to be expected). So those are two limitations already in place: it has to be live, and it has to work with the Kinect v1. So what next?
My digging led me to a GitHub repo initally created in 2014, and last updated in 2017. AleDal's KinectXbox360-UE4 plugin seemed to be exactly what I needed. I look at the installation instructions, which seemed simple enough:
So I create a new C++ project in Unreal, close Unreal to load in the plugin, reopen the project to enable the plugin, and try to restart to compile it. And I get a message saying that one of the KinectPlugin module is missing or was built in a different version (which makes sense).
So then I hit yes to rebuilding, and let it do its thing. A few minutes later I get this very helpful message:
Now you should know that I actually didn't know what that meant, as I neither use C++ all that often nor have I ever even made a C++ Unreal Project before. So I tried the Generate Visual Studio project files option as a more explicit compilation, yet that also continued to fail.
I do a bit more research and find out that I can just try to compile it with Visual Studio to get actual errors that I can (attempt to) fix! And I do just that. Opening up the VS Solution and building certainly gave me an informative slew of errors that I could start working my way through:
This is the part where I start refactoring all of the outdated code. The first lines I tackled were the Cannot open include file: 'ModuleManager.h': No such file or directory and the Cannot open include file: 'AllowWindowsPlatformTypes.h': No such file or directory.
After some digging online, it turns out that these depenedancies required a more explicit location in newer versions of Unreal. In the original files, the lines that were causing trouble were: #include "ModuleManager.h", #include "HideWindowsPlatformTypes.h", and #include "AllowWindowsPlatformTypes.h". The fix for these was simply to change them to #include "Modules/ModuleManager.h", #include "Windows/HideWindowsPlatformTypes.h", and #include "Windows/AllowWindowsPlatformTypes.h"
With another build (and a prayer), I hoped that resolving those include statements would fix my issue. Of course, here's what I get now:
That's a lot more than I got originally! If I know anything about programming (debatable) it's that this is probably caused by something deceptively simple. I first notice that the bulk of these errors coming from roughly the same code chunk in the KinectSensor.cpp file. My instincts told me to look into that ENQUEUE_UNIQUE_RENDER_COMMAND_FOURPARAMETER.
Like I said earlier, I rarely use C++, so I had no idea what I was looking at here. I would soon learn that this is C++'s flavor of a lambda expression. As for the ENQUEUE function itself, I found out from here that its parameters worked like so:
ENQUEUE_UNIQUE_RENDER_COMMAND_FOURPARAMETER(TypeName,ParamType1,ParamName1,ParamValue1,ParamType2,ParamName2,ParamValue2,ParamType3,ParamName3,ParamValue3,ParamType4,ParamName4,ParamValue4,Code)
The problem with this function is that it's been deprecated in the newer API, and needed to be converted into the newer equivalent of ENQUEUE_RENDER_COMMAND. The only thing I could find about reworking this function is from this forum post. The response on it wasn't directly helpful, but it was enough to get an idea of what needed to go where. The lambda expression in C++ takes the form of:
[ captures ] ( params ) lambda-specifiers requires(optional) { body }
All I did in my solution was define and assign the captures before the lambda, as the next few errors were complaining about undefined identifiers. For the structure of the lambda, I just mirrored what was laid out in the forum post as a start. I passed in the original params as captures to the lambda, and the body itself stayed the same.
const void* ImageData = buffer;
int32 Stride = Width * 4;
FKinectTextureParams Params = m_RenderParams;
ENQUEUE_RENDER_COMMAND(ImageData)(
[Stride, Params, &ImageData] (FRHICommandListImmediate& RHICmdList) {
RHIUpdateTexture2D(Params.Texture2DResource->GetTexture2DRHI(), 0, *Params.UpdateRegions, Stride, (uint8*)ImageData);
});
We compile and...
It looks so much better already! Now just a few more minor errors to clean up. I start with the FTexture2DResource. This is yet another instance of a deprecated type, which was replaced by FTextureResource. I just replaced all mentions of the original (in both the .cpp and corresponding header file) with the new type. As for the fmin error, I added an include statement for the cmath library. I'm still not too sure what this one is about honestly. But after making these changes, things compiled with no error!
Is this done properly? I have no idea, since I haven't tested the texturing from the plugin yet. But this was enough for it to compile properly, and any fixes that need to be done can be done later. Now I head back into the engine to see what the acutal Blueprints have in store for me.
The only things I really had to do were to drop in a copy of the BP_Bones, assign the skeletal mesh to the PoseableMesh (I just used the one provided in the examples from the OG repo), and added some print string nodes just to make sure my Kinect and skeleton were acutally being detected. And now for results!
Live mocap capture in Unreal 4 using a Kinect v1! It's super janky, but it's fascinating that it still works after all these years (with just a little elbow grease on my part). Next I'll need to figure out how to retarget this skeleton to another character. But for now, I'm content with it as is. Thanks for reading, and see you next time!