I downloaded and played around with ToonCrafter last night until 2AM. As usual, AI bros on social media are using terms like 'game changer!' and 'this new thing changes the old thing forever!'.
So let's break it down.
1. ToonCrafter looks neat, but as with all things generative AI, demos and real world use are very different. Demos are curated to show the best examples. In practise actual film and animation shots are so varied and expansive that an AI model often can't understand what's going on in a scene and will produce large errors that are very time consuming to manually repair.
2. Motion tweening between keyframes has been possible for around 35 years. It has already been used for tweening between keyframes for facial expressions since the 90s. Motion tweening certainly could do with updates and use machine learning but for it to be truly useful it has to be available inside apps like Clip Paint Studio, TVPaint and ProCreate Dreams. Animators need layered files, vectors, camera controls and reusable assets.
3. Continuing from the last point, because generative imaging outputs compressed video files there are no reusable assets to work with. That means recolouring, retiming, rescaling, rekeyframing and cleaning up is extremely difficult or often impossible.
4. The compute power required for ToonCrafter is very high, needing at least 24GB of video memory for efficiency, and even then it takes a very long time to output low res 512 pixel compressed videos. That's not something an animation team can work with. They'll spend hours and hours waiting for output and waste a lot of time feeling disappointed by the unpredictable output.
Even the output of the included demo assets can vary widely from useless to not good. Dropping down the resolution can allow it to generate on lower end GPUs but this is what you can expect from an RTX 4070…
5. Accurate lip synch is a known problem in generative video and here in ToonCrafter it would be impossible.
6. ToonCrafter currently won’t work on a Mac because some dependencies are incompatible. ComfyUI also has this issue because the sources for all the various plugins come from different places, some of which are more compatible with one operating system than another.
So as neat at ToonCrafter is, it's not a game changer. You might be able to use it to animate some small items that you can comp into a scene but you'll be disappointed trying to use it, now or in the future, on large scene elements and character animation. As always, AI bros just want clicks. They can be helpful with their tutorials, but they are unhelpful when they exaggerate things that are outside their field of knowledge.