AI4AnimationPy is a Python framework developed by Paul Starke and Sebastian Starke that enables character animation through neural networks while providing comprehensive tools for motion capture processing, training, inference, and animation engineering. The framework ports the AI4Animation project from Unity to Python, eliminating the Unity dependency for data processing, feature extraction, inference, and post-processing while maintaining a similar game-engine-style architecture based on Entity-Component-System (ECS) patterns and update loops. All computation runs on NumPy or PyTorch, allowing training, inference, and visualization to occur within a single unified environment without requiring external tools or communication pipelines.
The framework addresses a significant workflow problem in AI-driven character animation research. Previously, researchers had to juggle disconnected tools where model research happened in Python while visualization required specialized software like Unity, necessitating custom communication pipelines through ONNX or data streaming. This fragmentation slowed iteration and made real-time validation difficult. AI4AnimationPy solves this by consolidating everything into one framework, enabling researchers to train neural networks on motion capture data, visualize results instantly without switching tools, run headless for server-side training, and extend functionality easily through modular ECS design.
The framework supports three execution modes: Standalone mode uses the built-in rendering pipeline, Headless mode runs without visualization for server-side training, and Manual mode allows developers to control update loop invocation timing and frequency for local or remote execution. This flexibility accommodates various deployment scenarios from interactive development to production inference.
Core features include an Entity-Component-System architecture with lifecycle management, game-engine-style update loops with callbacks for Update, Draw, and GUI operations, and a comprehensive math library providing vectorized forward kinematics, quaternion operations, axis-angle conversions, matrix operations, and motion mirroring. The framework includes neural network implementations for MLPs, Autoencoders, and Codebook Matching with integrated training utilities. A real-time renderer provides deferred shading, shadow mapping, screen-space ambient occlusion, bloom effects, and FXAA antialiasing, with GPU-accelerated skinned mesh rendering for skeletal animations.
Additional capabilities include a FABRIK solver for real-time inverse kinematics, animation modules for joint contacts and trajectory generation, multiple camera systems with smooth blending, and motion import from GLB, FBX, and BVH file formats. The internal motion format uses NPZ files storing 3D positions and 4D quaternions for each skeleton joint per frame. A command-line interface enables batch conversion of entire motion capture directories.
The framework demonstrates significant performance improvements over the original Unity-based AI4Animation. Training data generation from 20 hours of motion capture takes less than 5 minutes compared to over 4 hours in Unity, while setup time for new experiments drops from over 4 hours to approximately 10 minutes. AI4AnimationPy supports backpropagation through inference and full PyTorch quantization, capabilities unavailable in the Unity version. Built-in visualization during training eliminates the need for external streaming infrastructure.
Interactive demos showcase the framework's capabilities across stylized biped locomotion controllers trained on style100 datasets, quadruped locomotion with gait transitions, future motion anticipation with training visualization, inverse kinematics solving, motion capture import workflows, and motion editing tools. Web-based demos are available for testing without local installation. Planned features include physics simulation with rigid bodies and collision detection, path planning and spline tooling, and audio support. The framework is licensed under CC BY-NC 4.0 and supports multiple public motion capture datasets including Cranberry, 100Style, LaFan, and others.
