Upose3D Accurately Predicts 3D Keypoints From Multiple Views

It scales well with additional camera views at constant runtime

Abstract

We introduce UPose3D, a novel approach for multi-view 3D human pose estimation, addressing challenges in accuracy and scalability. Our method advances existing pose estimation frameworks by improving robustness and flexibility without requiring direct 3D annotations. At the core of our method, a pose compiler module refines predictions from a 2D keypoints estimator that operates on a single image by leveraging temporal and cross-view information. Our novel cross-view fusion strategy is scalable to any number of cameras, while our synthetic data generation strategy ensures generalization across diverse actors, scenes, and viewpoints. Finally, UPose3D leverages the prediction uncertainty of both the 2D keypoint estimator and the pose compiler module. This provides robustness to outliers and noisy data, resulting in state-of-the-art performance in out-of-distribution settings. In addition, for in-distribution settings, UPose3D yields a performance rivaling methods that rely on 3D annotated data, while being the state-of-the-art among methods relying only on 2D supervision.

Technical Video

BibTeX

@misc{davoodnia2024upose3d,
      title={UPose3D: Uncertainty-Aware 3D Human Pose Estimation with Cross-View and Temporal Cues}, 
      author={Vandad Davoodnia and Saeed Ghorbani and Marc-André Carbonneau and Alexandre Messier and Ali Etemad},
      year={2024},
      eprint={2404.14634},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2404.14634}, 
      doi={https://doi.org/10.48550/arXiv.2404.14634}
}