Multimodal Learning

In this review, we summarize the latest research findings and analyze the developmental trends of driver fatigue detection. We present the work on integration of RGB-D camera and deep learning, where Generative Adversarial Networks and multi-channel schemes are utilized to enhance the performance.

In this paper, we show a representation grouping effect during this process: the InfoNCE objective indirectly groups semantically similar representations together via randomly emerged within-modal anchors. We introduce Prototypical Contrastive Language Image Pretraining (ProtoCLIP) to enhance such grouping by boosting its efficiency and increasing its robustness against modality gap.

This paper proposed the First deep-learning based music-driven conducting motion generation method, and presented a large-scale music motion dataset ConductorMotion100 with unprecedented 100 hours length.

In this paper, we construct a large-scale Multi-modal E-commerce Products classification dataset MEP-3M, which consists of over 3 million products and 599 fine-grained product categories. This paper won Best Dataset Paper award.

In this demo, we present the VirtualConductor, a system that can generate conducting video from a given piece of music and a single user’s image. This demo won the ICME 2021 Best Demo award.