AMC Assistant Professor Wei XUE has received Best Paper Award at ACM Multimedia 2026 for the research paper“Discrete Coding and Masked Modeling for Text-to-Stereo Audio Generation.” ACM Multimedia is the most influential international conference in the field of multimedia.
Co-authored by researchers from HKUST, The Chinese University of Hong Kong (CUHK), and Dolby Laboratories, the study explores how AI can generate stereo audio directly from text descriptions while capturing both sound content and spatial relationships. The research introduces three key components: StereoCodec, which represents spatial audio as discrete tokens; StereoGen, which generates stereo audio from these representations; and StereoCLAP, which evaluates the semantic and spatial consistency between text descriptions and generated audio scenes.
Research Impact
The work anticipates future needs in the audio industry by building foundational components for spatial audio AI: discrete spatial audio tokens, a foundational generative model, and an evaluation model for semantic and spatial consistency. These components can support immersive stereo sound generation for film and television, game audio generation, virtual tours, and artistic creation. They also provide building blocks for immersive VR experiences and future audio world models.
The paper is co-authored by Yiming LI, Zhen YE, Jiahao PAN, Sitong CHENG, Boyi KANG, & Wei XUE from HKUST, Peiwen SUN from The Chinese University of Hong Kong (CUHK), Jia DAI, Kai LI, & Lie LU from Dolby Laboratories.