Chinese-LiPS Dataset

Dataset Overview

The Chinese-LiPS dataset is a multimodal audio-visual speech recognition (AVSR) dataset. The dataset includes:

• Approximately 100 hours of multimodal data, including speech, video, and manual transcriptions.
• 36,208 video clips featuring 207 professional speakers, covering topics such as instruction, lectures, and educational presentations.
• Audio: Stereo WAV format with a 48 kHz sampling rate.
• Video: 1080p resolution for slides and 720p resolution for lip-reading, both at 30 fps.
• Annotations: JSON format with transcriptions.

Download

Dataset Splits

Dataset Access

You can access the dataset using the links below:

• Download from OneDrive
• Download from Huggingface
• Download from Baidu Netdisk (提取码: vg2a)

Dataset Organization

The dataset includes three main modalities: audio, slide video, and lip-reading video. The dataset is organized into several files:

• image.zip: This file contains the first frame of each slide video segment in the test dataset, which is used for OCR and visual language model processing. The images are in JPG format.
• processed_train.zip, processed_test.zip, processed_val.zip: This files contain data processed using the Whisper-Flamingo library. It includes the full dataset with 16 kHz audio, 96×96 lip-reading videos with 25 fps.
• train.zip, test.zip, val.zip: These files contain the training, testing, and validation sets. Each zip file has the following structure:

  train.zip/test.zip/val.zip
  ├── ID1_age_gender_topic/
  │   ├── WAV/
  │   │   ├── ID1_age_gender_topic_001.json         # Audio segment annotation file
  │   │   ├── ID1_age_gender_topic_001.wav          # Audio file (48 kHz)
  │   ├── PPT/
  │   │   ├── ID1_age_gender_topic_001_PPT.mp4      # Slide video file (1080p 30fps)
  │   ├── FACE/
  │   │   ├── ID1_age_gender_topic_001_FACE.mp4       # Lip-reading video file (720p 30fps)
  ├── ID2_age_gender_topic/
  │   ├── WAV/
  │   │   ├── ID2_age_gender_topic_001.json
  │   │   ├── ID2_age_gender_topic_001.wav
  │   ├── PPT/
  │   │   ├── ID2_age_gender_topic_001_PPT.mp4
  │   ├── FACE/
  │   │   ├── ID2_age_gender_topic_001_FACE.mp4
  ├── ...
  

• asr.zip: Contains the complete datasets from train.zip, test.zip, and val.zip, organized by TOPIC. Zip file has the following structure:

  asr.zip
  ├── topic1/
  │   ├── ID1_age_gender_topic/
  │   │   ├── WAV/
  │   │   │   ├── ID1_age_gender_topic_001.json         
  │   │   │   ├── ID1_age_gender_topic_001.wav          
  │   │   ├── PPT/
  │   │   │   ├── ID1_age_gender_topic_001_PPT.mp4      
  │   │   ├── FACE/
  │   │   │   ├── ID1_age_gender_topic_001_FACE.mp4     
  │   ├── ID2_age_gender_topic/
  │   │   ├── ...
  ├── topic2/
  │   ├── ...
  

• meta_all.csv, meta_train.csv, meta_valid.csv, meta_test.csv: These metadata files contain the fields ID, TOPIC, WAV, PPT, FACE, and TEXT.

The TOPIC field is abbreviated in Chinese as follows: DZJJ = E-sports & Gaming, JKYS = Health & Wellness, KJ = Science & Technology, LY = Travel & Exploration, QC = Automobile & Industry, RWLS = Culture & History, TY = Sports & Competitions, YS = Movies & TV Series, ZX = Others.

Citation

@misc{zhao2025chineselipschineseaudiovisualspeech,
  title={Chinese-LiPS: A Chinese audio-visual speech recognition dataset with Lip-reading and Presentation Slides}, 
  author={Jinghua Zhao and Yuhang Jia and Shiyao Wang and Jiaming Zhou and Hui Wang and Yong Qin},
  year={2025},
  eprint={2504.15066},
  archivePrefix={arXiv},
  primaryClass={cs.MM},
  url={https://arxiv.org/abs/2504.15066}
}

Contact

If you have any questions or suggestions, feel free to reach out via email at zhao1jing1hua@gmail.com.