This repository contains the dataset and code for the ACL Findings 2024 paper "Semantic Role Labeling from Chinese Speech via End-to-End Learning", by Huiyao Chen, Xinxin Li, Meishan Zhang*, Min Zhang. The dataset is included in the data/ folder. Please cite our paper if it is helpful to your work.
To run our code, please install all the dependency packages by using the following command:
pip install -r requirements.txt
For CPB1.0 dataset, the predicate-argument relations needs to be downloaded here, which is not publicly available, so we only provide the corresponding recording files. For AS-SRL dataset, we provide jsonl files that have been preprocessed with the data format that needs to be provided during the training process:
{
"id": int,
"audio": {
"path": str
},
"sentence": str,
"duration": float,
"srl": [{"pred": str, "position":[int, int], "arguments": [{"value": str, "position":[int, int], "role": str}]}]
}
Download the corresponding Whisper and BERT models in hugging face.
To pretrain Whipser, run:
bash pretrain_whisper.sh
python merge_lora.py --lora_model= --output_dir=
To construct the alignment tool, run
python generate_align_data.py
cd fast_align
bash fast_algn.sh
All models, whether they train only SRL or a combination of speech and SRL, are implemented using the train.sh file, except for individual parameter settings that are different. If you want to train the model, run:
bash train.sh
For SRLGParser(oracle), set parameter:
is_e2e=False
For SRLGParser(+GS AUG), in addition to being consistent with the SRLGParser(oracle) training parameters described above, the data needs to be generated by Gumbel Softmax first and then merged with the original training data:
python generate_aug_data.py
python merge.py
For Whisper◦SRLGParser(gold SRL) which is a end-to-end learning model, set parameter:
is_e2e=True
is_remove_augment=True
For Whisper◦SRLGParser, which is a end-to-end learning model and performs best in the SRL from speech, set parameter:
is_e2e=True
is_use_gumbel_softmax_loss=True
Notes If you want to reproduce the result of the paper, use get_metric_old.py when training, we have updated the mapping function to get better alignment in get_metric.py.
python evaluation.py
If you use the code, please cite this paper.
@inproceedings{chen-etal-2024-semantic,
title = "Semantic Role Labeling from {C}hinese Speech via End-to-End Learning",
author = "Chen, Huiyao and
Li, Xinxin and
Zhang, Meishan and
Zhang, Min",
editor = "Ku, Lun-Wei and
Martins, Andre and
Srikumar, Vivek",
booktitle = "Findings of the Association for Computational Linguistics: ACL 2024",
month = aug,
year = "2024",
address = "Bangkok, Thailand",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.findings-acl.527/",
doi = "10.18653/v1/2024.findings-acl.527",
pages = "8898--8911",
abstract = "Semantic Role Labeling (SRL), crucial for understanding semantic relationships in sentences, has traditionally focused on text-based input. However, the increasing use of voice assistants and the need for hands-free interaction have highlighted the importance of SRL from speech.SRL from speech can be accomplished via a two-step pipeline directly: transcribing speech to text via Automatic Speech Recognition (ASR) and then applying text-based SRL, which could lead to error propagation and loss of useful acoustic features.Addressing these challenges, we present the first end-to-end approach for SRL from speech, integrating ASR and SRL in a joint-learning framework, focusing on the Chinese language. By employing a Stright-Through Gumbel-Softmax module for connecting ASR and SRL models, it enables gradient back-propagation and joint optimization, enhancing robustness and effectiveness.Experiments on the Chinese Proposition Bank 1.0 (CPB1.0) and a newly annotated dataset AS-SRL based on AISHELL-1 demonstrate the superiority of the end-to-end model over traditional pipelines, with significantly improved performance."
}