Zero-Shot Audio Captioning via Audibility Guidance

Tal Shaharabany, Ariel Shaulov, Lior Wolf
School of Computer Science, Tel Aviv University
arXiv Code Dataset Demo
Description of image

Illustrative instances of caption generation produced by our pipeline portraits.

Abstract

The task of audio captioning is similar in essence to tasks such as image and video captioning. However, it has received much less attention. We propose three desiderata for captioning audio - (i) fluency of the generated text, (ii) faithfulness of the generated text to the input audio, and the somewhat related (iii) audibility, which is the quality of being able to be perceived based only on audio. Our method is a zero-shot method, i.e., we do not learn to perform captioning. Instead, captioning occurs as an inference process that involves three networks that correspond to the three desired qualities: (i) A Large Language Model, in our case, for reasons of convenience, GPT-2, (ii) A model that provides a matching score between an audio file and a text, for which we use a multimodal matching network called ImageBind, and (iii) A text classifier, trained using a dataset we collected automatically by instructing GPT-4 with prompts designed to direct the generation of both audible and inaudible sentences. We present our results on the AudioCap dataset, demonstrating that audibility guidance significantly enhances performance compared to the baseline, which lacks this objective.

Audibility Dataset

Examples from the GPT-4 generated dataset comprising both audible and inaudible sentences. We prompted ChatGPT with the task of creating examples for a classifier that distinguishes between these two categories.

Audible Not-Audible
The barking of a dog in excitement. Magnets attract metals.
Ringing phone awaits an answer. Ice covers the lake in winter.
The buzz of a drone flying overhead. Icebergs float on water.
Birds chirping in early morning. An umbrella stands closed by the door.
Jingling coins are counted or played with. A statue in a park.
The swishing sound of a washing machine. Rusting car sits in the yard.
Whips cracked in the rodeo. Resolved issue is fixed.
The meow of a cat. Soccer balls are stored in a mesh bag.
The school bell rings, signaling the end of class. Skimmed milk has less fat.
Raindrops tapping on rooftops. A pair of hiking boots rests next to a backpack.

BibTeX

If you find this project useful for your research, please cite the following: 

@article{shaharabany2023zero,
        title={Zero-Shot Audio Captioning via Audibility Guidance},
        author={Shaharabany, Tal and Shaulov, Ariel and Wolf, Lior},
        journal={arXiv preprint arXiv:2309.03884},
        year={2023}
      }