This project aims to develop an android app with an end-to-end speech recognition system that converts the in-cabin announcements over the public address system to text without the use of server. Our approach has three fundamental steps, as shown in Fig 1 –
- De-noising algorithm – The in-cabin flight announcements are degraded by uncorrelated additive noises like engine noise, reverb, crowd talking, etc. It is essential to enhance the signal by de-noising it to improve speech recognition accuracy.
- Acoustic model – Maps the denoised audio’s MEL spectrogram features to their respective characters.
- Post-processing algorithms/ Spelling correction – Corrects the spelling of the native decoder to improve word readability and lower CER (Character Error Rate).
We explored several de-noising algorithms. We implemented a Logarithmic Spectral Amplitude-based signal estimator and extended a neural net-based DeepMMSE algorithm-based noise estimator in Java. This algorithm can track stationary and non-stationary noises like a baby crying, crowd talking, and other airplane interior noises with high precision on the phone, as seen in Fig 2. When integrated with the acoustic model, we achieve the target translation error of <20% for various background noises for SNR as low as 0dB.
We implemented an end-to-end deep learning based acoustic model on TensorFlow 2.4. The architecture has 3 CNN layers and 4 BiLSTM layers with CTC loss function as shown in Fig 3. 80 MEL features are used as acoustic features. We train the model with 6.5k hours of data (LibriSpeech, Commonvoice, Fischer etc.), including augmentation, for over 20 epochs. We include a screenshot of a real-time demo of the app for +5dB SNR below.
Ground truth 1: “while you are getting comfortable in the cabin this is a reminder that all carry on baggage must fit securely either in the overhead compartments or under the seat in front of you”
Ground truth 2: “ladies and gentlemen on behalf of the captain and the rest of the crew I would like to welcome you aboard southwest airlines flight three with non stop service to phoenix”
