Contact us
All posts

· Updated

Sounds right: an introduction to audio classification

Audio classification answers one question: what is this sound? A model listens to a short window of audio and returns a label, or a set of labels with scores. Glass breaking. A dog barking. A pump bearing starting to fail. The wake word “hey device”. The output is a decision, not a transcript.

Where it sits among audio AI tasks

  • Speech recognition (ASR) turns speech into text. It cares about which words were said.
  • Text-to-speech (TTS) goes the other way and generates a voice from text.
  • Audio LLMs take audio as input and reason about it in language. They can transcribe, summarize, answer questions about a recording, or describe a scene.
  • Audio classification maps a sound to one of a known set of categories, or to several at once.

The lines blur. Keyword spotting involves speech, but it is built as a classifier over a small vocabulary. In practice, classification is the task you run all the time on a small budget, and the others are what you call once it says something worth hearing has happened.

How it is done today

Features: log-mel spectrograms

Almost every modern pipeline starts the same way. The waveform is cut into short overlapping frames, each frame becomes a frequency spectrum, that spectrum is mapped onto the mel scale (which spaces frequencies roughly as human hearing does), and the energies are log-compressed. The result, a log-mel spectrogram, is a 2D image of time against frequency. Hand-crafted features such as MFCCs are still used in very small models, but most networks now learn directly from log-mels.

Pretrained audio encoders and transfer learning

The biggest change since the SVM and LSTM era is that you rarely train from scratch. Large encoders are pretrained once on huge general audio collections, then fine-tuned or used as frozen feature extractors for a specific task with a few hundred or a few thousand labeled examples.

Much of this work builds on AudioSet, Google’s collection of about 2 million human-labeled 10-second clips drawn from YouTube, organized under an ontology of 632 sound event classes. Two well-known encoders show where the field went:

  • AST (Audio Spectrogram Transformer) from MIT treats the spectrogram as a grid of patches and runs a standard transformer over it, with no convolutions. The authors describe it as the first purely attention-based audio classifier.
  • BEATs from Microsoft is pretrained with self-supervision: it learns to predict discrete “acoustic tokens” from masked audio, so it learns from unlabeled sound before seeing any labels.

Labeled audio is expensive and unlabeled audio is cheap, which is why self-supervision matters. The usual recipe for a new product: start from a strong pretrained encoder, fine-tune on your own labeled data, and spend your effort on that data.

Audio-language models for zero-shot labels

A newer family of models learns a shared space for audio and text. CLAP (Contrastive Language-Audio Pretraining) from Microsoft, and LAION’s open CLAP models, are trained on paired audio and captions. To classify, you write the candidate labels as text (“a baby crying”, “a smoke alarm”, “running water”) and the model scores the audio against each one. No training is needed for new classes.

This is most useful early in a project: sorting field recordings, finding examples of a rare sound, or checking whether a category is separable before paying for labels. On a narrow task a fine-tuned model usually beats it, so treat zero-shot as a starting point and a labeling tool.

Audio LLMs go one step further. Qwen2-Audio, for example, pairs an audio encoder initialized from Whisper with a 7B language model and answers free-form questions such as “what is this sound?”. These models are flexible but far too large to run continuously on a small device, so they belong at the end of a pipeline.

Where small CNNs still win

On a microcontroller, a compact convolutional network on log-mels is still the default choice. CNNs quantize well to 8-bit integers, map cleanly onto embedded NPUs, and need only about a second of audio context. YAMNet, a MobileNet v1 model that predicts 521 audio event classes from AudioSet, is a long-standing example of this style. A common pattern is to use a big transformer as a teacher and distill its knowledge into a small CNN student that fits the target chip.

Applications

  • Wake words and keyword spotting. Detecting a fixed phrase or a small set of commands. Google’s Speech Commands dataset of one-second spoken words was built for exactly this kind of small, low-false-alarm model.
  • Sound event detection for safety and security. Glass breaking, smoke and CO alarms, screams, gunshots, a baby crying, a car horn or siren near a vehicle.
  • Machine condition monitoring. Spotting abnormal sounds from pumps, fans, valves and motors before they fail. Normal recordings are plentiful and failures are rare, so this is usually framed as anomaly detection. The DCASE challenge has run an anomalous sound detection task on this problem since 2020.
  • Health. Cough detection and counting, breathing and snoring patterns, and other body sounds, typically as a screening or trend signal rather than a diagnosis.
  • Wildlife monitoring. Identifying birds, bats and frogs from field recorders, and detecting chainsaws or vehicles in protected areas.

Running it on the edge

Why always-listening audio belongs on the device

A classifier that listens continuously has three reasons to run locally.

  • Privacy. Streaming a home’s or a factory’s audio to a server is a hard sell. On-device, raw audio never leaves; only events do.
  • Power. A battery-powered sensor cannot afford to keep a radio sending audio all day.
  • Latency and reliability. An alarm or a wake word needs a response in well under a second, whether or not the network is up.

Microcontroller constraints

Always-on audio often lives on a microcontroller. The constraints are concrete:

  • Memory. Activations and the audio buffer share on-chip SRAM, often a megabyte or less, while model weights usually stay in flash or MRAM next to the firmware. Even a high-end part like the Alif Ensemble E3 below has at most 13.5 MB of SRAM and 5.5 MB of MRAM.
  • Power budget. The always-on path must run continuously on a battery or a small supply, so the processor spends most of its time asleep between frames.
  • Quantization. Embedded NPUs run integer math. Arm’s Ethos-U NPUs, for example, require integer-quantized models before they can be accelerated: 8-bit weights, with 8-bit or 16-bit activations. Quantization-aware training usually recovers most of the accuracy lost in the conversion.
  • NPUs. A new generation of MCU-class parts includes a dedicated NPU next to the CPU. Alif Semiconductor’s Ensemble E3 pairs two Cortex-M55 cores with two Arm Ethos-U55 NPUs, has up to 13.5 MB of SRAM, and has low-power PDM microphone inputs. NXP’s i.MX RT700 crossover MCU adds an eIQ Neutron NPU and a separate always-on sense compute subsystem with its own Cortex-M33 and HiFi 1 DSP. Further up, NXP’s i.MX 93 application processor includes an Arm Ethos-U65 microNPU.

The cascade pattern

The design that ties this together is a cascade. A tiny model runs all the time on the lowest-power core with a deliberately low threshold. When it fires, it wakes a larger, more accurate model on the NPU, an application processor or a gateway, which confirms the event. Only confirmed events trigger the expensive work: ASR, an audio LLM, a cloud call, or an alert.

Average power stays close to that of the smallest model, while the false-alarm rate is set by the largest. Tune the first stage’s miss rate carefully, because anything it misses is gone for good.

Practical challenges

  • Noisy real-world data. Clean benchmark clips do not prepare a model for HVAC hum, traffic or reverberant rooms. Collect data where the device will live and augment with real background noise.
  • Class imbalance. The events you care about are rare, so a model can look accurate by never firing. Measure false alarms per hour and miss rate on realistic streams.
  • Multi-label overlap. Real scenes contain several sounds at once. Use multi-label outputs with per-class thresholds, and label data with that in mind.
  • Evaluate on the target microphone. MEMS microphones, enclosures, gain settings and on-device preprocessing all change the signal. Test on audio captured by the actual device through the production front end. A model that scores well on a laptop microphone can fail on the product.

At EdgeOps.AI we build audio classifiers, ASR and TTS models, and audio LLMs, and deploy them on microcontrollers with NPUs, embedded boards and phones.