A single ECS-DoT System on Chip (SoC) now sees and hears at the same time. A camera watches for a person, and only while someone is in front of it does the microphone open to listen for spoken commands. No image or audio ever leaves the device.

Most edge AI designs do one thing: they detect a face, or they spot a keyword. Real products need more than that. A smart thermostat, a battery-powered door panel or a wearable must combine what it sees with what it hears, and it must do so on a power budget measured in milliwatts. This post walks through a reference design that does exactly that on the ECS-DoT.

The Platform: One Small Chip, Two Sensors

The reference design runs on the ECS-DoT development kit (DVK), with the main core clocked at 50 MHz and 512 KB of on-chip memory. There is no external memory. Everything (camera driver, microphone driver, two neural networks and the audio front end) fits in one 447,392 -byte firmware image, using 85.3% of the chip’s memory.

The camera and the microphone share no pins and no DMA channels. That lets audio keep streaming into memory in the background while the core is busy looking at a camera frame.

Vision: Is Anyone There?

The vision half answers one question: is there a face in front of the camera? It started life as a standalone face detection reference design on the ECS-DoT and has since been optimized to reduce latency down to 13.8 ms without loss of accuracy.

• Capture. The OV5640 delivers 320 × 240 frames. As each frame arrives, the chip gathers its color and brightness statistics. It then balances the color, stretches the contrast, halves the resolution to 160 × 120, pads it to a 160 × 160 square, and converts it to the 8-bit format the neural network expects.
• Model. A compact, 8-layer version of the OneFace detector, pruned from the original 12-layer network and retrained, with about 17,000 parameters. It runs entirely on the chip’s DLA (deep learning accelerator). The network divides the image into a 20 × 20 grid. Each cell scores how likely it is to hold a face and estimates the face’s size, position and five landmarks. The highest-scoring cell wins.
• Decision. A face counts only when it clears two thresholds together, an absolute score and a margin above the rest of the scene, which keeps false alarms down.
• Speed. Inference is 9.3 million multiply-accumulates per frame and takes about 13.8 ms on the DLA.

On 114 held-out test frames from the OV5640, the detector found 72 of 74 faces (97.3%) and raised no false alarms on the 40 frames without a face.

Audio: What Did They Say?

The audio half is a keyword spotter that listens for five commands: up, down, left, right and go, plus a sixth class for everything else. It also began as its own ECS-DoT reference design before being merged in and optimized for faster inference time on the DLA.

• Trigger. The keyword spotter only listens while the camera sees a face; with nobody in front of the device, the microphone stays off. While it’s listening, a sound clearly louder than the room’s background (three times its ambient level) starts a 1-second window. The window begins one 200 ms chunk early, so the start of the word is never clipped.
• Features. The CPU condenses that second of audio into a mel spectrogram, a picture of how its frequencies change over time, and shrinks it down for the network. It processes each chunk as soon as the DMA delivers it, so about three-quarters of the preprocessing is done before the word ends.
• Model. A small two-layer neural network, with 256 inputs, 64 hidden units and 6 outputs, and about 17,000 parameters. Like the face detector, it runs entirely on the DLA, using 8-bit integer math. Each output scores how likely the clip is to be one of the five commands or background sound. The highest-scoring class wins, but a word is reported only if the model is at least 70% confident.
• Speed. Inference is 16,768 multiply-accumulates per audio frame and takes 174 µs on the DLA.

Bringing It Together: The Camera Decides When to Listen

The two halves are joined by a simple rule: the microphone is open only while a face is in view. Vision acts as the gate, and keyword spotting is only performed when the ECS-DoT detects a face. Audio streams into two alternating 200 ms buffers: while one fills, the CPU works on the other, so recording never pauses, and most of the preprocessing is done before the word ends. The two AI models never run at the same time, but rather alternate between the two mentioned above.

Why let the camera decide when to listen? A microphone that opens only when someone is present never captures audio of an empty room, and the keyword spotter can’t be falsely triggered by background noise when nobody is there to speak. Privacy and reliability come from the architecture, not from a setting.

 

Figure 1. The Presence Gate and its Three States.

A single missed face check does not close the microphone; the chip keeps listening and checks again. The microphone only closes after two misses in a row.

The tricky part is that face checks keep happening while the microphone is open. A word spoken during a check is processed straight after, so it is heard a moment late but never lost.

The Numbers

All timing numbers below were measured on the board, using the chip’s own cycle counter.

Power figures cover the SoC’s core supply only, not the camera, microphone or the rest of the board. The energy figures are upper bounds, taken at peak current for the whole stage. Power is measured differentially from the inference power and subtracting the baseline power when no inference is running on the core voltage rail, giving us the power consumed by the DLA.

What’s Next

This reference design shows that vision and audio can run together on one ECS-DoT, with no cloud, no external memory and a core power budget of a few milliwatts. The architecture keeps low power in mind: one sensor (camera) decides when another should switch on (microphone). In a finished product, ECS-DoT sits alongside the host processor as an always-on AI co-processor, so the host can stay asleep until there is a person in view and a command to act on.

EMASS plans to develop a truly multimodal model that performs a single model combining both the audio and visual data for the same inference. This reference design is the first combining audio and vision, and it is just the start. With the ECS-DoT’s powerful deep learning accelerator, the possibilities are endless for ultra-low power intelligent edge solutions.

The reference design builds and flashes natively from the EMASS IDE on Windows, and a browser dashboard shows the live state: whether a face is seen, whether the microphone is open, and the last word heard.