Most edge AI designs do one thing: they detect a face, or they spot a keyword. Real products need more than that. A smart thermostat, a battery-powered door panel or a wearable must combine what it sees with what it hears, and it must do so on a power budget measured in milliwatts. This post walks through a reference design that does exactly that on the ECS-DoT.

The Platform: One Small Chip, Two Sensors

The reference design runs on the ECS-DoT development kit (DVK), with the main core clocked at 50 MHz and 512 KB of on-chip memory. There is no external memory. Everything (camera driver, microphone driver, two neural networks and the audio front end) fits in one 447,392 -byte firmware image, using 85.3% of the chip’s memory.

The camera and the microphone share no pins and no DMA channels. That lets audio keep streaming into memory in the background while the core is busy looking at a camera frame.

Vision: Is Anyone There?

The vision half answers one question: is there a face in front of the camera? It started life as a standalone face detection reference design on the ECS-DoT and has since been optimized to reduce latency down to 13.8 ms without loss of accuracy.

• Capture. The OV5640 delivers 320 × 240 frames. As each frame arrives, the chip gathers its color and brightness statistics. It then balances the color, stretches the contrast, halves the resolution to 160 × 120, pads it to a 160 × 160 square, and converts it to the 8-bit format the neural network expects.
• Model. A compact, 8-layer version of the OneFace detector, pruned from the original 12-layer network and retrained, with about 17,000 parameters. It runs entirely on the chip’s DLA (deep learning accelerator). The network divides the image into a 20 × 20 grid. Each cell scores how likely it is to hold a face and estimates the face’s size, position and five landmarks. The highest-scoring cell wins.
• Decision. A face counts only when it clears two thresholds together, an absolute score and a margin above the rest of the scene, which keeps false alarms down.
• Speed. Inference is 9.3 million multiply-accumulates per frame and takes about 13.8 ms on the DLA.

On 114 held-out test frames from the OV5640, the detector found 72 of 74 faces (97.3%) and raised no false alarms on the 40 frames without a face.

Audio: What Did They Say?

The audio half is a keyword spotter that listens for five commands: up, down, left, right and go, plus a sixth class for everything else. It also began as its own ECS-DoT reference design before being merged in and optimized for faster inference time on the DLA.

• Trigger. The keyword spotter only listens while the camera sees a face; with nobody in front of the device, the microphone stays off. While it’s listening, a sound clearly louder than the room’s background (three times its ambient level) starts a 1-second window. The window begins one 200 ms chunk early, so the start of the word is never clipped.
• Features. The CPU condenses that second of audio into a mel spectrogram, a picture of how its frequencies change over time, and shrinks it down for the network. It processes each chunk as soon as the DMA delivers it, so about three-quarters of the preprocessing is done before the word ends.
• Model. A small two-layer neural network, with 256 inputs, 64 hidden units and 6 outputs, and about 17,000 parameters. Like the face detector, it runs entirely on the DLA, using 8-bit integer math. Each output scores how likely the clip is to be one of the five commands or background sound. The highest-scoring class wins, but a word is reported only if the model is at least 70% confident.
• Speed. Inference is 16,768 multiply-accumulates per audio frame and takes 174 µs on the DLA.

Bringing It Together: The Camera Decides When to Listen

The two halves are joined by a simple rule: the microphone is open only while a face is in view. Vision acts as the gate, and keyword spotting is only performed when the ECS-DoT detects a face. Audio streams into two alternating 200 ms buffers: while one fills, the CPU works on the other, so recording never pauses, and most of the preprocessing is done before the word ends. The two AI models never run at the same time, but rather alternate between the two mentioned above.

Why let the camera decide when to listen? A microphone that opens only when someone is present never captures audio of an empty room, and the keyword spotter can’t be falsely triggered by background noise when nobody is there to speak. Privacy and reliability come from the architecture, not from a setting.

Figure 1. The Presence Gate and its Three States.

A single missed face check does not close the microphone; the chip keeps listening and checks again. The microphone only closes after two misses in a row.

The tricky part is that face checks keep happening while the microphone is open. A word spoken during a check is processed straight after, so it is heard a moment late but never lost.

The Numbers

All timing numbers below were measured on the board, using the chip’s own cycle counter.

Power figures cover the SoC’s core supply only, not the camera, microphone or the rest of the board. The energy figures are upper bounds, taken at peak current for the whole stage. Power is measured differentially from the inference power and subtracting the baseline power when no inference is running on the core voltage rail, giving us the power consumed by the DLA.

What’s Next

This reference design shows that vision and audio can run together on one ECS-DoT, with no cloud, no external memory and a core power budget of a few milliwatts. The architecture keeps low power in mind: one sensor (camera) decides when another should switch on (microphone). In a finished product, ECS-DoT sits alongside the host processor as an always-on AI co-processor, so the host can stay asleep until there is a person in view and a command to act on.

EMASS plans to develop a truly multimodal model that performs a single model combining both the audio and visual data for the same inference. This reference design is the first combining audio and vision, and it is just the start. With the ECS-DoT’s powerful deep learning accelerator, the possibilities are endless for ultra-low power intelligent edge solutions.

The reference design builds and flashes natively from the EMASS IDE on Windows, and a browser dashboard shows the live state: whether a face is seen, whether the microphone is open, and the last word heard.

We can already determine if and when sensitive equipment was dropped, if perishable goods were sufficiently refrigerated during transit, if your asset was in fact shipped via air instead of by sea or ground, or the exact XYZ location a package is stored within a warehouse down to the individual pallet. Now imagine we can greatly extend the battery of life of this system by selecting the right ultra-low power SoC and building our architecture carefully around it.

The sensors themselves are inexpensive and for the most part low power when used with the correct duty cycle and averaged over long periods of time. Radios are available for Bluetooth, 5G, or any number of low-power wide area networks (LP-WAN) and can also be considered low power over long periods of time when used only when needed. Until recently, the hard part was finding an AI-based edge processor that itself operated at ultra-low power while bringing more intelligence and reducing inference latency.

From tracking to understanding

The opportunity is to combine location, movement and environmental data with on-device intelligence so the system can understand what is happening to an asset—not simply where it is.

Ultra-Low Power AI at the Edge

This is where our ECS-DoT AI system-on-chip (SoC) comes in. ECS-DoT SoCs are ultra low-power and feature real-time AI inference engines to support always-on multimodal sensor fusion at the milli-Watt scale. There are many companies developing edge AI chips and solutions, but the ECS-DoT is designed to deliver exceptional performance while consuming dramatically less power than conventional approaches.

The ECS-DoT and its sensors sit in low power modes most of the time, until they are needed to determine if other system resources (additional sensors, higher-level processing, radio) need to be activated. The lowest power sensors that can make a decision make a decision, while “expensive” resources wake only when the context warrants.

Let’s take cold storage of produce for example. The ECS-DoT and temperature/gas sensors are considered higher power than a barometer or accelerometer, so we use these two sensors to send an interrupt to the ECS-DoT to wake up and measure temperature / ambient gases when we have detected an impact or anomalous pressure change. This allows us to keep relatively low duty cycle on the high current consumers and lower average system power.

Low Power, Fast Decisions

Not only does ECS-DoT management of system resources lead to low battery usage, but the extraordinary efficiency of our AI algorithms also translates into ultra-low wake-to-decision latency. This approach means we can keep the entire system in lower power states longer instead of staying awake and processing more data for longer.

AI capabilities allow users to go beyond simple motion thresholds, for example, to characterize specific activity such as impacts, tampering, or changes in environmental conditions like temperature and humidity. This leads to context-aware edge intelligence through multimodal sensor fusion and is instrumental in making event-based reporting possible.

Since a radio or apps processor is the hungriest component in the system, a smart system lean on the intelligence of the embedded controller and lower power sensors to only report events that are necessary.

Optimizing Battery Life and Contextual Awareness

Depending on variables that will change from user to user – such as the configuration of sensors and requirements for reporting – can increase the battery life of the system OR increase the contextual awareness of the asset tracker.

A real-world implementation developed with Bosch integrates ECS-DoT with an accelerometer, an inertial measurement unit (IMU, which combines an accelerometer and a gyroscope), a barometer, a magnetometer, and a gas meter.

A Tiered Approach to Sensor Monitoring

The inclusion of both a standalone accelerometer and an IMU may seem redundant at first, but there’s a good reason for it.

Standalone accelerometers are inexpensive, and they operate with ~100x less power than a more complex IMU for typical use cases. Our edge AI processor can rely on the standalone accelerometer to determine if an event is significant.

If not, the processor returns the entire system to deep sleep mode. If the data from the accelerometer indicates that more data would be useful, however, the processor then activates the IMU to get a richer data set, and more degrees of freedom for measurement.

The potential savings in power consumption derived from this tiered approach to waking and sensing easily justifies the use of both accelerometer and IMU.

Why the tiered approach matters

The potential savings in power consumption derived from waking and sensing only when necessary can justify the use of both a low-power standalone accelerometer and a richer IMU.

Flexible Sensor Configurations for Different Applications

This combination of sensors that EMASS and Bosch put together can be used for many different purposes, and can also be optimized for each use case by selecting different sensor configuration profiles and different models on the ECS-DoT.

One profile is better for monitoring fragile systems, another is more suitable for perishables, while a third is more appropriate for tracking assets within busy warehouses.

Of course, this initial combination with Bosch is just one possible configuration. Engineers can create sensor modules that best fit their specific applications.

Microphones and sensors that detect sound, light, flow, temperature, pressure, and other phenomena create endless opportunities for always on mW intelligent systems.

Beyond Tracking: Intelligent Asset Monitoring

Our ECS-DoT SoCs open new possibilities for intelligent asset monitoring, helping organizations move beyond simply knowing where their assets are to understand their condition, context, and movement in real time in the always on milliwatt domain.