esp-sr/docs/en/benchmark
2022-12-02 15:13:10 +08:00
..
README.rst docs:fix_bugs 2022-12-02 15:13:10 +08:00

Resource Consumption
=====================

:link_to_translation:`zh_CN:[中文]`

AFE
---

Resource Occupancy(ESP32)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

+-----------------+-----------------+-----------------+-----------------+
| algorithm Type  | RAM             | Average cpu     | Frame Length    |
|                 |                 | loading(compute |                 |
|                 |                 | with 2 cores)   |                 |
+=================+=================+=================+=================+
| AEC(HIGH_PERF)  | 114 KB          | 11%             | 32 ms           |
+-----------------+-----------------+-----------------+-----------------+
| NS              | 27 KB           | 5%              | 10 ms           |
+-----------------+-----------------+-----------------+-----------------+
| AFE Layer       | 73 KB           |                 |                 |
+-----------------+-----------------+-----------------+-----------------+

Resource Occupancy(ESP32S3)
~~~~~~~~~~~~~~~~~~~~~~~~~~~

+-----------------+-----------------+-----------------+-----------------+
| algorithm Type  | RAM             | Average cpu     | Frame Length    |
|                 |                 | loading(compute |                 |
|                 |                 | with 2 cores)   |                 |
+=================+=================+=================+=================+
| AEC(LOW_COST)   | 152.3 KB        | 8%              | 32 ms           |
+-----------------+-----------------+-----------------+-----------------+
| AEC(HIGH_PERF)  | 166 KB          | 11%             | 32 ms           |
+-----------------+-----------------+-----------------+-----------------+
| BSS(LOW_COST)   | 198.7 KB        | 6%              | 64 ms           |
+-----------------+-----------------+-----------------+-----------------+
| BSS(HIGH_PERF)  | 215.5 KB        | 7%              | 64 ms           |
+-----------------+-----------------+-----------------+-----------------+
| NS              | 27 KB           | 5%              | 10 ms           |
+-----------------+-----------------+-----------------+-----------------+
| MISO            | 56 KB           | 8%              | 16 ms           |
+-----------------+-----------------+-----------------+-----------------+
| AFE Layer       | 227 KB          |                 |                 |
+-----------------+-----------------+-----------------+-----------------+

WakeNet
-------

.. _resource-occupancyesp32-1:

Resource Occupancy(ESP32)
~~~~~~~~~~~~~~~~~~~~~~~~~

+-------------+-------------+-------------+-------------+-------------+
| Model Type  | Parameter   | RAM         | Average     | Frame       |
|             | Num         |             | Running     | Length      |
|             |             |             | Time per    |             |
|             |             |             | Frame       |             |
+=============+=============+=============+=============+=============+
| Quantised   | 41 K        | 15 KB       | 5.5 ms      | 30 ms       |
| WakeNet5    |             |             |             |             |
+-------------+-------------+-------------+-------------+-------------+
| Quantised   | 165 K       | 20 KB       | 10.5 ms     | 30 ms       |
| WakeNet5X2  |             |             |             |             |
+-------------+-------------+-------------+-------------+-------------+
| Quantised   | 371 K       | 24 KB       | 18 ms       | 30 ms       |
| WakeNet5X3  |             |             |             |             |
+-------------+-------------+-------------+-------------+-------------+

.. _resource-occupancyesp32s3-1:

Resource Occupancy(ESP32S3)
~~~~~~~~~~~~~~~~~~~~~~~~~~~

+----------------+-------+---------+----------------+--------------+
| Model Type     | RAM   | PSRAM   | Average        | Frame Length |
|                |       |         | Running Time   |              |
|                |       |         | per Frame      |              |
+================+=======+=========+================+==============+
| Quantised      | 50 KB | 1640 KB | 10.0 ms        | 32 ms        |
| WakeNet8 @ 2   |       |         |                |              |
| channel        |       |         |                |              |
+----------------+-------+---------+----------------+--------------+
| Quantised      | 16 KB | 324 KB  | 3.0 ms         | 32 ms        |
| WakeNet9 @ 2   |       |         |                |              |
| channel        |       |         |                |              |
+----------------+-------+---------+----------------+--------------+
| Quantised      | 20 KB | 347 KB  | 4.3 ms         | 32 ms        |
| WakeNet9 @ 3   |       |         |                |              |
| channel        |       |         |                |              |
+----------------+-------+---------+----------------+--------------+

Performance
~~~~~~~~~~~

+-------------+-------------+-------------+-------------+-------------+
| Distance    | Quiet       | Stationary  | Speech      | AEC         |
|             |             | Noise (SNR  | Noise (SNR  | I           |
|             |             | = 4 dB)     | = 4 dB)     | nterruption |
|             |             |             |             | (-10 dB)    |
+=============+=============+=============+=============+=============+
| 1 m         | 98%         | 96%         | 94%         | 96%         |
+-------------+-------------+-------------+-------------+-------------+
| 3 m         | 98%         | 96%         | 94%         | 94%         |
+-------------+-------------+-------------+-------------+-------------+

False triggering rate: 1 time in 12 hours

**Note**: We use the ESP32-S3-Korvo V4.0 development board and the WakeNet9(Alexa) model in our test.

MultiNet
--------

.. _resource-occupancyesp32-2:

Resource Occupancy(ESP32)
~~~~~~~~~~~~~~~~~~~~~~~~~~

+-------------+-------------+-------------+-------------+-------------+
| Model Type  | Internal    | PSRAM       | Average     | Frame       |
|             | RAM         |             | Running     | Length      |
|             |             |             | Time per    |             |
|             |             |             | Frame       |             |
+=============+=============+=============+=============+=============+
| MultiNet 2  | 13.3 KB     | 9KB         | 38 ms       | 30 ms       |
+-------------+-------------+-------------+-------------+-------------+

.. _resource-occupancyesp32s3-2:

Resource Occupancy(ESP32S3)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

+-------------+-------------+-------------+-------------+-------------+
| Model Type  | Internal    | PSRAM       | Average     | Frame       |
|             | RAM         |             | Running     | Length      |
|             |             |             | Time per    |             |
|             |             |             | Frame       |             |
+=============+=============+=============+=============+=============+
| MultiNet 4  | 16.8KB      | 1866 KB     | 18 ms       | 32 ms       |
+-------------+-------------+-------------+-------------+-------------+
| MultiNet 4  | 10.5 KB     | 1009 KB     | 11 ms       | 32 ms       |
| Q8          |             |             |             |             |
+-------------+-------------+-------------+-------------+-------------+
| MultiNet 5  | 16 KB       | 2310 KB     | 12 ms       | 32 ms       |
| Q8          |             |             |             |             |
+-------------+-------------+-------------+-------------+-------------+

Performance with AFE
~~~~~~~~~~~~~~~~~~~~

+-----------+-----------+-----------+-----------+-----------+
| Model     | Distance  | Quiet     | S         | Speech    |
| Type      |           |           | tationary | Noise     |
|           |           |           | Noise     | (SNR = 4  |
|           |           |           | (SNR = 4  | dB)       |
|           |           |           | dB)       |           |
+===========+===========+===========+===========+===========+
| MultiNet  | 3 m       | 98%       | 93%       | 92%       |
| 4         |           |           |           |           |
+-----------+-----------+-----------+-----------+-----------+
| MultiNet  | 3 m       | 94%       | 92%       | 91%       |
| 4 Q8      |           |           |           |           |
+-----------+-----------+-----------+-----------+-----------+