mirror of
https://github.com/espressif/esp-sr.git
synced 2025-09-15 15:28:44 +08:00
doc: Update flash_model,wakenet,multinet doc
This commit is contained in:
sxy
parent
09f3e84d92
commit
728335198d
@ -40,27 +40,14 @@ This option needs to be turned on. Users do not need to modify it. Please keep t
|
||||
|
||||
### 1.3 use wakenet
|
||||
|
||||
This option is turned on by default. When the user only uses `AEC` or `BSS`, etc., and does not need to run `WakeNet` or `MultiNet`, please turn off this option, which will reduce the size of the project firmware in some cases.
|
||||
This option is turned on by default. When the user only uses `AEC` or `BSS`, etc., and does not need to run `WakeNet` or `MultiNet`, please turn off this option, which will reduce the size of the project firmware.
|
||||
|
||||
- First Wake word
|
||||
|
||||
Select the first wake word. Please select the corresponding wake word in the options as needed.
|
||||
|
||||
ESP32 支持：
|
||||
|
||||
- WakeNet 5 (quantized with 16-bit)
|
||||
|
||||
ESP32S3 支持：
|
||||
|
||||
- WakeNet 7 (quantized with 16-bit)
|
||||
- WakeNet 7 (quantized with 8-bit)
|
||||
- WakeNet 8 (quantized with 16-bit)
|
||||
|
||||
- Second Wake wod
|
||||
|
||||
For second wake word, please select the corresponding wake-up words in the options as needed.
|
||||
|
||||
**Note: this option only supports ESP32S3, ESP32S3 supports the selection of up to two wake words and supports the user to switch in the code.**
|
||||
- Select wake words by menuconfig, `ESP Speech Recognition -> Select wake words`. The model name of wake word in parentheses is used to initialize wakenet handle.
|
||||
|
||||
- If you want to select multiple wake words, please select `Load Multiple Wake Words` ( **Note this option only supports ESP32S3**)
|
||||
|
||||
Then you can select multiple wake words at the same time
|
||||
|
||||
|
||||
For more details, please refer to [WakeNet](../wake_word_engine/README.md) .
|
||||
|
||||
@ -112,19 +99,16 @@ For more details, please refer to [MultiNet](../speech_command_recognition/READM
|
||||
|
||||
## 2. How to use
|
||||
|
||||
After the user completes the above configuration choices, please refer to esp-skainet to initialize and use the application layer. Here is an introduction to the code implementation of model data loading in the project.
|
||||
Here is an introduction to the code implementation of model data loading in the project. If you want get more detailes, please refer to esp-skainet examples.
|
||||
|
||||
### 2.1 Use ESP32
|
||||
### 2.1.1 ESP32
|
||||
|
||||
When the user uses ESP32, since it only supports loading the model data directly from the Flash, the model data in the code will automatically read the required data from the Flash according to the address.
|
||||
When the user uses ESP32, since it only supports loading the model data directly from the Flash, the model data in the code will automatically read the required data from the Flash according to the address.
|
||||
Now The ESP32S3 API is compatible with ESP32. You can refer to the ESP32S3 method to load and initialize the model.
|
||||
|
||||
### 2.2 Use ESP32S3
|
||||
### 2.1.2 ESP32S3
|
||||
|
||||
#### 2.2.1 Model data in SPIFFS
|
||||
|
||||
When the user configuration #1.2 model data storage location is `spiffs partition`, the user needs to:
|
||||
|
||||
- Write a partition table:
|
||||
- Step1: Write a partition table:
|
||||
|
||||
```
|
||||
model, data, spiffs, , SIZE,
|
||||
@ -134,46 +118,45 @@ When the user configuration #1.2 model data storage location is `spiffs partitio
|
||||
```
|
||||
Recommended model partition size: 500K
|
||||
```
|
||||
After completing the above configuration, the project will automatically generate `model.bin` after the project is compiled, and flash it to the spiffs partition.
|
||||
|
||||
- Initialize the spiffs partition
|
||||
|
||||
**Use the provided API**: User can use `srmodel_spiffs_init()` API to initialize spiffs.
|
||||
|
||||
**Write by yourself**: When users need to store other files in the spiffs partition at the same time, such as web pages, they can write the spiffs initialization function by themselves. Pay attention to the configuration of `esp_vfs_spiffs_conf`:
|
||||
|
||||
- base_path: The model storage `base_path` is `srmodel` and cannot be changed
|
||||
- partition_label: The partition label of the model is `model`, which needs to be consistent with the `Name` in the above partition table
|
||||
- Step2: Initialize the spiffs partition
|
||||
User can use `esp_srmodel_init()` API to initialize spiffs and return all loaded models.
|
||||
- base_path: The model storage `base_path` is `srmodel` and cannot be changed
|
||||
- partition_label: The partition label of the model is `model`, which needs to be consistent with the `Name` in the above partition table
|
||||
|
||||
After completing the above configuration, the project will automatically generate `model.bin` after the project is compiled, and flash it to the spiffs partition.
|
||||
**<font color=red>Note: After the user changes the model, be sure to run `idf.py clean` before compiling again.</font>**
|
||||
|
||||
**<font color=red>Note: After the user changes the model, be sure to run `idf.py clean` before compiling again.</font>**
|
||||
## 2.2 ESP32S3
|
||||
```
|
||||
//
|
||||
// step1: initialize spiffs and return models in spiffs
|
||||
//
|
||||
srmodel_list_t *models = esp_srmodel_init();
|
||||
|
||||
#### 2.2.1 Model data in SD Card
|
||||
//
|
||||
// step2: select the specific model by keywords
|
||||
//
|
||||
char *wn_name = esp_srmodel_filter(models, ESP_WN_PREFIX, NULL); // select wakenet model
|
||||
char *nm_name = esp_srmodel_filter(models, ESP_MN_PREFIX, NULL); // select multinet model
|
||||
char *alexa_wn_name = esp_srmodel_filter(models, ESP_WN_PREFIX, "alexa"); // select wakenet with "alexa" wake word.
|
||||
char *en_mn_name = esp_srmodel_filter(models, ESP_MN_PREFIX, ESP_MN_ENGLISH); // select english multinet model
|
||||
char *cn_mn_name = esp_srmodel_filter(models, ESP_MN_PREFIX, ESP_MN_CHINESE); // select english multinet model
|
||||
|
||||
// It also works if you use the model name directly in your code.
|
||||
char *my_wn_name = "wn9_hilexin"
|
||||
// we recommend you to check that it is loaded correctly
|
||||
if (!esp_srmodel_exists(models, my_wn_name))
|
||||
printf("%s can not be loaded correctly\n")
|
||||
|
||||
When the user configuration #1.2 model data storage location is `SD Card`, the user needs to:
|
||||
//
|
||||
// step3: initialize model
|
||||
//
|
||||
esp_wn_iface_t *wakenet = esp_wn_handle_from_name(wn_name);
|
||||
model_iface_data_t *wn_model_data = wakenet->create(wn_name, DET_MODE_2CH_90);
|
||||
|
||||
- Manually move model data
|
||||
esp_mn_iface_t *multinet = esp_mn_handle_from_name(mn_name);
|
||||
model_iface_data_t *mn_model_data = multinet->create(mn_name, 6000);
|
||||
|
||||
Move the model to SDCard. After the user completes the above configuration, you can compile first. After compiling, copy the files in the `ESP-SR_PATH/model/target/` directory to the root directory of the SD card.
|
||||
|
||||
- Custom model path
|
||||
```
|
||||
|
||||
If the user wants to place the model in the specified folder, he can modify the `get_model_base_path()` function by himself, located in `ESP-SR_PATH/model/model_path.c`.
|
||||
For example, if the specified folder is `espmodel` in the SD card directory, the function can be modified to:
|
||||
|
||||
```
|
||||
char *get_model_base_path(void)
|
||||
{
|
||||
#if defined CONFIG_MODEL_IN_SDCARD
|
||||
return "sdcard/espmodel";
|
||||
#elif defined CONFIG_MODEL_IN_SPIFFS
|
||||
return "srmodel";
|
||||
#else
|
||||
return NULL;
|
||||
#endif
|
||||
}
|
||||
```
|
||||
|
||||
- Initialize the SD card
|
||||
|
||||
The user needs to initialize the SD card to enable the system to record the SD card. The user can directly call `sd_card_mount("/sdcard")` to initialize the SD card that supports the development board if use esp-skainet. Otherwise, you need to write it yourself.
|
||||
@ -40,27 +40,15 @@ ESP32S3：
|
||||
|
||||
### 1.3 use wakenet
|
||||
|
||||
此选项默认打开，当用户只使用 AEC 或者 BSS 等，无须运行 WakeNet 或 MultiNet 时，请关闭次选项，将会在一些情况下减小工程固件的大小。
|
||||
|
||||
- First Wake word
|
||||
|
||||
首选唤醒词选择，请用户根据需要在选项中选择相应的唤醒词。
|
||||
|
||||
ESP32 支持：
|
||||
|
||||
- WakeNet 5 (quantized with 16-bit) 系列
|
||||
|
||||
ESP32S3 支持：
|
||||
|
||||
- WakeNet 7 (quantized with 16-bit) 系列
|
||||
- WakeNet 7 (quantized with 8-bit) 系列
|
||||
- WakeNet 8 (quantized with 16-bit) 系列
|
||||
|
||||
- Second Wake wod
|
||||
|
||||
备选唤醒词语，请用户根据需要在选项中选择相应的唤醒词
|
||||
|
||||
**注：该选项只支持 ESP32S3，即 ESP32S3 支持最多选择两个唤醒词，支持用户在代码中进行切换。**
|
||||
此选项默认打开，当用户只使用 AEC 或者 BSS 等，无须运行 WakeNet 或 MultiNet 时，请关闭次选项，将会减小工程固件的大小。
|
||||
- 根据menuconfig列表选择唤醒词模型，`ESP Speech Recognition -> Select wake words`. 括号中为唤醒词模型的名字，你需要在代码用名字切换，初始化wakenet.
|
||||
|
||||
- 如果想加载多个唤醒词，以便在代码中进行唤醒词的切换，首选选择'Load Multiple Wake Words'
|
||||
|
||||
然后按照列表选择多个唤醒词:
|
||||
|
||||
|
||||
**注：多唤醒词选项只支持 ESP32S3，具体根据客户硬件flash容量，选择合适数量的唤醒词。**
|
||||
|
||||
更多细节请参考 [WakeNet](../wake_word_engine/README.md) 。
|
||||
|
||||
@ -112,17 +100,17 @@ ESP32 芯片只支持中文命令词识别。ESP32S3 支持中文和英文命令
|
||||
## 2. 模型使用
|
||||
|
||||
当用户完成以上的配置选择后，应用层请参考 esp-skainet 进行初始化和使用。这里介绍一下模型数据加载在用户工程中的代码实现。
|
||||
也可以参考代码 [model_path.c](../../src/model_path.c)
|
||||
|
||||
### 2.1 使用 ESP32
|
||||
|
||||
当用户使用 ESP32 时，由于只支持从 Flash 中直接加载模型数据，因此代码中模型数据会自动按照地址从 Flash 中读取所需数据。
|
||||
为了和ESP32S3进行兼容，代码中模型的初始化方法是和ESP32S3相同的，可参考下面ESP32S3的模型加载API．
|
||||
|
||||
### 2.2 使用 ESP32S3
|
||||
|
||||
#### 2.2.1 模型数据存储在 SPIFFS
|
||||
|
||||
当用户配置 #1.2 模型数据存储位置是 `spiffs partition` 时，用户需要：
|
||||
|
||||
- 编写分区表：
|
||||
|
||||
```
|
||||
@ -134,19 +122,14 @@ ESP32 芯片只支持中文命令词识别。ESP32S3 支持中文和英文命令
|
||||
Recommended model partition size: 500K
|
||||
```
|
||||
- 初始化 spiffs 分区
|
||||
|
||||
**直接调用提供的 API**：用户可以直接调用 `srmodel_spiffs_init()` API 来初始化 spiffs。
|
||||
|
||||
**自行编写**：当用户需要在 spiffs 分区同时存放其他文件，如 web 网页时，可以自行编写 spiffs 初始化函数，需要注意 `esp_vfs_spiffs_conf`的配置：
|
||||
|
||||
- base_path：模型的存储 `base_path` 为 `srmodel`，不可更改
|
||||
- partition_label：模型的分区 label 为 `model`，需要和 上述分区表中的 `Name` 保持一致
|
||||
**调用提供的 API**：用户可以直接调用 `esp_srmodel_init()` API 来初始化 spiffs，并返回spiffs中的模型。
|
||||
- base_path：模型的存储 `base_path` 为 `srmodel`，不可更改
|
||||
- partition_label：模型的分区 label 为 `model`，需要和 上述分区表中的 `Name` 保持一致
|
||||
|
||||
完成上述配置后，模型会在工程编译完成后自动生成 `model.bin`，并在用户烧写时候烧写到 spiffs 分区。
|
||||
完成上述配置后，模型会在工程编译完成后自动生成 `model.bin`，并在用户调用`idf.py flash`时烧写到 spiffs 分区。
|
||||
|
||||
**<font color=red>注：当用户更改模型后，再次编译前请务必进行 `idf.py clean`</font>**
|
||||
|
||||
#### 2.2.1 模型存储在 SD Card
|
||||
#### 2.2.２ 模型存储在 SD Card
|
||||
|
||||
当用户配置 #1.2 模型数据存储位置是 `SD Card` 时，用户需要：
|
||||
|
||||
@ -175,4 +158,38 @@ ESP32 芯片只支持中文命令词识别。ESP32S3 支持中文和英文命令
|
||||
|
||||
用户需要初始化 SD 卡，来使系统能够记载 SD 卡，如果用户使用 esp-skainet，可以直接调用 `esp_sdcard_init("/sdcard", num);` 来初始化其支持开发板的 SD 卡。否则，需要自己编写。
|
||||
|
||||
完成以上操作后，便可以进行工程的烧录。
|
||||
完成以上操作后，便可以进行工程的烧录。
|
||||
|
||||
#### 2.2.３ 代码中模型初始化与使用
|
||||
```
|
||||
//
|
||||
// step1: initialize spiffs and return models in spiffs
|
||||
//
|
||||
srmodel_list_t *models = esp_srmodel_init();
|
||||
|
||||
//
|
||||
// step2: select the specific model by keywords
|
||||
//
|
||||
char *wn_name = esp_srmodel_filter(models, ESP_WN_PREFIX, NULL); // select wakenet model
|
||||
char *nm_name = esp_srmodel_filter(models, ESP_MN_PREFIX, NULL); // select multinet model
|
||||
char *alexa_wn_name = esp_srmodel_filter(models, ESP_WN_PREFIX, "alexa"); // select wakenet with "alexa" wake word.
|
||||
char *en_mn_name = esp_srmodel_filter(models, ESP_MN_PREFIX, ESP_MN_ENGLISH); // select english multinet model
|
||||
char *cn_mn_name = esp_srmodel_filter(models, ESP_MN_PREFIX, ESP_MN_CHINESE); // select english multinet model
|
||||
|
||||
// It also works if you use the model name directly in your code.
|
||||
char *my_wn_name = "wn9_hilexin"
|
||||
// we recommend you to check that it is loaded correctly
|
||||
if (!esp_srmodel_exists(models, my_wn_name))
|
||||
printf("%s can not be loaded correctly\n")
|
||||
|
||||
//
|
||||
// step3: initialize model
|
||||
//
|
||||
esp_wn_iface_t *wakenet = esp_wn_handle_from_name(wn_name);
|
||||
model_iface_data_t *wn_model_data = wakenet->create(wn_name, DET_MODE_2CH_90);
|
||||
|
||||
esp_mn_iface_t *multinet = esp_mn_handle_from_name(mn_name);
|
||||
model_iface_data_t *mn_model_data = multinet->create(mn_name, 6000);
|
||||
|
||||
```
|
||||
|
||||
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 43 KiB After Width: | Height: | Size: 29 KiB |
BIN
docs/img/wn_menu1.png
Normal file
BIN
docs/img/wn_menu1.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 22 KiB |
BIN
docs/img/wn_menu2.png
Normal file
BIN
docs/img/wn_menu2.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 20 KiB |
BIN
docs/img/wn_menu3.png
Normal file
BIN
docs/img/wn_menu3.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 21 KiB |
@ -34,10 +34,10 @@
|
||||
|
||||
### 2.2 Resource Occupancy(ESP32S3)
|
||||
|
||||
|Model Type|Parameter Num|RAM|Average Running Time per Frame| Frame Length|
|
||||
|Model Type|RAM|PSRAM|Average Running Time per Frame| Frame Length|
|
||||
|:---:|:---:|:---:|:---:|:---:|
|
||||
|Quantised WakeNet7_2CH|810 K|45 KB|10 ms|32 ms|
|
||||
|Quantised WakeNet8_2CH|821 K|50 KB|10 ms|32 ms|
|
||||
|Quantised WakeNet8 @ 2 channel|50 KB|1640 KB|10 ms|32 ms|
|
||||
|Quantised WakeNet9 @ 2 channel|16 KB|320 KB|4 ms|32 ms|
|
||||
|
||||
### 2.3 Performance
|
||||
|
||||
@ -62,13 +62,13 @@ False triggering rate: 1 time in 12 hours
|
||||
|
||||
|Model Type|Internal RAM|PSRAM|Average Running Time per Frame| Frame Length|
|
||||
|:---:|:---:|:---:|:---:|:---:|
|
||||
|MultiNet 4.5|16.8KB|1866 KB|18 ms|32 ms|
|
||||
|MultiNet 4.5 Q8|10.5 KB|1009 KB|11 ms|32 ms|
|
||||
|MultiNet 5 Q8|||||
|
||||
|MultiNet 4|16.8KB|1866 KB|18 ms|32 ms|
|
||||
|MultiNet 4 Q8|10.5 KB|1009 KB|11 ms|32 ms|
|
||||
|MultiNet 5 Q8|14 KB||12 ms|32 ms|
|
||||
|
||||
### 2.3 Performance with AFE
|
||||
|
||||
|Model Type|Distance| Quiet | Stationary Noise (SNR = 4 dB)| Speech Noise (SNR = 4 dB)|
|
||||
|:---:|:---:|:---:|:---:|:---:|:---:|
|
||||
|MultiNet 4.5|3 m|98%|93%|92%|
|
||||
|MultiNet 4.5 Q8|3 m|94%|92%|91%|
|
||||
|MultiNet 4|3 m|98%|93%|92%|
|
||||
|MultiNet 4 Q8|3 m|94%|92%|91%|
|
||||
|
||||
@ -105,69 +105,8 @@ Here we provide two methods of adding speech commands:
|
||||
|
||||
MultiNet supports online dynamic addition / deletion / modification of speech commands during operation, without changing models or adjusting parameters. For details, please refer to the example in ESP-Skainet.
|
||||
|
||||
Users only need to call the following APIs:
|
||||
Please refer to [esp_mn_speech_commands](../../src/esp_mn_speech_commands.h) for details of APIs:
|
||||
|
||||
```
|
||||
/**
|
||||
* @brief Initialze the Speech Commands link of MultiNet
|
||||
*
|
||||
* @return
|
||||
* - ESP_OK Success
|
||||
* - ESP_ERR_NO_MEM No memory
|
||||
* - ESP_ERR_INVALID_STATE The Speech Commands link has been initialized
|
||||
*/
|
||||
esp_err_t esp_mn_commands_init(void);
|
||||
|
||||
/**
|
||||
* @brief Add one speech commands with phoneme and command ID
|
||||
*
|
||||
* @param command_id The command ID
|
||||
*
|
||||
* @param phoneme_string The phoneme string of the speech commands
|
||||
*
|
||||
* @return
|
||||
* - ESP_OK Success
|
||||
* - ESP_ERR_INVALID_STATE Fail
|
||||
*/
|
||||
esp_err_t esp_mn_commands_add(int command_id, char *phoneme_string);
|
||||
|
||||
/**
|
||||
* @brief Modify one speech commands with new phoneme
|
||||
*
|
||||
* @param old_phoneme_string The old phoneme string of the speech commands
|
||||
*
|
||||
* @param new_phoneme_string The new phoneme string of the speech commands
|
||||
*
|
||||
* @return
|
||||
* - ESP_OK Success
|
||||
* - ESP_ERR_INVALID_STATE Fail
|
||||
*/
|
||||
esp_err_t esp_mn_commands_modify(char *old_phoneme_string, char *new_phoneme_string);
|
||||
|
||||
/**
|
||||
* @brief Remove one speech commands by phoneme
|
||||
*
|
||||
* @param phoneme_string The phoneme string of the speech commands
|
||||
*
|
||||
* @return
|
||||
* - ESP_OK Success
|
||||
* - ESP_ERR_INVALID_STATE Fail
|
||||
*/
|
||||
esp_err_t esp_mn_commands_remove(char *phoneme_string);
|
||||
|
||||
/**
|
||||
* @brief Update the speech commands of MultiNet, must be used after [add/remove/modify] the speech commands
|
||||
*
|
||||
* @param multinet The multinet handle
|
||||
*
|
||||
* @param model_data The model object to query
|
||||
*
|
||||
* @return
|
||||
* - ESP_OK Success
|
||||
* - ESP_ERR_INVALID_STATE Fail
|
||||
*/
|
||||
esp_err_t esp_mn_commands_update(const esp_mn_iface_t *multinet, const model_iface_data_t *model_data);
|
||||
```
|
||||
|
||||
## 4. Run speech commands recognition
|
||||
|
||||
@ -177,26 +116,8 @@ Speech commands recognition needs to be run together with the audio front-end (A
|
||||
|
||||
### 4.1 MultiNet Initialization
|
||||
|
||||
Before using MultiNet, you need to define the following variables:
|
||||
- Initialize multinet model
|
||||
|
||||
- Model version
|
||||
|
||||
Users need to declare the following model versions in the code. Users can directly use this in the following ways without changing it.
|
||||
|
||||
```
|
||||
const esp_mn_iface_t *multinet = &MULTINET_MODEL;
|
||||
```
|
||||
|
||||
- Model handle
|
||||
|
||||
The users needs to use the `create` interface to generate the model handle `model_data` for subsequent operations.
|
||||
|
||||
```
|
||||
model_iface_data_t *model_data = multinet->create(&MULTINET_COEFF, time_out_time_ms);
|
||||
```
|
||||
|
||||
- MULTINET_COEFF: Model parameters can be filled in directly without modifying
|
||||
- time_out_time_ms: The waiting exit time when the speech commands cannot be detected by MultiNet. The unit is `ms`. it supports customization. The recommended range is [5000, 10000]
|
||||
|
||||
- Set speech commands
|
||||
|
||||
@ -276,22 +197,5 @@ Exit the speech recognition when the return status is `ESP_MN_STATE_TIMEOUT`, it
|
||||
|
||||
## 5. Other configurations
|
||||
|
||||
### 5.1 Threshold setting
|
||||
|
||||
MultiNet supports set or get the threshold of each speech command, which can help users optimize recognition performance.
|
||||
|
||||
- Get the threshold of one speech commands
|
||||
|
||||
```
|
||||
multinet->get_command_det_threshold(model_data, phrase_id);
|
||||
```
|
||||
|
||||
- Set the threshold of one speech commands
|
||||
|
||||
When setting the threshold, users are advised to get the threshold first and increase or decrease it appropriately on the basis of the original threshold.
|
||||
|
||||
The threshold range is (0, 1).
|
||||
|
||||
```
|
||||
multinet->set_command_det_threshold(model_data, phrase_id, threshold);
|
||||
```
|
||||
### 5.1 Threshold setting
|
||||
This function is still under development.
|
||||
@ -110,69 +110,8 @@ MultiNet 支持多种且灵活的命令词设置方式，用户无论通过那
|
||||
|
||||
MultiNet 支持在运行过程中在线动态添加/删除/修改命令词，该过程无须更换模型和调整参数。具体可以参考 ESP-Skainet 中 example。
|
||||
|
||||
只需用户调用以下 API 即可：
|
||||
具体API说明请参考 [esp_mn_speech_commands](../../src/esp_mn_speech_commands.h)：
|
||||
|
||||
```
|
||||
/**
|
||||
* @brief Initialze the Speech Commands link of MultiNet
|
||||
*
|
||||
* @return
|
||||
* - ESP_OK Success
|
||||
* - ESP_ERR_NO_MEM No memory
|
||||
* - ESP_ERR_INVALID_STATE The Speech Commands link has been initialized
|
||||
*/
|
||||
esp_err_t esp_mn_commands_init(void);
|
||||
|
||||
/**
|
||||
* @brief Add one speech commands with phoneme and command ID
|
||||
*
|
||||
* @param command_id The command ID
|
||||
*
|
||||
* @param phoneme_string The phoneme string of the speech commands
|
||||
*
|
||||
* @return
|
||||
* - ESP_OK Success
|
||||
* - ESP_ERR_INVALID_STATE Fail
|
||||
*/
|
||||
esp_err_t esp_mn_commands_add(int command_id, char *phoneme_string);
|
||||
|
||||
/**
|
||||
* @brief Modify one speech commands with new phoneme
|
||||
*
|
||||
* @param old_phoneme_string The old phoneme string of the speech commands
|
||||
*
|
||||
* @param new_phoneme_string The new phoneme string of the speech commands
|
||||
*
|
||||
* @return
|
||||
* - ESP_OK Success
|
||||
* - ESP_ERR_INVALID_STATE Fail
|
||||
*/
|
||||
esp_err_t esp_mn_commands_modify(char *old_phoneme_string, char *new_phoneme_string);
|
||||
|
||||
/**
|
||||
* @brief Remove one speech commands by phoneme
|
||||
*
|
||||
* @param phoneme_string The phoneme string of the speech commands
|
||||
*
|
||||
* @return
|
||||
* - ESP_OK Success
|
||||
* - ESP_ERR_INVALID_STATE Fail
|
||||
*/
|
||||
esp_err_t esp_mn_commands_remove(char *phoneme_string);
|
||||
|
||||
/**
|
||||
* @brief Update the speech commands of MultiNet, must be used after [add/remove/modify] the speech commands
|
||||
*
|
||||
* @param multinet The multinet handle
|
||||
*
|
||||
* @param model_data The model object to query
|
||||
*
|
||||
* @return
|
||||
* - ESP_OK Success
|
||||
* - ESP_ERR_INVALID_STATE Fail
|
||||
*/
|
||||
esp_err_t esp_mn_commands_update(const esp_mn_iface_t *multinet, const model_iface_data_t *model_data);
|
||||
```
|
||||
|
||||
## 4. 运行命令词识别
|
||||
|
||||
@ -184,30 +123,10 @@ MultiNet 支持多种且灵活的命令词设置方式，用户无论通过那
|
||||
|
||||
### 4.1 MultiNet 初始化
|
||||
|
||||
在使用命令词识别模型前首先需要定义以下变量：
|
||||
|
||||
- 模型版本声明
|
||||
|
||||
用户需要在代码中声明以下模型版本，用户直接按以下方式使用，无须更改。
|
||||
|
||||
```
|
||||
const esp_mn_iface_t *multinet = &MULTINET_MODEL;
|
||||
```
|
||||
|
||||
- 生成模型句柄
|
||||
|
||||
用户需要使用 `create` 接口生成模型句柄`model_data`，以供后续操作。
|
||||
|
||||
```
|
||||
model_iface_data_t *model_data = multinet->create(&MULTINET_COEFF, time_out_time_ms);
|
||||
```
|
||||
|
||||
- MULTINET_COEFF： 模型参数，用户无须更改，直接填入
|
||||
- time_out_time_ms：当 MultiNet 检测不到命令词时的等待退出时间, 单位为 `ms`，支持自定义，建议范围为 [5000, 10000]
|
||||
|
||||
- 模型加载与初始化
|
||||
请参考[flash_model](../flash_model/README_CN.md)
|
||||
|
||||
- 设置命令词
|
||||
|
||||
请参考上文 #3。
|
||||
|
||||
### 4.2 MultiNet 运行
|
||||
@ -289,19 +208,4 @@ MultiNet 支持多种且灵活的命令词设置方式，用户无论通过那
|
||||
|
||||
### 5.1 阈值设置
|
||||
|
||||
MultiNet 支持对每个命令词的阈值进行设置或者查看，可以帮助用户更好的进行识别调优。
|
||||
|
||||
- 获取某个命令词的阈值
|
||||
|
||||
```
|
||||
multinet->get_command_det_threshold(model_data, phrase_id);
|
||||
```
|
||||
|
||||
- 设置某个命令词的阈值
|
||||
|
||||
用户在设置阈值的时候，建议先获取其阈值，在原本阈值基础上进行合适的增减。
|
||||
`threshold` 范围为 (0, 1)。
|
||||
|
||||
```
|
||||
multinet->set_command_det_threshold(model_data, phrase_id, threshold);
|
||||
```
|
||||
该功能仍在开发中．
|
||||
|
||||
@ -1,24 +1,24 @@
|
||||
# WakeNet
|
||||
# wakeNet
|
||||
|
||||
WakeNet, which is a wake word engine built upon neural network, is specially designed for low-power embedded MCUs. Now, the WakeNet model supports up to 5 wake words.
|
||||
wakeNet, which is a wake word engine built upon neural network, is specially designed for low-power embedded MCUs. Now, the wakeNet model supports up to 5 wake words.
|
||||
|
||||
## Overview
|
||||
|
||||
Please see the flow diagram of WakeNet below:
|
||||
Please see the flow diagram of wakeNet below:
|
||||
|
||||
<center>
|
||||
<img src="../img/wakenet_workflow.png" width = "800" />
|
||||
<img src="../img/wakeNet_workflow.png" width = "800" />
|
||||
</center>
|
||||
|
||||
- Speech Feature:
|
||||
The WakeNet uses [MFCC](https://en.wikipedia.org/wiki/Mel-frequency_cepstrum) to obtain the features of the input audio clip (16 KHz, 16 bit, single track). The window width and step width of each frame of the audio clip are both 30 ms.
|
||||
The wakeNet uses [MFCC](https://en.wikipedia.org/wiki/Mel-frequency_cepstrum) to obtain the features of the input audio clip (16 KHz, 16 bit, single track). The window width and step width of each frame of the audio clip are both 30 ms.
|
||||
|
||||
- Neural Network:
|
||||
Now, the neural network structure has been updated to the sixth edition, among which,
|
||||
- WakeNet1 and WakeNet2 had been out of use.
|
||||
- WakeNet3 and WakeNet4 had been out of use.
|
||||
- WakeNet5(WakeNet5X2,WakeNetX3), WakeNet7, WakeNet8 are built upon the [Dilated Convolution](https://arxiv.org/pdf/1609.03499.pdf) structure.
|
||||
Note that，The network structure of WakeNet5,WakeNet5X2 and WakeNet5X3 is same, but the parameter of WakeNetX2 and WakeNetX3 is more than WakeNet5. Please refer to [Resource Occupancy](#performance-test) for details.
|
||||
- wakeNet1,wakeNet2,wakeNet3,wakeNet4,wakeNet6,wakeNet7 had been out of use.
|
||||
- wakeNet5 only support ESP32 chip.
|
||||
- wakeNet8,wakeNet9 only support ESP32S3 chip, which are built upon the [Dilated Convolution](https://arxiv.org/pdf/1609.03499.pdf) structure.
|
||||
Note that，The network structure of wakeNet5,wakeNet5X2 and wakeNet5X3 is same, but the parameter of wakeNetX2 and wakeNetX3 is more than wakeNet5. Please refer to [Resource Occupancy](#performance-test) for details.
|
||||
|
||||
|
||||
- Keyword Triggering Method
|
||||
@ -26,22 +26,22 @@ Please see the flow diagram of WakeNet below:
|
||||
|
||||
The following table shows the model support of Espressif SoCs:
|
||||
|
||||
|
||||
|
||||
|
||||
## Use WakeNet
|
||||
## Use wakeNet
|
||||
|
||||
- How to select the WakeNet model
|
||||
- How to select the wakeNet model
|
||||
|
||||
Please refer to [Flash model 介绍](../flash_model/README.md).
|
||||
|
||||
- How to run WakeNet
|
||||
- How to run wakeNet
|
||||
|
||||
WakeNet is currently included in the [AFE](../audio_front_end/README.md), which is running by default, and returns the detect results through the AFE fetch interface.
|
||||
wakeNet is currently included in the [AFE](../audio_front_end/README.md), which is running by default, and returns the detect results through the AFE fetch interface.
|
||||
|
||||
If users wants to close WakeNet, please use:
|
||||
If users wants to close wakeNet, please use:
|
||||
|
||||
```
|
||||
afe_config.wakenet_init = False.
|
||||
afe_config.wakeNet_init = False.
|
||||
```
|
||||
|
||||
## Performance Test
|
||||
|
||||
@ -15,9 +15,9 @@ WakeNet的流程图如下：
|
||||
|
||||
- neural network：
|
||||
神经网络结构已经更新到第6版，其中：
|
||||
- wakeNet1和wakeNet2已经停止使用。
|
||||
- wakeNet3和wakeNet4基于[CRNN](https://arxiv.org/abs/1703.05390)结构。
|
||||
- WakeNet5(WakeNet5X2,WakeNetX3) 和 WakeNet7 和 WakeNet8 基于 the [Dilated Convolution](https://arxiv.org/pdf/1609.03499.pdf) 结构。
|
||||
- wakeNet1,wakeNet2,wakeNet3,wakeNet4已经停止使用。
|
||||
- wakeNet5应用于ESP32芯片。
|
||||
- wakeNet8和wakeNet9应用于ESP32S3芯片，模型基于 [Dilated Convolution](https://arxiv.org/pdf/1609.03499.pdf) 结构。
|
||||
|
||||
注意，WakeNet5,WakeNet5X2 和 WakeNet5X3 的网络结构一致，但是 WakeNet5X2 和 WakeNet5X3 的参数比 WakeNet5 要多。请参考 [性能测试](#性能测试) 来获取更多细节。
|
||||
|
||||
|
||||
Loading…
Reference in New Issue
Block a user