mirror of
https://github.com/modelscope/FunASR
synced 2025-09-15 14:48:36 +08:00
702ec03ad8
* add hotword for deploy_tools * Support wfst decoder and contextual biasing (#1039) * Support wfst decoder and contextual biasing * Turn on fstbin compilation --------- Co-authored-by: gongbo.gb <gongbo.gb@alibaba-inc.com> * mv funasr/runtime runtime * Fix crash caused by OOV in hotwords list * funasr infer * funasr infer * funasr infer * funasr infer * funasr infer * fix some bugs about fst hotword; support wfst for websocket server and clients; mv runtime out of funasr; modify relative docs * del onnxruntime/include/gflags * update tensor.h * update run_server.sh * update deploy tools * update deploy tools * update websocket-server * update funasr-wss-server * Remove self loop propagation * Update websocket_protocol_zh.md * Update websocket_protocol_zh.md * update hotword protocol * author zhaomingwork: change hotwords for h5 and java * update hotword protocol * catch exception for json_fst_hws * update hotword on message * update onnx benchmark for ngram&hotword * update docs * update funasr-wss-serve * add NONE for LM_DIR * update docs * update run_server.sh * add whats-new * modify whats-new * update whats-new * update whats-new * Support decoder option for beam searching * update benchmark_onnx_cpp * Support decoder option for websocket * fix bug of CompileHotwordEmbedding * update html client * update docs --------- Co-authored-by: gongbo.gb <35997837+aibulamusi@users.noreply.github.com> Co-authored-by: gongbo.gb <gongbo.gb@alibaba-inc.com> Co-authored-by: 游雁 <zhifu.gzf@alibaba-inc.com>
35 lines
1.2 KiB
Bash
Executable File
35 lines
1.2 KiB
Bash
Executable File
#!/bin/bash
|
|
#
|
|
|
|
if [ -f path.sh ]; then . path.sh; fi
|
|
|
|
lm_dir=$1
|
|
tgt_lang=$2
|
|
|
|
arpa_lm=${lm_dir}/lm.arpa
|
|
[ ! -f $arpa_lm ] && echo No such file $arpa_lm && exit 1;
|
|
|
|
# Compose the language model to FST
|
|
cat "$arpa_lm" | \
|
|
grep -v '<s> <s>' | \
|
|
grep -v '</s> <s>' | \
|
|
grep -v '</s> </s>' | \
|
|
grep -v -i '<unk>' | \
|
|
arpa2fst --read-symbol-table=$tgt_lang/words.txt --keep-symbols=true - | fstprint | \
|
|
fst/eps2disambig.pl | fst/s2eps.pl | fstcompile --isymbols=$tgt_lang/words.txt \
|
|
--osymbols=$tgt_lang/words.txt --keep_isymbols=false --keep_osymbols=false | \
|
|
fstrmepsilon | fstarcsort --sort_type=ilabel > $tgt_lang/G.fst
|
|
|
|
|
|
echo "Checking how stochastic G is (the first of these numbers should be small):"
|
|
fstisstochastic $tgt_lang/G.fst
|
|
|
|
# Compose the token, lexicon and language-model FST into the final decoding graph
|
|
fsttablecompose $tgt_lang/L.fst $tgt_lang/G.fst | fstdeterminizestar --use-log=true | \
|
|
fstminimizeencoded | fstarcsort --sort_type=ilabel > $tgt_lang/LG.fst || exit 1;
|
|
fsttablecompose $tgt_lang/T.fst $tgt_lang/LG.fst > $tgt_lang/TLG.fst || exit 1;
|
|
|
|
echo "Composing decoding graph TLG.fst succeeded"
|
|
rm -r $tgt_lang/LG.fst # We don't need to keep this intermediate FST
|
|
|