speech_recognition/FunASR
1
0
Fork 0
mirror of https://github.com/modelscope/FunASR synced 2025-09-15 14:48:36 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity

Merge branch 'main' of github.com:alibaba-damo-academy/FunASR

Browse Source 
add
This commit is contained in:
游雁 2023-11-14 15:16:30 +08:00
commit aff445e78c
57 changed files with 2061 additions and 304 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
runtime/deploy_tools/funasr-runtime-deploy-offline-cpu-en.sh
View File

@ -323,7 +323,7 @@ initConfiguration(){
$DEFAULT_FUNASR_CONFIG_DIR=$DEFAULT_FUNASR_CONFIG_DIR_BAK
fi
fi
DEFAULT_FUNASR_CONFIG_DIR=${DEFAULT_FUNASR_CONFIG_DIR}/.funasr
DEFAULT_FUNASR_CONFIG_DIR=${DEFAULT_FUNASR_CONFIG_DIR}/.funasr_en
fi
if [ ! -z "$DEFAULT_FUNASR_CONFIG_DIR" ]; then

2
runtime/deploy_tools/funasr-runtime-deploy-offline-cpu-zh.sh
View File

@ -324,7 +324,7 @@ initConfiguration(){
$DEFAULT_FUNASR_CONFIG_DIR=$DEFAULT_FUNASR_CONFIG_DIR_BAK
fi
fi
DEFAULT_FUNASR_CONFIG_DIR=${DEFAULT_FUNASR_CONFIG_DIR}/.funasr
DEFAULT_FUNASR_CONFIG_DIR=${DEFAULT_FUNASR_CONFIG_DIR}/.funasr_offline
fi
if [ ! -z "$DEFAULT_FUNASR_CONFIG_DIR" ]; then

8
runtime/docs/SDK_advanced_guide_offline.md
View File

@ -83,13 +83,13 @@ nohup bash run_server.sh \
Introduction to run_server.sh parameters:
```text
--download-model-dir: Model download address, download models from Modelscope by setting the model ID.
--model-dir: Modelscope model ID.
--model-dir: modelscope model ID or local model path.
--quantize: True for quantized ASR model, False for non-quantized ASR model. Default is True.
--vad-dir: Modelscope model ID.
--vad-dir: modelscope model ID or local model path.
--vad-quant: True for quantized VAD model, False for non-quantized VAD model. Default is True.
--punc-dir: Modelscope model ID.
--punc-dir: modelscope model ID or local model path.
--punc-quant: True for quantized PUNC model, False for non-quantized PUNC model. Default is True.
--itn-dir modelscope model ID
--itn-dir modelscope model ID or local model path.
--port: Port number that the server listens on. Default is 10095.
--decoder-thread-num: Number of inference threads that the server starts. Default is 8.
--io-thread-num: Number of IO threads that the server starts. Default is 1.

8
runtime/docs/SDK_advanced_guide_offline_en.md
View File

@ -65,13 +65,13 @@ nohup bash run_server.sh \
Introduction to run_server.sh parameters:
```text
--download-model-dir: Model download address, download models from Modelscope by setting the model ID.
--model-dir: Modelscope model ID.
--model-dir: modelscope model ID or local model path.
--quantize: True for quantized ASR model, False for non-quantized ASR model. Default is True.
--vad-dir: Modelscope model ID.
--vad-dir: modelscope model ID or local model path.
--vad-quant: True for quantized VAD model, False for non-quantized VAD model. Default is True.
--punc-dir: Modelscope model ID.
--punc-dir: modelscope model ID or local model path.
--punc-quant: True for quantized PUNC model, False for non-quantized PUNC model. Default is True.
--itn-dir modelscope model ID
--itn-dir modelscope model ID or local model path.
--port: Port number that the server listens on. Default is 10095.
--decoder-thread-num: Number of inference threads that the server starts. Default is 8.
--io-thread-num: Number of IO threads that the server starts. Default is 1.

8
runtime/docs/SDK_advanced_guide_offline_en_zh.md
View File

@ -150,13 +150,13 @@ nohup bash run_server.sh \
**run_server.sh命令参数介绍**
```text
--download-model-dir 模型下载地址，通过设置model ID从Modelscope下载模型
--model-dir modelscope model ID
--model-dir modelscope model ID 或者 本地模型路径
--quantize True为量化ASR模型，False为非量化ASR模型，默认是True
--vad-dir modelscope model ID
--vad-dir modelscope model ID 或者 本地模型路径
--vad-quant True为量化VAD模型，False为非量化VAD模型，默认是True
--punc-dir modelscope model ID
--punc-dir modelscope model ID 或者 本地模型路径
--punc-quant True为量化PUNC模型，False为非量化PUNC模型，默认是True
--itn-dir modelscope model ID
--itn-dir modelscope model ID 或者 本地模型路径
--port 服务端监听的端口号，默认为 10095
--decoder-thread-num 服务端启动的推理线程数，默认为 8
--io-thread-num 服务端启动的IO线程数，默认为 1

10
runtime/docs/SDK_advanced_guide_offline_zh.md
View File

@ -164,14 +164,14 @@ nohup bash run_server.sh \
**run_server.sh命令参数介绍**
```text
--download-model-dir 模型下载地址，通过设置model ID从Modelscope下载模型
--model-dir modelscope model ID
--model-dir modelscope model ID 或者 本地模型路径
--quantize True为量化ASR模型，False为非量化ASR模型，默认是True
--vad-dir modelscope model ID
--vad-dir modelscope model ID 或者 本地模型路径
--vad-quant True为量化VAD模型，False为非量化VAD模型，默认是True
--punc-dir modelscope model ID
--punc-dir modelscope model ID 或者 本地模型路径
--punc-quant True为量化PUNC模型，False为非量化PUNC模型，默认是True
--lm-dir modelscope model ID
--itn-dir modelscope model ID
--lm-dir modelscope model ID 或者 本地模型路径
--itn-dir modelscope model ID 或者 本地模型路径
--port 服务端监听的端口号，默认为 10095
--decoder-thread-num 服务端启动的推理线程数，默认为 8
--io-thread-num 服务端启动的IO线程数，默认为 1

8
runtime/docs/SDK_advanced_guide_online.md
View File

@ -100,14 +100,14 @@ nohup bash run_server_2pass.sh \
### More details about the script run_server_2pass.sh:
```text
--download-model-dir: Model download address, download models from Modelscope by setting the model ID.
--model-dir: Modelscope model ID.
--model-dir: modelscope model ID or local model path.
--online-model-dir modelscope model ID
--quantize: True for quantized ASR model, False for non-quantized ASR model. Default is True.
--vad-dir: Modelscope model ID.
--vad-dir: modelscope model ID or local model path.
--vad-quant: True for quantized VAD model, False for non-quantized VAD model. Default is True.
--punc-dir: Modelscope model ID.
--punc-dir: modelscope model ID or local model path.
--punc-quant: True for quantized PUNC model, False for non-quantized PUNC model. Default is True.
--itn-dir modelscope model ID
--itn-dir modelscope model ID or local model path.
--port: Port number that the server listens on. Default is 10095.
--decoder-thread-num: Number of inference threads that the server starts. Default is 8.
--io-thread-num: Number of IO threads that the server starts. Default is 1.

10
runtime/docs/SDK_advanced_guide_online_zh.md
View File

@ -108,14 +108,14 @@ nohup bash run_server_2pass.sh \
**run_server_2pass.sh命令参数介绍**
```text
--download-model-dir 模型下载地址，通过设置model ID从Modelscope下载模型
--model-dir modelscope model ID
--online-model-dir modelscope model ID
--model-dir modelscope model ID 或者 本地模型路径
--online-model-dir modelscope model ID 或者 本地模型路径
--quantize True为量化ASR模型，False为非量化ASR模型，默认是True
--vad-dir modelscope model ID
--vad-dir modelscope model ID 或者 本地模型路径
--vad-quant True为量化VAD模型，False为非量化VAD模型，默认是True
--punc-dir modelscope model ID
--punc-dir modelscope model ID 或者 本地模型路径
--punc-quant True为量化PUNC模型，False为非量化PUNC模型，默认是True
--itn-dir modelscope model ID
--itn-dir modelscope model ID 或者 本地模型路径
--port 服务端监听的端口号，默认为 10095
--decoder-thread-num 服务端启动的推理线程数，默认为 8
--io-thread-num 服务端启动的IO线程数，默认为 1

2
runtime/html5/static/main.js
View File

@ -315,7 +315,7 @@ function handleWithTimestamp(tmptext,tmptime)
{
return tmptext;
}
tmptext=tmptext.replace(/。/g, ","); // in case there are a lot of "。"
tmptext=tmptext.replace(/。|？|，|、|\?|\./g, ","); // replace all punc for parse
var words=tmptext.split(",");
var jsontime=JSON.parse(tmptime); //JSON.parse(tmptime.replace(/\]\]\[\[/g, "],[")); // in case there are a lot segments by VAD
var char_index=0;

11
runtime/onnxruntime/CMakeLists.txt
View File

@ -20,11 +20,12 @@ endif()
# for onnxruntime
IF(WIN32)
if(CMAKE_CL_64)
link_directories(${ONNXRUNTIME_DIR}\\lib)
else()
add_definitions(-D_WIN_X86)
endif()
file(REMOVE ${PROJECT_SOURCE_DIR}/third_party/glog/src/config.h
${PROJECT_SOURCE_DIR}/third_party/glog/src/glog/export.h
${PROJECT_SOURCE_DIR}/third_party/glog/src/glog/logging.h
${PROJECT_SOURCE_DIR}/third_party/glog/src/glog/raw_logging.h
${PROJECT_SOURCE_DIR}/third_party/glog/src/glog/stl_logging.h
${PROJECT_SOURCE_DIR}/third_party/glog/src/glog/vlog_is_on.h)
ELSE()
link_directories(${ONNXRUNTIME_DIR}/lib)
link_directories(${FFMPEG_DIR}/lib)

29
runtime/onnxruntime/bin/CMakeLists.txt
View File

@ -1,33 +1,42 @@
include_directories(${CMAKE_SOURCE_DIR}/include)
add_executable(funasr-onnx-offline "funasr-onnx-offline.cpp")
if(WIN32)
add_compile_options("$<$<CXX_COMPILER_ID:MSVC>:/execution-charset:utf-8>")
add_compile_options("$<$<CXX_COMPILER_ID:MSVC>:/source-charset:utf-8>")
include_directories(${ONNXRUNTIME_DIR}/include)
include_directories(${FFMPEG_DIR}/include)
include_directories(${PROJECT_SOURCE_DIR}/third_party)
SET(RELATION_SOURCE "../src/audio.cpp" "../src/resample.cpp" "../src/util.cpp" "../src/alignedmem.cpp" "../src/encode_converter.cpp")
endif()
add_executable(funasr-onnx-offline "funasr-onnx-offline.cpp" ${RELATION_SOURCE})
target_link_libraries(funasr-onnx-offline PUBLIC funasr)
add_executable(funasr-onnx-offline-vad "funasr-onnx-offline-vad.cpp")
add_executable(funasr-onnx-offline-vad "funasr-onnx-offline-vad.cpp" ${RELATION_SOURCE})
target_link_libraries(funasr-onnx-offline-vad PUBLIC funasr)
add_executable(funasr-onnx-online-vad "funasr-onnx-online-vad.cpp")
add_executable(funasr-onnx-online-vad "funasr-onnx-online-vad.cpp" ${RELATION_SOURCE})
target_link_libraries(funasr-onnx-online-vad PUBLIC funasr)
add_executable(funasr-onnx-online-asr "funasr-onnx-online-asr.cpp")
add_executable(funasr-onnx-online-asr "funasr-onnx-online-asr.cpp" ${RELATION_SOURCE})
target_link_libraries(funasr-onnx-online-asr PUBLIC funasr)
add_executable(funasr-onnx-offline-punc "funasr-onnx-offline-punc.cpp")
add_executable(funasr-onnx-offline-punc "funasr-onnx-offline-punc.cpp" ${RELATION_SOURCE})
target_link_libraries(funasr-onnx-offline-punc PUBLIC funasr)
add_executable(funasr-onnx-online-punc "funasr-onnx-online-punc.cpp")
add_executable(funasr-onnx-online-punc "funasr-onnx-online-punc.cpp" ${RELATION_SOURCE})
target_link_libraries(funasr-onnx-online-punc PUBLIC funasr)
add_executable(funasr-onnx-offline-rtf "funasr-onnx-offline-rtf.cpp")
add_executable(funasr-onnx-offline-rtf "funasr-onnx-offline-rtf.cpp" ${RELATION_SOURCE})
target_link_libraries(funasr-onnx-offline-rtf PUBLIC funasr)
add_executable(funasr-onnx-2pass "funasr-onnx-2pass.cpp")
add_executable(funasr-onnx-2pass "funasr-onnx-2pass.cpp" ${RELATION_SOURCE})
target_link_libraries(funasr-onnx-2pass PUBLIC funasr)
add_executable(funasr-onnx-2pass-rtf "funasr-onnx-2pass-rtf.cpp")
add_executable(funasr-onnx-2pass-rtf "funasr-onnx-2pass-rtf.cpp" ${RELATION_SOURCE})
target_link_libraries(funasr-onnx-2pass-rtf PUBLIC funasr)
add_executable(funasr-onnx-online-rtf "funasr-onnx-online-rtf.cpp")
add_executable(funasr-onnx-online-rtf "funasr-onnx-online-rtf.cpp" ${RELATION_SOURCE})
target_link_libraries(funasr-onnx-online-rtf PUBLIC funasr)
# include_directories(${FFMPEG_DIR}/include)

90
runtime/onnxruntime/readme.md
View File

@ -1,34 +1,56 @@
# Please ref to [websocket service](https://github.com/alibaba-damo-academy/FunASR/tree/main/runtime/websocket)
# If you want to compile the file yourself, you can follow the steps below.
## Building for Linux/Unix
### Download onnxruntime
```shell
wget https://github.com/microsoft/onnxruntime/releases/download/v1.14.0/onnxruntime-linux-x64-1.14.0.tgz
tar -zxvf onnxruntime-linux-x64-1.14.0.tgz
```
### Download ffmpeg
```shell
wget https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/dep_libs/ffmpeg-N-111383-g20b8688092-linux64-gpl-shared.tar.xz
tar -xvf ffmpeg-N-111383-g20b8688092-linux64-gpl-shared.tar.xz
```
### Install deps
```shell
# openblas
sudo apt-get install libopenblas-dev #ubuntu
# sudo yum -y install openblas-devel #centos
# openssl
apt-get install libssl-dev #ubuntu
# yum install openssl-devel #centos
```
### Build runtime
```shell
git clone https://github.com/alibaba-damo-academy/FunASR.git && cd FunASR/runtime/onnxruntime
mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=release .. -DONNXRUNTIME_DIR=/path/to/onnxruntime-linux-x64-1.14.0 -DFFMPEG_DIR=/path/to/ffmpeg-N-111383-g20b8688092-linux64-gpl-shared
make -j 4
```
# Please ref to [websocket service](https://github.com/alibaba-damo-academy/FunASR/tree/main/runtime/websocket)
# If you want to compile the file yourself, you can follow the steps below.
## Building for Linux/Unix
### Download onnxruntime
```shell
wget https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/dep_libs/onnxruntime-linux-x64-1.14.0.tgz
tar -zxvf onnxruntime-linux-x64-1.14.0.tgz
```
### Download ffmpeg
```shell
wget https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/dep_libs/ffmpeg-master-latest-linux64-gpl-shared.tar.xz
tar -xvf ffmpeg-master-latest-linux64-gpl-shared.tar.xz
```
### Install deps
```shell
# openblas
sudo apt-get install libopenblas-dev #ubuntu
# sudo yum -y install openblas-devel #centos
# openssl
apt-get install libssl-dev #ubuntu
# yum install openssl-devel #centos
```
### Build runtime
```shell
git clone https://github.com/alibaba-damo-academy/FunASR.git && cd FunASR/runtime/onnxruntime
mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=release .. -DONNXRUNTIME_DIR=/path/to/onnxruntime-linux-x64-1.14.0 -DFFMPEG_DIR=/path/to/ffmpeg-master-latest-linux64-gpl-shared
make -j 4
```
## Building for Windows
### Download onnxruntime
https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/dep_libs/onnxruntime-win-x64-1.16.1.zip
Download and unzip to d:\ffmpeg-master-latest-win64-gpl-shared
### Download ffmpeg
https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/dep_libs/ffmpeg-master-latest-win64-gpl-shared.zip
Download and unzip to d:\onnxruntime-win-x64-1.16.1
### Build runtime
```
git clone https://github.com/alibaba-damo-academy/FunASR.git
cd FunASR/runtime/onnxruntime
mkdir build
cd build
cmake ../ -D FFMPEG_DIR=d:/ffmpeg-master-latest-win64-gpl-shared -D ONNXRUNTIME_DIR=d:/onnxruntime-win-x64-1.16.1
```
Visual Studio open FunASR/runtime/onnxruntime/build/FunASROnnx.sln start build

20
runtime/onnxruntime/src/CMakeLists.txt
View File

@ -4,18 +4,20 @@ set(files ${files1})
message("files: "${files})
if(WIN32)
add_compile_options("$<$<CXX_COMPILER_ID:MSVC>:/execution-charset:utf-8>")
add_compile_options("$<$<CXX_COMPILER_ID:MSVC>:/source-charset:utf-8>")
endif()
add_library(funasr SHARED ${files})
if(WIN32)
set(EXTRA_LIBS pthread yaml-cpp csrc kaldi-decoder fst glog gflags)
if(CMAKE_CL_64)
target_link_directories(funasr PUBLIC ${CMAKE_SOURCE_DIR}/win/lib/x64)
else()
target_link_directories(funasr PUBLIC ${CMAKE_SOURCE_DIR}/win/lib/x86)
endif()
target_include_directories(funasr PUBLIC ${CMAKE_SOURCE_DIR}/win/include )
target_compile_definitions(funasr PUBLIC -D_FUNASR_API_EXPORT)
set(EXTRA_LIBS yaml-cpp csrc kaldi-decoder fst glog gflags avutil avcodec avformat swresample onnxruntime)
include_directories(${ONNXRUNTIME_DIR}/include)
include_directories(${FFMPEG_DIR}/include)
target_link_directories(funasr PUBLIC ${ONNXRUNTIME_DIR}/lib)
target_link_directories(funasr PUBLIC ${FFMPEG_DIR}/lib)
target_compile_definitions(funasr PUBLIC -D_FUNASR_API_EXPORT -DNOMINMAX -DYAML_CPP_DLL)
else()
set(EXTRA_LIBS pthread yaml-cpp csrc kaldi-decoder fst glog gflags avutil avcodec avformat swresample)
include_directories(${ONNXRUNTIME_DIR}/include)

2
runtime/onnxruntime/src/audio.cpp
View File

@ -1193,7 +1193,7 @@ void Audio::Split(VadModel* vad_obj, int chunk_len, bool input_finished, ASR_TYP
}
}else if(speech_end_i != -1){ // [-1,100]
if(speech_start == -1 or speech_offline_start == -1){
if(speech_start == -1 || speech_offline_start == -1){
LOG(ERROR) <<"Vad start is null while vad end is available. Set vad start 0" ;
speech_start = 0;
}

3
runtime/onnxruntime/src/bias-lm.h
View File

@ -7,6 +7,9 @@
#include "vocab.h"
#include "util/text-utils.h"
#include <yaml-cpp/yaml.h>
#ifdef _WIN32
#include "win_func.h"
#endif
// node type
#define ROOT_NODE 0
#define VALUE_ZERO 0.0f

4
runtime/onnxruntime/src/funasrruntime.cpp
View File

@ -266,7 +266,7 @@
if(msg_vec.size()==0){
continue;
}
if(lang == "en-bpe" and p_result->msg != ""){
if(lang == "en-bpe" && p_result->msg != ""){
p_result->msg += " ";
}
p_result->msg += msg_vec[0];
@ -355,7 +355,7 @@
if(msg_vec.size()==0){
continue;
}
if(lang == "en-bpe" and p_result->msg != ""){
if(lang == "en-bpe" && p_result->msg != ""){
p_result->msg += " ";
}
p_result->msg += msg_vec[0];

15
runtime/onnxruntime/src/precomp.h
View File

@ -18,20 +18,7 @@
#include <cstring>
#ifdef _WIN32
#include<io.h>
#ifndef R_OK
#define R_OK 4
#endif
#ifndef W_OK
#define W_OK 2
#endif
#ifndef X_OK
#define X_OK 0
#endif
#ifndef F_OK
#define F_OK 0
#endif
#define access _access
#include <win_func.h>
#else
#include <unistd.h>
#endif

2
runtime/onnxruntime/src/vocab.cpp
View File

@ -199,7 +199,7 @@ string Vocab::Vector2StringV2(vector<int> in, std::string language)
}
}
if (language == "en-bpe" and combine != ""){
if (language == "en-bpe" && combine != ""){
combine = WordFormat(combine);
if (words.size() != 0){
combine = " " + combine;

38
runtime/onnxruntime/src/win_func.h Normal file
View File

@ -0,0 +1,38 @@
#ifndef WIN_FUNC_
#define WIN_FUNC_
#ifdef _WIN32
#ifndef WIN32_LEAN_AND_MEAN
#define WIN32_LEAN_AND_MEAN
#endif
#include <windows.h>
#include <winsock.h>
#include<io.h>
#ifndef R_OK
#define R_OK 4
#endif
#ifndef W_OK
#define W_OK 2
#endif
#ifndef X_OK
#define X_OK 0
#endif
#ifndef F_OK
#define F_OK 0
#endif
#define access _access
static inline int gettimeofday(struct timeval* tv, void* /*tz*/) {
FILETIME ft;
ULARGE_INTEGER li;
ULONGLONG tt;
GetSystemTimeAsFileTime(&ft);
li.LowPart = ft.dwLowDateTime;
li.HighPart = ft.dwHighDateTime;
tt = (li.QuadPart - 116444736000000000ULL) / 10;
tv->tv_sec = tt / 1000000;
tv->tv_usec = tt % 1000000;
return 0;
}
#endif
#endif

148
runtime/onnxruntime/third_party/glog/CMakeLists.txt vendored
View File

@ -1,4 +1,4 @@
cmake_minimum_required (VERSION 3.16)
cmake_minimum_required (VERSION 3.21)
project (glog
VERSION 0.7.0
DESCRIPTION "C++ implementation of the Google logging module"
@ -131,6 +131,7 @@ check_function_exists (sigaltstack HAVE_SIGALTSTACK)
check_cxx_symbol_exists (backtrace execinfo.h HAVE_EXECINFO_BACKTRACE)
check_cxx_symbol_exists (backtrace_symbols execinfo.h
HAVE_EXECINFO_BACKTRACE_SYMBOLS)
check_cxx_symbol_exists (_chsize_s io.h HAVE__CHSIZE_S)
# NOTE gcc does not fail if you pass a non-existent -Wno-* option as an
# argument. However, it will happily fail if you pass the corresponding -W*
@ -213,50 +214,61 @@ if (WITH_TLS)
endif (WITH_TLS)
set (_PC_FIELDS
"gregs[REG_PC]"
"gregs[REG_EIP]"
"gregs[REG_RIP]"
"sc_ip"
"uc_regs->gregs[PT_NIP]"
"gregs[R15]"
"arm_pc"
"mc_eip"
"mc_rip"
"__gregs[REG_EIP]"
"__gregs[REG_RIP]"
"ss.eip"
"__ss.__eip"
"ss.rip"
"__ss.__rip"
"ss.srr0"
"__ss.__srr0"
"uc_mcontext.gregs[REG_PC]" # Solaris x86 (32 + 64 bit)
"uc_mcontext.gregs[REG_EIP]" # Linux (i386)
"uc_mcontext.gregs[REG_RIP]" # Linux (x86_64)
"uc_mcontext.sc_ip" # Linux (ia64)
"uc_mcontext.pc" # Linux (mips)
"uc_mcontext.uc_regs->gregs[PT_NIP]" # Linux (ppc)
"uc_mcontext.gregs[R15]" # Linux (arm old [untested])
"uc_mcontext.arm_pc" # Linux (arm arch 5)
"uc_mcontext.gp_regs[PT_NIP]" # Suse SLES 11 (ppc64)
"uc_mcontext.mc_eip" # FreeBSD (i386)
"uc_mcontext.mc_rip" # FreeBSD (x86_64 [untested])
"uc_mcontext.__gregs[_REG_EIP]" # NetBSD (i386)
"uc_mcontext.__gregs[_REG_RIP]" # NetBSD (x86_64)
"uc_mcontext->ss.eip" # OS X (i386, <=10.4)
"uc_mcontext->__ss.__eip" # OS X (i386, >=10.5)
"uc_mcontext->ss.rip" # OS X (x86_64)
"uc_mcontext->__ss.__rip" # OS X (>=10.5 [untested])
"uc_mcontext->ss.srr0" # OS X (ppc, ppc64 [untested])
"uc_mcontext->__ss.__srr0" # OS X (>=10.5 [untested])
)
set (_PC_HEADERS ucontext.h signal.h)
if (HAVE_UCONTEXT_H AND NOT DEFINED PC_FROM_UCONTEXT)
cmake_push_check_state (RESET)
if (HAVE_UCONTEXT_H AND NOT PC_FROM_UCONTEXT)
foreach (_PC_FIELD ${_PC_FIELDS})
foreach (_PC_HEADER ${_PC_HEADERS})
set (_TMP
${CMAKE_CURRENT_BINARY_DIR}${CMAKE_FILES_DIRECTORY}/CMakeTmp/uctfield.cpp)
file (WRITE ${_TMP} "
#define _GNU_SOURCE 1
#include <${_PC_HEADER}>
int main(void)
{
ucontext_t u;
return u.${_PC_FIELD} == 0;
}
")
try_compile (HAVE_PC_FROM_UCONTEXT ${CMAKE_CURRENT_BINARY_DIR} ${_TMP}
COMPILE_DEFINITIONS _GNU_SOURCE=1)
set (CMAKE_REQUIRED_DEFINITIONS -D_GNU_SOURCE)
set (_PC_HEADERS ucontext.h signal.h)
if (HAVE_PC_FROM_UCONTEXT)
set (PC_FROM_UCONTEXT ${_PC_FIELD} CACHE)
endif (HAVE_PC_FROM_UCONTEXT)
foreach (_PC_FIELD IN LISTS _PC_FIELDS)
foreach (_PC_HEADER IN LISTS _PC_HEADERS)
# Replace non-alphanumeric characters by underscores since the name will be
# used as preprocessor definition.
string (REGEX REPLACE "[^a-zA-Z0-9]" "_" HAVE_UCONTEXT_FIELD_NAME
"HAVE_PC_FROM_UCONTEXT_${_PC_FIELD}")
# Strip trailing underscores for readability
string (REGEX REPLACE "_+$" "" HAVE_UCONTEXT_FIELD_NAME
"${HAVE_UCONTEXT_FIELD_NAME}")
check_struct_has_member (ucontext_t ${_PC_FIELD} ${_PC_HEADER}
${HAVE_UCONTEXT_FIELD_NAME} LANGUAGE CXX)
if (${HAVE_UCONTEXT_FIELD_NAME})
set (PC_FROM_UCONTEXT ${_PC_FIELD} CACHE STRING
"<${_PC_HEADER}> ucontext_t PC member")
mark_as_advanced (PC_FROM_UCONTEXT)
break ()
endif (${HAVE_UCONTEXT_FIELD_NAME})
endforeach (_PC_HEADER)
if (${HAVE_UCONTEXT_FIELD_NAME})
break ()
endif (${HAVE_UCONTEXT_FIELD_NAME})
endforeach (_PC_FIELD)
endif (HAVE_UCONTEXT_H AND NOT PC_FROM_UCONTEXT)
cmake_pop_check_state ()
endif (HAVE_UCONTEXT_H AND NOT DEFINED PC_FROM_UCONTEXT)
set (GOOGLE_NAMESPACE google)
set (_START_GOOGLE_NAMESPACE_ "namespace ${GOOGLE_NAMESPACE} {")
@ -468,6 +480,7 @@ set (GLOG_SRCS
src/demangle.h
src/logging.cc
src/raw_logging.cc
src/signalhandler.cc
src/symbolize.cc
src/symbolize.h
src/utilities.cc
@ -475,10 +488,6 @@ set (GLOG_SRCS
src/vlog_is_on.cc
)
if (HAVE_PTHREAD OR WIN32 OR CYGWIN)
list (APPEND GLOG_SRCS src/signalhandler.cc)
endif (HAVE_PTHREAD OR WIN32 OR CYGWIN)
if (CYGWIN OR WIN32)
list (APPEND GLOG_SRCS
src/windows/port.cc
@ -491,9 +500,20 @@ add_library (glog_internal OBJECT
${GLOG_SRCS}
)
target_compile_features (glog_internal PUBLIC $<TARGET_PROPERTY:glog,COMPILE_FEATURES>)
set_target_properties (glog_internal PROPERTIES DEFINE_SYMBOL GOOGLE_GLOG_IS_A_DLL)
# Some generators (such as Xcode) do not generate any output if the target does
# not reference at least one source file.
set (_glog_EMPTY_SOURCE ${glog_BINARY_DIR}${CMAKE_FILES_DIRECTORY}/glog.cc)
add_custom_command (
OUTPUT ${_glog_EMPTY_SOURCE}
COMMAND ${CMAKE_COMMAND} -E touch ${_glog_EMPTY_SOURCE}
)
add_library (glog
$<TARGET_OBJECTS:glog_internal>
${_glog_EMPTY_SOURCE}
)
target_compile_features (glog PUBLIC cxx_std_14)
@ -615,38 +635,30 @@ if (NOT WITH_FUZZING STREQUAL "none")
endif (NOT WITH_FUZZING STREQUAL "none")
if (BUILD_TESTING)
add_library (glogtest STATIC
$<TARGET_OBJECTS:glog_internal>
)
target_include_directories (glogtest PUBLIC
$<TARGET_PROPERTY:glog,INCLUDE_DIRECTORIES>)
target_compile_definitions (glogtest PUBLIC
$<TARGET_PROPERTY:glog,COMPILE_DEFINITIONS> GLOG_STATIC_DEFINE)
target_link_libraries (glogtest PUBLIC
$<TARGET_PROPERTY:glog,LINK_LIBRARIES>)
set (_GLOG_TEST_LIBS glogtest)
add_library (glog_test INTERFACE)
target_link_libraries (glog_test INTERFACE $<TARGET_OBJECTS:glog_internal> $<TARGET_PROPERTY:glog,LINK_LIBRARIES>)
target_compile_definitions (glog_test INTERFACE GLOG_STATIC_DEFINE $<TARGET_PROPERTY:glog,COMPILE_DEFINITIONS>)
target_include_directories (glog_test INTERFACE $<TARGET_PROPERTY:glog,INCLUDE_DIRECTORIES>)
if (HAVE_LIB_GTEST)
list (APPEND _GLOG_TEST_LIBS GTest::gtest)
target_link_libraries (glog_test INTERFACE GTest::gtest)
endif (HAVE_LIB_GTEST)
if (HAVE_LIB_GMOCK)
list (APPEND _GLOG_TEST_LIBS GTest::gmock)
target_link_libraries (glog_test INTERFACE GTest::gmock)
endif (HAVE_LIB_GMOCK)
add_executable (logging_unittest
src/logging_unittest.cc
)
target_link_libraries (logging_unittest PRIVATE ${_GLOG_TEST_LIBS})
target_link_libraries (logging_unittest PRIVATE glog_test)
add_executable (stl_logging_unittest
src/stl_logging_unittest.cc
)
target_link_libraries (stl_logging_unittest PRIVATE ${_GLOG_TEST_LIBS})
target_link_libraries (stl_logging_unittest PRIVATE glog_test)
if (HAVE_NO_DEPRECATED)
set_property (TARGET stl_logging_unittest APPEND PROPERTY COMPILE_OPTIONS
@ -663,35 +675,35 @@ if (BUILD_TESTING)
src/symbolize_unittest.cc
)
target_link_libraries (symbolize_unittest PRIVATE ${_GLOG_TEST_LIBS})
target_link_libraries (symbolize_unittest PRIVATE glog_test)
endif (HAVE_SYMBOLIZE)
add_executable (demangle_unittest
src/demangle_unittest.cc
)
target_link_libraries (demangle_unittest PRIVATE ${_GLOG_TEST_LIBS})
target_link_libraries (demangle_unittest PRIVATE glog_test)
if (HAVE_STACKTRACE)
add_executable (stacktrace_unittest
src/stacktrace_unittest.cc
)
target_link_libraries (stacktrace_unittest PRIVATE ${_GLOG_TEST_LIBS})
target_link_libraries (stacktrace_unittest PRIVATE glog_test)
endif (HAVE_STACKTRACE)
add_executable (utilities_unittest
src/utilities_unittest.cc
)
target_link_libraries (utilities_unittest PRIVATE ${_GLOG_TEST_LIBS})
target_link_libraries (utilities_unittest PRIVATE glog_test)
if (HAVE_STACKTRACE AND HAVE_SYMBOLIZE)
add_executable (signalhandler_unittest
src/signalhandler_unittest.cc
)
target_link_libraries (signalhandler_unittest PRIVATE ${_GLOG_TEST_LIBS})
target_link_libraries (signalhandler_unittest PRIVATE glog_test)
endif (HAVE_STACKTRACE AND HAVE_SYMBOLIZE)
add_test (NAME demangle COMMAND demangle_unittest)
@ -737,7 +749,7 @@ if (BUILD_TESTING)
src/mock-log.h
)
target_link_libraries (mock-log_unittest PRIVATE ${_GLOG_TEST_LIBS})
target_link_libraries (mock-log_unittest PRIVATE glog_test)
add_test (NAME mock-log COMMAND mock-log_unittest)
endif (HAVE_LIB_GMOCK)
@ -799,17 +811,17 @@ if (BUILD_TESTING)
add_executable (cleanup_immediately_unittest
src/cleanup_immediately_unittest.cc)
target_link_libraries (cleanup_immediately_unittest PRIVATE ${_GLOG_TEST_LIBS})
target_link_libraries (cleanup_immediately_unittest PRIVATE glog_test)
add_executable (cleanup_with_absolute_prefix_unittest
src/cleanup_with_absolute_prefix_unittest.cc)
target_link_libraries (cleanup_with_absolute_prefix_unittest PRIVATE ${_GLOG_TEST_LIBS})
target_link_libraries (cleanup_with_absolute_prefix_unittest PRIVATE glog_test)
add_executable (cleanup_with_relative_prefix_unittest
src/cleanup_with_relative_prefix_unittest.cc)
target_link_libraries (cleanup_with_relative_prefix_unittest PRIVATE ${_GLOG_TEST_LIBS})
target_link_libraries (cleanup_with_relative_prefix_unittest PRIVATE glog_test)
set (CLEANUP_LOG_DIR ${CMAKE_CURRENT_BINARY_DIR}/cleanup_tests)

2
runtime/onnxruntime/third_party/glog/cmake/GetCacheVariables.cmake vendored
View File

@ -1,5 +1,5 @@
cmake_policy (PUSH)
cmake_policy (VERSION 3.3)
cmake_policy (VERSION 3.16...3.27)
include (CMakeParseArguments)

18
runtime/onnxruntime/third_party/glog/cmake/TestPackageConfig.cmake vendored
View File

@ -1,13 +1,3 @@
# Create the build directory
execute_process (
COMMAND ${CMAKE_COMMAND} -E make_directory ${TEST_BINARY_DIR}
RESULT_VARIABLE _DIRECTORY_CREATED_SUCCEEDED
)
if (NOT _DIRECTORY_CREATED_SUCCEEDED EQUAL 0)
message (FATAL_ERROR "Failed to create build directory")
endif (NOT _DIRECTORY_CREATED_SUCCEEDED EQUAL 0)
if (GENERATOR_TOOLSET)
list (APPEND _ADDITIONAL_ARGS -T ${GENERATOR_TOOLSET})
endif (GENERATOR_TOOLSET)
@ -21,7 +11,7 @@ execute_process (
# Capture the PATH environment variable content set during project generation
# stage. This is required because later during the build stage the PATH is
# modified again (e.g., for MinGW AppVeyor CI builds) by adding back the
# directory containing git.exe. Incidently, the Git installation directory
# directory containing git.exe. Incidentally, the Git installation directory
# also contains sh.exe which causes MinGW Makefile generation to fail.
COMMAND ${CMAKE_COMMAND} -E env PATH=${PATH}
${CMAKE_COMMAND} -C ${INITIAL_CACHE}
@ -29,9 +19,9 @@ execute_process (
${_ADDITIONAL_ARGS}
-DCMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY=ON
-DCMAKE_FIND_PACKAGE_NO_SYSTEM_PACKAGE_REGISTRY=ON
-DCMAKE_PREFIX_PATH=${PACKAGE_DIR}
${SOURCE_DIR}
WORKING_DIRECTORY ${TEST_BINARY_DIR}
-Dglog_ROOT=${PACKAGE_DIR}
-S ${SOURCE_DIR}
-B ${TEST_BINARY_DIR}
RESULT_VARIABLE _GENERATE_SUCCEEDED
)

9
runtime/onnxruntime/third_party/glog/src/base/commandlineflags.h vendored
View File

@ -59,7 +59,7 @@
#else
#include <glog/logging.h>
#include "glog/logging.h"
#define DECLARE_VARIABLE(type, shorttype, name, tn) \
namespace fL##shorttype { \
@ -140,9 +140,12 @@
: memchr("tTyY1\0", getenv(envname)[0], 6) != nullptr)
#define EnvToInt(envname, dflt) \
(!getenv(envname) ? (dflt) : strtol(getenv(envname), nullptr, 10))
(!getenv(envname) ? (dflt) \
: static_cast<int>(strtol(getenv(envname), nullptr, 10)))
#define EnvToUInt(envname, dflt) \
(!getenv(envname) ? (dflt) : strtoul(getenv(envname), nullptr, 10))
(!getenv(envname) \
? (dflt) \
: static_cast<unsigned>(strtoul(getenv(envname), nullptr, 10)))
#endif // BASE_COMMANDLINEFLAGS_H__

5
runtime/onnxruntime/third_party/glog/src/config.h.cmake.in vendored
View File

@ -130,12 +130,15 @@
/* define if gmtime_r is available in time.h */
#cmakedefine HAVE_GMTIME_R
/* define if _chsize_s is available in io.h */
#cmakedefine HAVE__CHSIZE_S
/* Define to the sub-directory in which libtool stores uninstalled libraries.
*/
#cmakedefine LT_OBJDIR
/* How to access the PC from a struct ucontext */
#cmakedefine PC_FROM_UCONTEXT
#cmakedefine PC_FROM_UCONTEXT ${PC_FROM_UCONTEXT}
/* define if we should print file offsets in traces instead of symbolizing. */
#cmakedefine PRINT_UNSYMBOLIZED_STACK_TRACES

68
runtime/onnxruntime/third_party/glog/src/demangle.cc vendored
View File

@ -36,7 +36,9 @@
#include "demangle.h"
#include <cstddef>
#include <cstdio> // for nullptr
#include <limits>
#include "utilities.h"
@ -109,6 +111,9 @@ struct State {
short nest_level; // For nested names.
bool append; // Append flag.
bool overflowed; // True if output gets overflowed.
uint32 local_level;
uint32 expr_level;
uint32 arg_level;
};
// We don't use strlen() in libc since it's not guaranteed to be async
@ -153,6 +158,9 @@ static void InitState(State *state, const char *mangled,
state->nest_level = -1;
state->append = true;
state->overflowed = false;
state->local_level = 0;
state->expr_level = 0;
state->arg_level = 0;
}
// Returns true and advances "mangled_cur" if we find "one_char_token"
@ -221,6 +229,10 @@ static bool ZeroOrMore(ParseFunc parse_func, State *state) {
// is set to true for later use. The output string is ensured to
// always terminate with '\0' as long as there is no overflow.
static void Append(State *state, const char * const str, ssize_t length) {
if (state->out_cur == nullptr) {
state->overflowed = true;
return;
}
for (ssize_t i = 0; i < length; ++i) {
if (state->out_cur + 1 < state->out_end) { // +1 for '\0'
*state->out_cur = str[i];
@ -294,7 +306,7 @@ static void MaybeAppendWithLength(State *state, const char * const str,
}
}
// A convenient wrapper arount MaybeAppendWithLength().
// A convenient wrapper around MaybeAppendWithLength().
static bool MaybeAppend(State *state, const char * const str) {
if (state->append) {
size_t length = StrLen(str);
@ -592,9 +604,23 @@ static bool ParseNumber(State *state, int *number_out) {
}
const char *p = state->mangled_cur;
int number = 0;
constexpr int int_max_by_10 = std::numeric_limits<int>::max() / 10;
for (;*p != '\0'; ++p) {
if (IsDigit(*p)) {
number = number * 10 + (*p - '0');
// Prevent signed integer overflow when multiplying
if (number > int_max_by_10) {
return false;
}
const int digit = *p - '0';
const int shifted = number * 10;
// Prevent signed integer overflow when summing
if (digit > std::numeric_limits<int>::max() - shifted) {
return false;
}
number = shifted + digit;
} else {
break;
}
@ -652,6 +678,10 @@ static bool ParseIdentifier(State *state, ssize_t length) {
} else {
MaybeAppendWithLength(state, state->mangled_cur, length);
}
if (length < 0 ||
static_cast<std::size_t>(length) > StrLen(state->mangled_cur)) {
return false;
}
state->mangled_cur += length;
return true;
}
@ -1067,22 +1097,33 @@ static bool ParseTemplateArgs(State *state) {
// ::= J <template-arg>* E # argument pack
// ::= X <expression> E
static bool ParseTemplateArg(State *state) {
// Avoid recursion above max_levels
constexpr uint32 max_levels = 5;
if (state->arg_level > max_levels) {
return false;
}
++state->arg_level;
State copy = *state;
if ((ParseOneCharToken(state, 'I') || ParseOneCharToken(state, 'J')) &&
ZeroOrMore(ParseTemplateArg, state) &&
ParseOneCharToken(state, 'E')) {
--state->arg_level;
return true;
}
*state = copy;
if (ParseType(state) ||
ParseExprPrimary(state)) {
--state->arg_level;
return true;
}
*state = copy;
if (ParseOneCharToken(state, 'X') && ParseExpression(state) &&
ParseOneCharToken(state, 'E')) {
--state->arg_level;
return true;
}
*state = copy;
@ -1103,11 +1144,20 @@ static bool ParseExpression(State *state) {
return true;
}
// Avoid recursion above max_levels
constexpr uint32 max_levels = 5;
if (state->expr_level > max_levels) {
return false;
}
++state->expr_level;
State copy = *state;
if (ParseOperatorName(state) &&
ParseExpression(state) &&
ParseExpression(state) &&
ParseExpression(state)) {
--state->expr_level;
return true;
}
*state = copy;
@ -1115,30 +1165,35 @@ static bool ParseExpression(State *state) {
if (ParseOperatorName(state) &&
ParseExpression(state) &&
ParseExpression(state)) {
--state->expr_level;
return true;
}
*state = copy;
if (ParseOperatorName(state) &&
ParseExpression(state)) {
--state->expr_level;
return true;
}
*state = copy;
if (ParseTwoCharToken(state, "st") && ParseType(state)) {
return true;
--state->expr_level;
}
*state = copy;
if (ParseTwoCharToken(state, "sr") && ParseType(state) &&
ParseUnqualifiedName(state) &&
ParseTemplateArgs(state)) {
--state->expr_level;
return true;
}
*state = copy;
if (ParseTwoCharToken(state, "sr") && ParseType(state) &&
ParseUnqualifiedName(state)) {
--state->expr_level;
return true;
}
*state = copy;
@ -1184,16 +1239,25 @@ static bool ParseExprPrimary(State *state) {
// [<discriminator>]
// := Z <(function) encoding> E s [<discriminator>]
static bool ParseLocalName(State *state) {
// Avoid recursion above max_levels
constexpr uint32 max_levels = 5;
if (state->local_level > max_levels) {
return false;
}
++state->local_level;
State copy = *state;
if (ParseOneCharToken(state, 'Z') && ParseEncoding(state) &&
ParseOneCharToken(state, 'E') && MaybeAppend(state, "::") &&
ParseName(state) && Optional(ParseDiscriminator(state))) {
--state->local_level;
return true;
}
*state = copy;
if (ParseOneCharToken(state, 'Z') && ParseEncoding(state) &&
ParseTwoCharToken(state, "Es") && Optional(ParseDiscriminator(state))) {
--state->local_level;
return true;
}
*state = copy;

2
runtime/onnxruntime/third_party/glog/src/demangle.h vendored
View File

@ -71,7 +71,7 @@
#define BASE_DEMANGLE_H_
#include "config.h"
#include <glog/logging.h>
#include "glog/logging.h"
_START_GOOGLE_NAMESPACE_

65
runtime/onnxruntime/third_party/glog/src/glog/logging.h.in vendored
View File

@ -60,10 +60,10 @@
#define GLOG_MSVC_POP_WARNING()
#endif
#include <glog/platform.h>
#include "glog/platform.h"
#if @ac_cv_have_glog_export@
#include <glog/export.h>
#include "glog/export.h"
#endif
// We care a lot about number of bits things take up. Unfortunately,
@ -290,11 +290,11 @@ typedef void(*CustomPrefixCallback)(std::ostream& s, const LogMessageInfo& l, vo
// "program with --v=1 or more";
// VLOG_EVERY_N(1, 10)
// << "I'm printed every 10th occurrence, and when you run the program "
// "with --v=1 or more. Present occurence is " << google::COUNTER;
// "with --v=1 or more. Present occurrence is " << google::COUNTER;
// VLOG_IF_EVERY_N(1, (size > 1024), 10)
// << "I'm printed on every 10th occurence of case when size is more "
// << "I'm printed on every 10th occurrence of case when size is more "
// " than 1024, when you run the program with --v=1 or more. ";
// "Present occurence is " << google::COUNTER;
// "Present occurrence is " << google::COUNTER;
//
// The supported severity levels for macros that allow you to specify one
// are (in increasing order of severity) INFO, WARNING, ERROR, and FATAL.
@ -470,6 +470,9 @@ DECLARE_bool(stop_logging_if_full_disk);
// Use UTC time for logging
DECLARE_bool(log_utc_time);
// Mailer used to send logging email
DECLARE_string(logmailer);
// Log messages below the GOOGLE_STRIP_LOG level will be compiled away for
// security reasons. See LOG(severtiy) below.
@ -596,8 +599,8 @@ DECLARE_bool(log_utc_time);
@ac_google_start_namespace@
// They need the definitions of integer types.
#include <glog/log_severity.h>
#include <glog/vlog_is_on.h>
#include "glog/log_severity.h"
#include "glog/vlog_is_on.h"
// Initialize google's logging library. You will see the program name
// specified by argv0 in log outputs.
@ -842,7 +845,7 @@ DEFINE_CHECK_OP_IMPL(Check_GT, > )
#define CHECK_OP_LOG(name, op, val1, val2, log) CHECK((val1) op (val2))
#elif DCHECK_IS_ON()
// In debug mode, avoid constructing CheckOpStrings if possible,
// to reduce the overhead of CHECK statments by 2x.
// to reduce the overhead of CHECK statements by 2x.
// Real DCHECK-heavy tests have seen 1.5x speedups.
// The meaning of "string" might be different between now and
@ -1042,13 +1045,13 @@ namespace google {
constexpr std::chrono::nanoseconds LOG_TIME_PERIOD = \
std::chrono::duration_cast<std::chrono::nanoseconds>( \
std::chrono::duration<double>(seconds)); \
static std::atomic<@ac_google_namespace@ ::int64> LOG_PREVIOUS_TIME_RAW; \
static std::atomic<@ac_google_namespace@::int64> LOG_PREVIOUS_TIME_RAW; \
GLOG_IFDEF_THREAD_SANITIZER( \
AnnotateBenignRaceSized(__FILE__, __LINE__, &LOG_TIME_PERIOD, \
sizeof(@ac_google_namespace @ ::int64), "")); \
sizeof(@ac_google_namespace@::int64), "")); \
GLOG_IFDEF_THREAD_SANITIZER( \
AnnotateBenignRaceSized(__FILE__, __LINE__, &LOG_PREVIOUS_TIME_RAW, \
sizeof(@ac_google_namespace @ ::int64), "")); \
sizeof(@ac_google_namespace@::int64), "")); \
const auto LOG_CURRENT_TIME = \
std::chrono::duration_cast<std::chrono::nanoseconds>( \
std::chrono::steady_clock::now().time_since_epoch()); \
@ -1062,8 +1065,8 @@ namespace google {
.count(), \
std::memory_order_relaxed); \
if (LOG_TIME_DELTA > LOG_TIME_PERIOD) \
@ac_google_namespace@ ::LogMessage( \
__FILE__, __LINE__, @ac_google_namespace@ ::GLOG_##severity) \
@ac_google_namespace@::LogMessage( \
__FILE__, __LINE__, @ac_google_namespace@::GLOG_##severity) \
.stream()
#define SOME_KIND_OF_LOG_EVERY_N(severity, n, what_to_do) \
@ -1344,6 +1347,15 @@ GLOG_MSVC_PUSH_DISABLE_WARNING(4275)
class GLOG_EXPORT LogStream : public std::ostream {
GLOG_MSVC_POP_WARNING()
public:
#if defined __has_attribute
# if __has_attribute (used)
// In some cases, like when compiling glog as a static library with GCC and
// linking against a Clang-built executable, this constructor will be
// removed by the linker. We use this attribute to prevent the linker from
// discarding it.
__attribute__ ((used))
# endif
#endif
LogStream(char *buf, int len, int64 ctr)
: std::ostream(NULL),
streambuf_(buf, len),
@ -1425,7 +1437,7 @@ public:
// is so that streaming can be done more efficiently.
static const size_t kMaxLogMessageLen;
// Theses should not be called directly outside of logging.*,
// These should not be called directly outside of logging.*,
// only passed as SendMethod arguments to other LogMessage methods:
void SendToLog(); // Actually dispatch to the logs
void SendToSyslogAndLog(); // Actually dispatch to syslog and the logs
@ -1652,8 +1664,8 @@ GLOG_EXPORT void LogToStderr();
GLOG_EXPORT void SetEmailLogging(LogSeverity min_severity,
const char* addresses);
// A simple function that sends email. dest is a commma-separated
// list of addressess. Thread-safe.
// A simple function that sends email. dest is a comma-separated
// list of addresses. Thread-safe.
GLOG_EXPORT bool SendEmail(const char* dest, const char* subject,
const char* body);
@ -1768,11 +1780,11 @@ class GLOG_EXPORT NullStream : public LogMessage::LogStream {
// (they'll never be actually displayed). This will be needed if a
// NullStream& is implicitly converted to LogStream&, in which case
// the overloaded NullStream::operator<< will not be invoked.
NullStream() : LogMessage::LogStream(message_buffer_, 1, 0) { }
NullStream();
NullStream(const char* /*file*/, int /*line*/,
const CheckOpString& /*result*/) :
LogMessage::LogStream(message_buffer_, 1, 0) { }
NullStream &stream() { return *this; }
const CheckOpString& /*result*/);
NullStream& stream();
private:
// A very short buffer for messages (which we discard anyway). This
// will be needed if NullStream& converted to LogStream& (e.g. as a
@ -1793,17 +1805,8 @@ inline NullStream& operator<<(NullStream &str, const T &) { return str; }
// trace), like LogMessageFatal.
class GLOG_EXPORT NullStreamFatal : public NullStream {
public:
NullStreamFatal() { }
NullStreamFatal(const char* file, int line, const CheckOpString& result) :
NullStream(file, line, result) { }
#if defined(_MSC_VER)
#pragma warning(push)
#pragma warning(disable : 4722)
#endif // _MSC_VER
[[noreturn]] ~NullStreamFatal() throw() { _exit(EXIT_FAILURE); }
#if defined(_MSC_VER)
#pragma warning(pop)
#endif // _MSC_VER
using NullStream::NullStream;
[[noreturn]] ~NullStreamFatal();
};
// Install a signal handler that will dump signal information and a stack

7
runtime/onnxruntime/third_party/glog/src/glog/raw_logging.h.in vendored
View File

@ -39,10 +39,9 @@
#include <ctime>
@ac_google_start_namespace@
#include <glog/log_severity.h>
#include <glog/logging.h>
#include <glog/vlog_is_on.h>
#include "glog/log_severity.h"
#include "glog/logging.h"
#include "glog/vlog_is_on.h"
#if defined(__GNUC__)
#pragma GCC diagnostic push

8
runtime/onnxruntime/third_party/glog/src/glog/vlog_is_on.h.in vendored
View File

@ -61,10 +61,10 @@
#ifndef BASE_VLOG_IS_ON_H_
#define BASE_VLOG_IS_ON_H_
#include <glog/log_severity.h>
#include <cstddef>
#include "glog/log_severity.h"
#if defined(__GNUC__)
// We emit an anonymous static int* variable at every VLOG_IS_ON(n) site.
// (Normally) the first time every VLOG_IS_ON(n) site is hit,
@ -75,6 +75,8 @@
#define VLOG_IS_ON(verboselevel) \
__extension__ \
({ static @ac_google_namespace@::SiteFlag vlocal__ = {NULL, NULL, 0, NULL}; \
GLOG_IFDEF_THREAD_SANITIZER( \
AnnotateBenignRaceSized(__FILE__, __LINE__, &vlocal__, sizeof(@ac_google_namespace@::SiteFlag), "")); \
@ac_google_namespace@::int32 verbose_level__ = (verboselevel); \
(vlocal__.level == NULL ? @ac_google_namespace@::InitVLOG3__(&vlocal__, &FLAGS_v, \
__FILE__, verbose_level__) : *vlocal__.level >= verbose_level__); \
@ -104,7 +106,7 @@ struct SiteFlag {
SiteFlag* next;
};
// Helper routine which determines the logging info for a particalur VLOG site.
// Helper routine which determines the logging info for a particular VLOG site.
// site_flag is the address of the site-local pointer to the controlling
// verbosity level
// site_default is the default to use for *site_flag

99
runtime/onnxruntime/third_party/glog/src/logging.cc vendored
View File

@ -33,8 +33,11 @@
#include <algorithm>
#include <cassert>
#include <cstddef>
#include <iomanip>
#include <iterator>
#include <string>
#ifdef HAVE_UNISTD_H
# include <unistd.h> // For _exit.
#endif
@ -56,18 +59,23 @@
#ifdef HAVE_SYSLOG_H
# include <syslog.h>
#endif
#ifdef HAVE__CHSIZE_S
#include <io.h> // for truncate log file
#endif
#include <vector>
#include <cerrno> // for errno
#include <sstream>
#include <regex>
#include <cctype> // for std::isspace
#ifdef GLOG_OS_WINDOWS
#include "windows/dirent.h"
#else
#include <dirent.h> // for automatic removal of old logs
#endif
#include "base/commandlineflags.h" // to get the program name
#include <glog/logging.h>
#include <glog/raw_logging.h>
#include "base/commandlineflags.h" // to get the program name
#include "base/googleinit.h"
#include "glog/logging.h"
#include "glog/raw_logging.h"
#ifdef HAVE_STACKTRACE
# include "stacktrace.h"
@ -1550,9 +1558,15 @@ static LogMessage::LogMessageData fatal_msg_data_shared;
// allocations).
static thread_local bool thread_data_available = true;
#if defined(__cpp_lib_byte) && __cpp_lib_byte >= 201603L
// std::aligned_storage is deprecated in C++23
alignas(LogMessage::LogMessageData) static thread_local std::byte
thread_msg_data[sizeof(LogMessage::LogMessageData)];
#else // !(defined(__cpp_lib_byte) && __cpp_lib_byte >= 201603L)
static thread_local std::aligned_storage<
sizeof(LogMessage::LogMessageData),
alignof(LogMessage::LogMessageData)>::type thread_msg_data;
#endif // defined(__cpp_lib_byte) && __cpp_lib_byte >= 201603L
#endif // defined(GLOG_THREAD_LOCAL_STORAGE)
LogMessage::LogMessageData::LogMessageData()
@ -1918,6 +1932,14 @@ void LogMessage::RecordCrashReason(
GLOG_EXPORT logging_fail_func_t g_logging_fail_func =
reinterpret_cast<logging_fail_func_t>(&abort);
NullStream::NullStream() : LogMessage::LogStream(message_buffer_, 1, 0) {}
NullStream::NullStream(const char* /*file*/, int /*line*/,
const CheckOpString& /*result*/)
: LogMessage::LogStream(message_buffer_, 1, 0) {}
NullStream& NullStream::stream() { return *this; }
NullStreamFatal::~NullStreamFatal() { _exit(EXIT_FAILURE); }
void InstallFailureFunction(logging_fail_func_t fail_func) {
g_logging_fail_func = fail_func;
}
@ -2205,6 +2227,13 @@ static string ShellEscape(const string& src) {
}
return result;
}
// Trim whitespace from both ends of the provided string.
static inline void trim(std::string &s) {
const auto toRemove = [](char ch) { return std::isspace(ch) == 0; };
s.erase(s.begin(), std::find_if(s.begin(), s.end(), toRemove));
s.erase(std::find_if(s.rbegin(), s.rend(), toRemove).base(), s.end());
}
#endif
// use_logging controls whether the logging functions LOG/VLOG are used
@ -2214,6 +2243,47 @@ static bool SendEmailInternal(const char*dest, const char *subject,
const char*body, bool use_logging) {
#ifndef GLOG_OS_EMSCRIPTEN
if (dest && *dest) {
// Split the comma-separated list of email addresses, validate each one and
// build a sanitized new comma-separated string without whitespace.
std::istringstream ss(dest);
std::ostringstream sanitized_dests;
std::string s;
while (std::getline(ss, s, ',')) {
trim(s);
if (s.empty()) {
continue;
}
// We validate the provided email addresses using the same regular
// expression that HTML5 uses[1], except that we require the address to
// start with an alpha-numeric character. This is because we don't want to
// allow email addresses that start with a special character, such as a
// pipe or dash, which could be misunderstood as a command-line flag by
// certain versions of `mail` that are vulnerable to command injection.[2]
// [1] https://html.spec.whatwg.org/multipage/input.html#valid-e-mail-address
// [2] e.g. https://nvd.nist.gov/vuln/detail/CVE-2004-2771
if (!std::regex_match(
s,
std::regex("^[a-zA-Z0-9]"
"[a-zA-Z0-9.!#$%&'*+/=?^_`{|}~-]*@[a-zA-Z0-9]"
"(?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\\.[a-zA-Z0-9]"
"(?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*$"))) {
if (use_logging) {
VLOG(1) << "Invalid destination email address:" << s;
} else {
fprintf(stderr, "Invalid destination email address: %s\n",
s.c_str());
}
return false;
}
if (!sanitized_dests.str().empty()) {
sanitized_dests << ",";
}
sanitized_dests << s;
}
// Avoid dangling reference
const std::string& tmp = sanitized_dests.str();
dest = tmp.c_str();
if ( use_logging ) {
VLOG(1) << "Trying to send TITLE:" << subject
<< " BODY:" << body << " to " << dest;
@ -2235,8 +2305,8 @@ static bool SendEmailInternal(const char*dest, const char *subject,
FILE* pipe = popen(cmd.c_str(), "w");
if (pipe != nullptr) {
// Add the body if we have one
if (body) {
// Add the body if we have one
if (body) {
fwrite(body, sizeof(char), strlen(body), pipe);
}
bool ok = pclose(pipe) != -1;
@ -2328,8 +2398,13 @@ const vector<string>& GetLoggingDirectories() {
logging_directories_list = new vector<string>;
if ( !FLAGS_log_dir.empty() ) {
// A dir was specified, we should use it
logging_directories_list->push_back(FLAGS_log_dir);
// Ensure the specified path ends with a directory delimiter.
if (std::find(std::begin(possible_dir_delim), std::end(possible_dir_delim),
FLAGS_log_dir.back()) == std::end(possible_dir_delim)) {
logging_directories_list->push_back(FLAGS_log_dir + "/");
} else {
logging_directories_list->push_back(FLAGS_log_dir);
}
} else {
GetTempDirectories(logging_directories_list);
#ifdef GLOG_OS_WINDOWS
@ -2367,7 +2442,7 @@ void GetExistingTempDirectories(vector<string>* list) {
}
void TruncateLogFile(const char *path, uint64 limit, uint64 keep) {
#ifdef HAVE_UNISTD_H
#if defined(HAVE_UNISTD_H) || defined(HAVE__CHSIZE_S)
struct stat statbuf;
const int kCopyBlockSize = 8 << 10;
char copybuf[kCopyBlockSize];
@ -2388,7 +2463,11 @@ void TruncateLogFile(const char *path, uint64 limit, uint64 keep) {
// all of base/...) with -D_FILE_OFFSET_BITS=64 but that's
// rather scary.
// Instead just truncate the file to something we can manage
#ifdef HAVE__CHSIZE_S
if (_chsize_s(fd, 0) != 0) {
#else
if (truncate(path, 0) == -1) {
#endif
PLOG(ERROR) << "Unable to truncate " << path;
} else {
LOG(ERROR) << "Truncated " << path << " due to EFBIG error";
@ -2433,7 +2512,11 @@ void TruncateLogFile(const char *path, uint64 limit, uint64 keep) {
// Truncate the remainder of the file. If someone else writes to the
// end of the file after our last read() above, we lose their latest
// data. Too bad ...
#ifdef HAVE__CHSIZE_S
if (_chsize_s(fd, write_offset) != 0) {
#else
if (ftruncate(fd, write_offset) == -1) {
#endif
PLOG(ERROR) << "Unable to truncate " << path;
}

7
runtime/onnxruntime/third_party/glog/src/mock-log.h vendored
View File

@ -36,13 +36,12 @@
#define GLOG_SRC_MOCK_LOG_H_
// For GOOGLE_NAMESPACE. This must go first so we get _XOPEN_SOURCE.
#include "utilities.h"
#include <gmock/gmock.h>
#include <string>
#include <gmock/gmock.h>
#include <glog/logging.h>
#include "glog/logging.h"
#include "utilities.h"
_START_GOOGLE_NAMESPACE_
namespace glog_testing {

13
runtime/onnxruntime/third_party/glog/src/raw_logging.cc vendored
View File

@ -39,12 +39,14 @@
#ifdef HAVE_UNISTD_H
# include <unistd.h> // for close() and write()
#endif
#include <fcntl.h> // for open()
#include <fcntl.h> // for open()
#include <ctime>
#include "config.h"
#include <glog/logging.h> // To pick up flag settings etc.
#include <glog/raw_logging.h>
#include "base/commandlineflags.h"
#include "config.h"
#include "glog/logging.h" // To pick up flag settings etc.
#include "glog/raw_logging.h"
#ifdef HAVE_STACKTRACE
# include "stacktrace.h"
@ -60,7 +62,8 @@
#endif
#if (defined(HAVE_SYSCALL_H) || defined(HAVE_SYS_SYSCALL_H)) && \
(!(defined(GLOG_OS_MACOSX))) && !defined(GLOG_OS_EMSCRIPTEN)
(!(defined(GLOG_OS_MACOSX)) && !(defined(GLOG_OS_OPENBSD))) && \
!defined(GLOG_OS_EMSCRIPTEN)
#define safe_write(fd, s, len) syscall(SYS_write, fd, s, len)
#else
// Not so safe, but what can you do?

10
runtime/onnxruntime/third_party/glog/src/signalhandler.cc vendored
View File

@ -31,13 +31,13 @@
//
// Implementation of InstallFailureSignalHandler().
#include "utilities.h"
#include "stacktrace.h"
#include "symbolize.h"
#include <glog/logging.h>
#include <csignal>
#include <ctime>
#include "glog/logging.h"
#include "stacktrace.h"
#include "symbolize.h"
#include "utilities.h"
#ifdef HAVE_UCONTEXT_H
# include <ucontext.h>
#endif

2
runtime/onnxruntime/third_party/glog/src/stacktrace.h vendored
View File

@ -34,7 +34,7 @@
#define BASE_STACKTRACE_H_
#include "config.h"
#include <glog/logging.h>
#include "glog/logging.h"
_START_GOOGLE_NAMESPACE_

2
runtime/onnxruntime/third_party/glog/src/stacktrace_libunwind-inl.h vendored
View File

@ -37,7 +37,7 @@ extern "C" {
#define UNW_LOCAL_ONLY
#include <libunwind.h>
}
#include <glog/raw_logging.h>
#include "glog/raw_logging.h"
#include "stacktrace.h"
_START_GOOGLE_NAMESPACE_

2
runtime/onnxruntime/third_party/glog/src/stacktrace_powerpc-inl.h vendored
View File

@ -121,7 +121,7 @@ int GetStackTrace(void** result, int max_depth, int skip_count) {
// This check is in case the compiler doesn't define _CALL_SYSV.
result[n++] = *(sp+1);
#else
#error Need to specify the PPC ABI for your archiecture.
#error Need to specify the PPC ABI for your architecture.
#endif
}
// Use strict unwinding rules.

9
runtime/onnxruntime/third_party/glog/src/stacktrace_unittest.cc vendored
View File

@ -27,14 +27,15 @@
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#include "utilities.h"
#include "stacktrace.h"
#include <cstdio>
#include <cstdlib>
#include "config.h"
#include "base/commandlineflags.h"
#include <glog/logging.h>
#include "stacktrace.h"
#include "config.h"
#include "glog/logging.h"
#include "utilities.h"
#ifdef HAVE_EXECINFO_BACKTRACE_SYMBOLS
# include <execinfo.h>

4
runtime/onnxruntime/third_party/glog/src/symbolize.cc vendored
View File

@ -119,7 +119,6 @@ _END_GOOGLE_NAMESPACE_
#include <elf.h>
#endif
#include <fcntl.h>
#include <glog/raw_logging.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>
@ -133,6 +132,7 @@ _END_GOOGLE_NAMESPACE_
#include <cstring>
#include "config.h"
#include "glog/raw_logging.h"
#include "symbolize.h"
// Re-runs fn until it doesn't cause EINTR.
@ -535,10 +535,8 @@ OpenObjectFileContainingPcAndGetStartAddress(uint64_t pc,
// Iterate over maps and look for the map containing the pc. Then
// look into the symbol tables inside.
char buf[1024]; // Big enough for line of sane /proc/self/maps
unsigned num_maps = 0;
LineReader reader(wrapped_maps_fd.get(), buf, sizeof(buf), 0);
while (true) {
num_maps++;
const char *cursor;
const char *eol;
if (!reader.ReadLine(&cursor, &eol)) { // EOF or malformed line.

4
runtime/onnxruntime/third_party/glog/src/symbolize.h vendored
View File

@ -54,9 +54,9 @@
#ifndef BASE_SYMBOLIZE_H_
#define BASE_SYMBOLIZE_H_
#include "utilities.h"
#include "config.h"
#include <glog/logging.h>
#include "glog/logging.h"
#include "utilities.h"
#ifdef HAVE_SYMBOLIZE

2
runtime/onnxruntime/third_party/glog/src/utilities.cc vendored
View File

@ -292,6 +292,8 @@ pid_t GetTID() {
return getpid(); // Linux: getpid returns thread ID when gettid is absent
#elif defined GLOG_OS_WINDOWS && !defined GLOG_OS_CYGWIN
return static_cast<pid_t>(GetCurrentThreadId());
#elif defined GLOG_OS_OPENBSD
return getthrid();
#elif defined(HAVE_PTHREAD)
// If none of the techniques above worked, we use pthread_self().
return (pid_t)(uintptr_t)pthread_self();

9
runtime/onnxruntime/third_party/glog/src/utilities.h vendored
View File

@ -29,7 +29,7 @@
//
// Author: Shinichiro Hamaji
//
// Define utilties for glog internal usage.
// Define utilities for glog internal usage.
#ifndef UTILITIES_H__
#define UTILITIES_H__
@ -52,11 +52,10 @@
#define PRIXS __PRIS_PREFIX "X"
#define PRIoS __PRIS_PREFIX "o"
#include "base/mutex.h" // This must go first so we get _XOPEN_SOURCE
#include <string>
#include <glog/logging.h>
#include "base/mutex.h" // This must go first so we get _XOPEN_SOURCE
#include "glog/logging.h"
#if defined(GLOG_OS_WINDOWS)
# include "port.h"
@ -174,7 +173,7 @@ inline T sync_val_compare_and_swap(T* ptr, T oldval, T newval) {
:"=a"(ret)
// GCC may produces %sil or %dil for
// constraint "r", but some of apple's gas
// dosn't know the 8 bit registers.
// doesn't know the 8 bit registers.
// We use "q" to avoid these registers.
:"q"(newval), "q"(ptr), "a"(oldval)
:"memory", "cc");

12
runtime/onnxruntime/third_party/glog/src/vlog_is_on.cc vendored
View File

@ -32,17 +32,17 @@
// Broken out from logging.cc by Soren Lassen
// logging_unittest.cc covers the functionality herein
#include "utilities.h"
#include <cstring>
#include <cstdlib>
#include <cerrno>
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <string>
#include "base/commandlineflags.h"
#include <glog/logging.h>
#include <glog/raw_logging.h>
#include "base/googleinit.h"
#include "glog/logging.h"
#include "glog/raw_logging.h"
#include "utilities.h"
// glog doesn't have annotation
#define ANNOTATE_BENIGN_RACE(address, description)

1145
runtime/onnxruntime/third_party/glog/src/windows/dirent.h vendored Normal file
View File

File diff suppressed because it is too large Load Diff

75
runtime/onnxruntime/third_party/glog/src/windows/port.cc vendored Normal file
View File

@ -0,0 +1,75 @@
/* Copyright (c) 2008, Google Inc.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are
* met:
*
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following disclaimer
* in the documentation and/or other materials provided with the
* distribution.
* * Neither the name of Google Inc. nor the names of its
* contributors may be used to endorse or promote products derived from
* this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*
* ---
* Author: Craig Silverstein
* Copied from google-perftools and modified by Shinichiro Hamaji
*/
#ifndef _WIN32
# error You should only be including windows/port.cc in a windows environment!
#endif
#include "port.h"
#include <cstdarg> // for va_list, va_start, va_end
#include <ctime>
#include "config.h"
// These call the windows _vsnprintf, but always NUL-terminate.
int safe_vsnprintf(char* str, std::size_t size, const char* format,
va_list ap) {
if (size == 0) // not even room for a \0?
return -1; // not what C99 says to do, but what windows does
str[size-1] = '\0';
return _vsnprintf(str, size-1, format, ap);
}
#ifndef HAVE_LOCALTIME_R
struct tm* localtime_r(const std::time_t* timep, std::tm* result) {
localtime_s(result, timep);
return result;
}
#endif // not HAVE_LOCALTIME_R
#ifndef HAVE_GMTIME_R
struct tm* gmtime_r(const std::time_t* timep, std::tm* result) {
gmtime_s(result, timep);
return result;
}
#endif // not HAVE_GMTIME_R
#ifndef HAVE_SNPRINTF
int snprintf(char *str, size_t size, const char *format, ...) {
va_list ap;
va_start(ap, format);
const int r = vsnprintf(str, size, format, ap);
va_end(ap);
return r;
}
#endif

181
runtime/onnxruntime/third_party/glog/src/windows/port.h vendored Normal file
View File

@ -0,0 +1,181 @@
/* Copyright (c) 2023, Google Inc.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are
* met:
*
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following disclaimer
* in the documentation and/or other materials provided with the
* distribution.
* * Neither the name of Google Inc. nor the names of its
* contributors may be used to endorse or promote products derived from
* this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*
* ---
* Author: Craig Silverstein
* Copied from google-perftools and modified by Shinichiro Hamaji
*
* These are some portability typedefs and defines to make it a bit
* easier to compile this code under VC++.
*
* Several of these are taken from glib:
* http://developer.gnome.org/doc/API/glib/glib-windows-compatability-functions.html
*/
#ifndef CTEMPLATE_WINDOWS_PORT_H_
#define CTEMPLATE_WINDOWS_PORT_H_
#include "config.h"
#ifdef _WIN32
#ifndef WIN32_LEAN_AND_MEAN
#define WIN32_LEAN_AND_MEAN /* We always want minimal includes */
#endif
#include <windows.h>
#include <winsock.h> /* for gethostname */
#include <io.h> /* because we so often use open/close/etc */
#include <direct.h> /* for _getcwd() */
#include <process.h> /* for _getpid() */
#include <cstdarg> /* template_dictionary.cc uses va_copy */
#include <cstdio> /* read in vsnprintf decl. before redifining it */
#include <cstring> /* for _strnicmp(), strerror_s() */
#include <ctime> /* for localtime_s() */
/* Note: the C++ #includes are all together at the bottom. This file is
* used by both C and C++ code, so we put all the C++ together.
*/
#include "glog/logging.h"
#ifdef _MSC_VER
/* 4244: otherwise we get problems when substracting two size_t's to an int
* 4251: it's complaining about a private struct I've chosen not to dllexport
* 4355: we use this in a constructor, but we do it safely
* 4715: for some reason VC++ stopped realizing you can't return after abort()
* 4800: we know we're casting ints/char*'s to bools, and we're ok with that
* 4996: Yes, we're ok using "unsafe" functions like fopen() and strerror()
* 4312: Converting uint32_t to a pointer when testing %p
* 4267: also subtracting two size_t to int
* 4722: Destructor never returns due to abort()
*/
#pragma warning(disable:4244 4251 4355 4715 4800 4996 4267 4312 4722)
/* file I/O */
#define PATH_MAX 1024
#define access _access
#define getcwd _getcwd
#define open _open
#define read _read
#define write(fd, p, n) _write(fd, p, n)
#define lseek _lseek
#define close _close
#define popen _popen
#define pclose _pclose
#define R_OK 04 /* read-only (for access()) */
#define S_ISDIR(m) (((m) & _S_IFMT) == _S_IFDIR)
#define O_WRONLY _O_WRONLY
#define O_CREAT _O_CREAT
#define O_EXCL _O_EXCL
#ifndef __MINGW32__
enum { STDIN_FILENO = 0, STDOUT_FILENO = 1, STDERR_FILENO = 2 };
#endif
#define S_IRUSR S_IREAD
#define S_IWUSR S_IWRITE
/* Not quite as lightweight as a hard-link, but more than good enough for us. */
#define link(oldpath, newpath) CopyFileA(oldpath, newpath, false)
#define strcasecmp _stricmp
#define strncasecmp _strnicmp
/* In windows-land, hash<> is called hash_compare<> (from xhash.h) */
/* VC11 provides std::hash */
#if defined(_MSC_VER) && (_MSC_VER < 1700)
#define hash hash_compare
#endif
/* Sleep is in ms, on windows */
#define sleep(secs) Sleep((secs) * 1000)
/* We can't just use _vsnprintf and _snprintf as drop-in-replacements,
* because they don't always NUL-terminate. :-( We also can't use the
* name vsnprintf, since windows defines that (but not snprintf (!)).
*/
#ifndef HAVE_SNPRINTF
extern int GLOG_EXPORT snprintf(char* str, size_t size, const char* format,
...);
#endif
extern int GLOG_EXPORT safe_vsnprintf(char* str, size_t size,
const char* format, va_list ap);
#define vsnprintf(str, size, format, ap) safe_vsnprintf(str, size, format, ap)
#ifndef va_copy
#define va_copy(dst, src) (dst) = (src)
#endif
/* Windows doesn't support specifying the number of buckets as a
* hash_map constructor arg, so we leave this blank.
*/
#define CTEMPLATE_SMALL_HASHTABLE
#define DEFAULT_TEMPLATE_ROOTDIR ".."
// ----------------------------------- SYSTEM/PROCESS
typedef int pid_t;
#define getpid _getpid
#endif // _MSC_VER
// ----------------------------------- THREADS
#if defined(HAVE_PTHREAD)
# include <pthread.h>
#else // no PTHREAD
typedef DWORD pthread_t;
typedef DWORD pthread_key_t;
typedef LONG pthread_once_t;
enum { PTHREAD_ONCE_INIT = 0 }; // important that this be 0! for SpinLock
#define pthread_self GetCurrentThreadId
#define pthread_equal(pthread_t_1, pthread_t_2) ((pthread_t_1)==(pthread_t_2))
#endif // HAVE_PTHREAD
#ifndef HAVE_LOCALTIME_R
extern GLOG_EXPORT std::tm* localtime_r(const std::time_t* timep,
std::tm* result);
#endif // not HAVE_LOCALTIME_R
#ifndef HAVE_GMTIME_R
extern GLOG_EXPORT std::tm* gmtime_r(const std::time_t* timep, std::tm* result);
#endif // not HAVE_GMTIME_R
inline char* strerror_r(int errnum, char* buf, std::size_t buflen) {
strerror_s(buf, buflen, errnum);
return buf;
}
#ifndef __cplusplus
/* I don't see how to get inlining for C code in MSVC. Ah well. */
#define inline
#endif
#endif /* _WIN32 */
#endif /* CTEMPLATE_WINDOWS_PORT_H_ */

2
runtime/onnxruntime/third_party/jieba/include/limonp/Logging.hpp vendored
View File

@ -50,7 +50,7 @@ class Logger {
#if defined(_WIN32) || defined(_WIN64)
errno_t e = localtime_s(&tmNow, &timeNow);
assert(e = 0);
assert(e == 0);
#else
struct tm * tm_tmp = localtime_r(&timeNow, &tmNow);
assert(tm_tmp != nullptr);

26
runtime/onnxruntime/third_party/kaldi/CMakeLists.txt vendored
View File

@ -2,7 +2,7 @@ cmake_minimum_required(VERSION 3.10 FATAL_ERROR)
project(kaldi)
add_library(kaldi-util
add_library(kaldi-util STATIC
base/kaldi-error.cc
base/kaldi-math.cc
util/kaldi-io.cc
@ -12,13 +12,23 @@ add_library(kaldi-util
)
#target_link_libraries(kaldi-util PUBLIC utils)
add_library(kaldi-decoder
add_library(kaldi-decoder STATIC
lat/determinize-lattice-pruned.cc
lat/lattice-functions.cc
decoder/lattice-faster-decoder.cc
decoder/lattice-faster-online-decoder.cc
)
if (WIN32)
target_link_libraries(kaldi-decoder PUBLIC kaldi-util)
else()
target_link_libraries(kaldi-decoder PUBLIC kaldi-util dl)
endif (WIN32)
if (WIN32)
target_compile_definitions (kaldi-decoder PUBLIC GLOG_NO_ABBREVIATED_SEVERITIES)
endif (WIN32)
include_directories(${CMAKE_SOURCE_DIR}/build/third_party/glog)
include_directories(${CMAKE_SOURCE_DIR}/third_party/glog/src)
@ -30,7 +40,13 @@ if(TRUE)
lm/arpa-lm-compiler.cc
lmbin/arpa2fst.cc
)
target_link_libraries(arpa2fst PUBLIC kaldi-util fst dl)
if (WIN32)
target_link_libraries(arpa2fst PUBLIC kaldi-util fst)
else()
target_link_libraries(arpa2fst PUBLIC kaldi-util fst dl)
endif (WIN32)
# FST tools binary
set(FST_BINS
@ -46,6 +62,10 @@ if(TRUE)
fstbin/${name}.cc
fstext/kaldi-fst-io.cc
)
if (WIN32)
target_link_libraries(${name} PUBLIC kaldi-util fst)
else()
target_link_libraries(${name} PUBLIC kaldi-util fst dl)
endif (WIN32)
endforeach()
endif()

2
runtime/onnxruntime/third_party/openfst/src/include/fst/log.h vendored
View File

@ -27,7 +27,9 @@
using std::string;
#ifndef _WIN32
DECLARE_int32(v);
#endif
class LogMessage {
public:

2
runtime/onnxruntime/third_party/openfst/src/lib/CMakeLists.txt vendored
View File

@ -1,7 +1,7 @@
FILE(GLOB HEADER_FILES ../include/fst/*.h)
add_library(fst
add_library(fst STATIC
compat.cc
flags.cc
fst-types.cc

2
runtime/onnxruntime/third_party/yaml-cpp/include/yaml-cpp/exceptions.h vendored
View File

@ -16,7 +16,7 @@
// This is here for compatibility with older versions of Visual Studio
// which don't support noexcept
#ifdef _MSC_VER
#define YAML_CPP_NOEXCEPT _NOEXCEPT
#define YAML_CPP_NOEXCEPT noexcept
#else
#define YAML_CPP_NOEXCEPT noexcept
#endif

2
runtime/onnxruntime/third_party/yaml-cpp/src/exceptions.cpp vendored
View File

@ -3,7 +3,7 @@
// This is here for compatibility with older versions of Visual Studio
// which don't support noexcept
#ifdef _MSC_VER
#define YAML_CPP_NOEXCEPT _NOEXCEPT
#define YAML_CPP_NOEXCEPT noexcept
#else
#define YAML_CPP_NOEXCEPT noexcept
#endif

9
runtime/websocket/CMakeLists.txt
View File

@ -9,7 +9,14 @@ set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/bin)
option(ENABLE_WEBSOCKET "Whether to build websocket server" ON)
option(ENABLE_PORTAUDIO "Whether to build portaudio" ON)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -pthread -fPIC")
if(WIN32)
else()
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -pthread -fPIC")
endif()
option(ENABLE_GLOG "Whether to build glog" ON)
option(ENABLE_FST "Whether to build openfst" ON) # ITN need openfst compiled
option(BUILD_SHARED_LIBS "Build shared libraries" ON)

33
runtime/websocket/bin/CMakeLists.txt
View File

@ -1,10 +1,27 @@
add_executable(funasr-wss-server "funasr-wss-server.cpp" "websocket-server.cpp")
add_executable(funasr-wss-server-2pass "funasr-wss-server-2pass.cpp" "websocket-server-2pass.cpp")
add_executable(funasr-wss-client "funasr-wss-client.cpp")
add_executable(funasr-wss-client-2pass "funasr-wss-client-2pass.cpp" "microphone.cpp")
target_link_libraries(funasr-wss-client PUBLIC funasr ssl crypto)
target_link_libraries(funasr-wss-client-2pass PUBLIC funasr ssl crypto portaudio)
target_link_libraries(funasr-wss-server PUBLIC funasr ssl crypto)
target_link_libraries(funasr-wss-server-2pass PUBLIC funasr ssl crypto)
if(WIN32)
include_directories(${ONNXRUNTIME_DIR}/include)
include_directories(${FFMPEG_DIR}/include)
include_directories(${OPENSSL_ROOT_DIR}//include)
link_directories(${OPENSSL_ROOT_DIR}/lib)
add_definitions(-D_WEBSOCKETPP_CPP11_RANDOM_DEVICE_)
add_definitions(-D_WEBSOCKETPP_CPP11_TYPE_TRAITS_)
add_compile_options("$<$<CXX_COMPILER_ID:MSVC>:/bigobj>")
add_compile_options("$<$<CXX_COMPILER_ID:MSVC>:/utf-8>")
SET(RELATION_SOURCE "../../onnxruntime/src/audio.cpp" "../../onnxruntime/src/resample.cpp" "../../onnxruntime/src/util.cpp" "../../onnxruntime/src/alignedmem.cpp" "../../onnxruntime/src/encode_converter.cpp")
endif()
add_executable(funasr-wss-server "funasr-wss-server.cpp" "websocket-server.cpp" ${RELATION_SOURCE})
add_executable(funasr-wss-server-2pass "funasr-wss-server-2pass.cpp" "websocket-server-2pass.cpp" ${RELATION_SOURCE})
add_executable(funasr-wss-client "funasr-wss-client.cpp" ${RELATION_SOURCE})
add_executable(funasr-wss-client-2pass "funasr-wss-client-2pass.cpp" "microphone.cpp" ${RELATION_SOURCE})
target_link_libraries(funasr-wss-client PUBLIC funasr ${OPENSSL_CRYPTO_LIBRARY} ${OPENSSL_SSL_LIBRARY})
target_link_libraries(funasr-wss-client-2pass PUBLIC funasr ${OPENSSL_CRYPTO_LIBRARY} ${OPENSSL_SSL_LIBRARY} portaudio)
target_link_libraries(funasr-wss-server PUBLIC funasr ${OPENSSL_CRYPTO_LIBRARY} ${OPENSSL_SSL_LIBRARY})
target_link_libraries(funasr-wss-server-2pass PUBLIC funasr ${OPENSSL_CRYPTO_LIBRARY} ${OPENSSL_SSL_LIBRARY})

7
runtime/websocket/bin/funasr-wss-server-2pass.cpp
View File

@ -11,8 +11,13 @@
// <string>] [--punc-quant <string>] [--punc-dir <string>]
// [--vad-quant <string>] [--vad-dir <string>] [--quantize
// <string>] --model-dir <string> [--] [--version] [-h]
#include <unistd.h>
#include "websocket-server-2pass.h"
#ifdef _WIN32
#include "win_func.h"
#else
#include <unistd.h>
#endif
#include <fstream>
#include "util.h"

4
runtime/websocket/bin/funasr-wss-server.cpp
View File

@ -11,7 +11,11 @@
// [--vad-quant <string>] [--vad-dir <string>] [--quantize
// <string>] --model-dir <string> [--] [--version] [-h]
#include "websocket-server.h"
#ifdef _WIN32
#include "win_func.h"
#else
#include <unistd.h>
#endif
#include <fstream>
#include "util.h"

47
runtime/websocket/readme.md
View File

@ -6,14 +6,14 @@
## Building for Linux/Unix
### Download onnxruntime
```shell
wget https://github.com/microsoft/onnxruntime/releases/download/v1.14.0/onnxruntime-linux-x64-1.14.0.tgz
wget https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/dep_libs/onnxruntime-linux-x64-1.14.0.tgz
tar -zxvf onnxruntime-linux-x64-1.14.0.tgz
```
### Download ffmpeg
```shell
wget https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/dep_libs/ffmpeg-N-111383-g20b8688092-linux64-gpl-shared.tar.xz
tar -xvf ffmpeg-N-111383-g20b8688092-linux64-gpl-shared.tar.xz
wget https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/dep_libs/ffmpeg-master-latest-linux64-gpl-shared.tar.xz
tar -xvf ffmpeg-master-latest-linux64-gpl-shared.tar.xz
```
### Install deps
@ -31,6 +31,45 @@ apt-get install libssl-dev #ubuntu
```shell
git clone https://github.com/alibaba-damo-academy/FunASR.git && cd FunASR/runtime/websocket
mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=release .. -DONNXRUNTIME_DIR=/path/to/onnxruntime-linux-x64-1.14.0 -DFFMPEG_DIR=/path/to/ffmpeg-N-111383-g20b8688092-linux64-gpl-shared
cmake -DCMAKE_BUILD_TYPE=release .. -DONNXRUNTIME_DIR=/path/to/onnxruntime-linux-x64-1.14.0 -DFFMPEG_DIR=/path/to/ffmpeg-master-latest-linux64-gpl-shared
make -j 4
```
## Building for Windows
### Download onnxruntime
https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/dep_libs/onnxruntime-win-x64-1.16.1.zip
Download to d:\ffmpeg-master-latest-win64-gpl-shared
### Download ffmpeg
https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/dep_libs/ffmpeg-master-latest-win64-gpl-shared.zip
Download to d:\onnxruntime-win-x64-1.16.1
### Download openssl
https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/dep_libs/openssl-1.1.1w.tar.gz
Download to d:/src/openssl-1.1.1w
```
# download and install perl: https://www.activestate.com/products/activeperl/downloads/
d:
cd d:/src/openssl-1.1.1w
perl Configure VC-WIN64A --prefix=d:/openssl-1.1.1w
# Open x64 Native Tools Command Prompt and execute the following compilation steps
nmake
nmake install
```
### Build runtime
```
git clone https://github.com/alibaba-damo-academy/FunASR.git
cd FunASR/runtime/websocket
mkdir build
cd build
cmake ../ -D OPENSSL_ROOT_DIR=d:/openssl-1.1.1w -D FFMPEG_DIR=d:/ffmpeg-master-latest-win64-gpl-shared -D ONNXRUNTIME_DIR=d:/onnxruntime-win-x64-1.16.1
```
Open FunASR/runtime/websocket/build/FunASRWebscoket.sln in Visual Studio and complete the compilation.

49
runtime/websocket/readme_zh.md
View File

@ -6,14 +6,14 @@
## Linux/Unix 平台编译
### 下载 onnxruntime
```shell
wget https://github.com/microsoft/onnxruntime/releases/download/v1.14.0/onnxruntime-linux-x64-1.14.0.tgz
wget https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/dep_libs/onnxruntime-linux-x64-1.14.0.tgz
tar -zxvf onnxruntime-linux-x64-1.14.0.tgz
```
### 下载 ffmpeg
```shell
wget https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/dep_libs/ffmpeg-N-111383-g20b8688092-linux64-gpl-shared.tar.xz
tar -xvf ffmpeg-N-111383-g20b8688092-linux64-gpl-shared.tar.xz
wget https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/dep_libs/ffmpeg-master-latest-linux64-gpl-shared.tar.xz
tar -xvf ffmpeg-master-latest-linux64-gpl-shared.tar.xz
```
### 安装依赖
@ -32,6 +32,45 @@ apt-get install libssl-dev #ubuntu
```shell
git clone https://github.com/alibaba-damo-academy/FunASR.git && cd FunASR/runtime/websocket
mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=release .. -DONNXRUNTIME_DIR=/path/to/onnxruntime-linux-x64-1.14.0 -DFFMPEG_DIR=/path/to/ffmpeg-N-111383-g20b8688092-linux64-gpl-shared
cmake -DCMAKE_BUILD_TYPE=release .. -DONNXRUNTIME_DIR=/path/to/onnxruntime-linux-x64-1.14.0 -DFFMPEG_DIR=/path/to/ffmpeg-master-latest-linux64-gpl-shared
make -j 4
```
```
## Windows 平台编译
### 下载 onnxruntime
https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/dep_libs/onnxruntime-win-x64-1.16.1.zip
下载并解压到 d:\ffmpeg-master-latest-win64-gpl-shared
### 下载 ffmpeg
https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/dep_libs/ffmpeg-master-latest-win64-gpl-shared.zip
下载并解压到 d:\onnxruntime-win-x64-1.16.1
### 编译 openssl
https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/dep_libs/openssl-1.1.1w.tar.gz
下载解压到 d:/src/openssl-1.1.1w
```
# 下载&安装 perl: https://www.activestate.com/products/activeperl/downloads/
d:
cd d:/src/openssl-1.1.1w
perl Configure VC-WIN64A --prefix=d:/openssl-1.1.1w
# 管理员身份打开 x64 Native Tools Command Prompt 执行以下编译步骤
nmake
nmake install
```
### 编译 runtime
```
git clone https://github.com/alibaba-damo-academy/FunASR.git
cd FunASR/runtime/websocket
mkdir build
cd build
cmake ../ -D OPENSSL_ROOT_DIR=d:/openssl-1.1.1w -D FFMPEG_DIR=d:/ffmpeg-master-latest-win64-gpl-shared -D ONNXRUNTIME_DIR=d:/onnxruntime-win-x64-1.16.1
```
Visual Studio 打开 FunASR/runtime/websocket/build/FunASRWebscoket.sln 完成编译