PaddlePaddle3.2.2运行报错illegal instruction

[CatMemeGangster]

问题描述

uv环境下根据官方部署CPU版paddlepaddle和paddlex运行正常，但调试开发脚本时修改batch_size和pool_size后import paddle时报错illegal instruction

/var/log/message系统日志显示Nov 27 13:57:06 localhost kernel: [92913.410635] traps: python[255349] trap invalid opcode ip:7f0853dd1ae4 sp:7fff655f4740 error:0 in libphi_core.so[7f084f3ce000+56fe000]

操作系统：Kylin Linux Advanced Server V10 (Halberd)
CPU：Intel(R) Xeon(R) CPU E5-2620 v4
标记：fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc cpuid pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx hypervisor lahf_lm pti ssbd ibrs ibpb stibp tsc_adjust arat md_clear flush_l1d arch_capabilities

[passerbya]

我也遇到了这个问题，我的CPU是E5-2667 v2，有 AVX、无 AVX2，报非法指令

[passerbya]

自己编译又遇到了：段错误，以下是gdb信息
(base) [root@localhost pybind]# gdb ./eager_generator
GNU gdb (GDB) Red Hat Enterprise Linux 9.2-10.el7
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./eager_generator...
(No debugging symbols found in ./eager_generator)
(gdb) bt
No stack.
(gdb) run
Starting program: /usr/local/data/install/Paddle/build/paddle/fluid/pybind/eager_generator
Missing separate debuginfos, use: debuginfo-install glibc-2.17-326.el7_9.3.x86_64
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7fff0e13b700 (LWP 22950)]
[New Thread 0x7fff0d93a700 (LWP 22951)]
[New Thread 0x7fff0d139700 (LWP 22952)]
[New Thread 0x7fff0c938700 (LWP 22953)]
[New Thread 0x7fff0c137700 (LWP 22954)]
[New Thread 0x7fff0b936700 (LWP 22955)]
[New Thread 0x7fff0b135700 (LWP 22956)]
[New Thread 0x7fff0a934700 (LWP 22957)]
[New Thread 0x7fff0a133700 (LWP 22958)]
[New Thread 0x7fff09932700 (LWP 22959)]
[New Thread 0x7fff09131700 (LWP 22960)]
[New Thread 0x7fff08930700 (LWP 22961)]
[New Thread 0x7fff0812f700 (LWP 22962)]
[New Thread 0x7fff0792e700 (LWP 22963)]
[New Thread 0x7fff0712d700 (LWP 22964)]
[New Thread 0x7fff0692c700 (LWP 22965)]
[New Thread 0x7fff0612b700 (LWP 22966)]
[New Thread 0x7fff0592a700 (LWP 22967)]
[New Thread 0x7fff05129700 (LWP 22968)]
[New Thread 0x7fff04928700 (LWP 22969)]
[New Thread 0x7fff04127700 (LWP 22970)]
[New Thread 0x7fff03926700 (LWP 22971)]
[New Thread 0x7fff03125700 (LWP 22972)]
argc must be 2 or 3

Thread 1 "eager_generator" received signal SIGSEGV, Segmentation fault.
0x00007fffce6054eb in malloc_consolidate () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install libgcc-4.8.5-44.el7.x86_64
(gdb) bt
#0 0x00007fffce6054eb in malloc_consolidate () from /lib64/libc.so.6
#1 0x00007fffce607135 in _int_malloc () from /lib64/libc.so.6
#2 0x00007fffce60b214 in calloc () from /lib64/libc.so.6
#3 0x00007fffcf276723 in ?? () from /usr/local/cuda/lib64/libcudart.so.11.0
#4 0x00007fffcf277e2b in ?? () from /usr/local/cuda/lib64/libcudart.so.11.0
#5 0x00007fffce5bf05a in __cxa_finalize () from /lib64/libc.so.6
#6 0x00007fffe8da80f3 in __do_global_dtors_aux () from /usr/local/install/Paddle/build/paddle/phi/libphi_gpu.so
#7 0x00007fffffffdc20 in ?? ()
#8 0x00007ffff7deb08a in _dl_fini () from /lib64/ld-linux-x86-64.so.2
Backtrace stopped: frame did not save the PC

[fangfangssj]

在运行前可以使用打印一些log信息，可以给一下log嘛
FLAGS_call_stack_level=2
GLOG_v=4

[passerbya]

(paddleocr3) [root@localhost build]# export FLAGS_call_stack_level=2
(paddleocr3) [root@localhost build]# export GLOG_v=4
(paddleocr3) [root@localhost build]# paddleocr ocr -i /root/1.jpg \

--use_doc_orientation_classify False
--use_doc_unwarping False
--use_textline_orientation False
非法指令

没有日志打印，应该是在so文件中已经崩溃了

[passerbya]

不过我搞定编译Paddle3.2.2了
1）先安装centos7自带的gcc10，然后再编译一个gcc10.2.0到/usr/local/gcc10，只升级glibc到2.27以上是不行的

先确保 scl 源是有的（如果之前没装过）

yum install -y centos-release-scl
yum install -y devtoolset-10-gcc devtoolset-10-gcc-c++

然后装 devtoolset-10 的整套工具链（包含运行时）

yum install -y devtoolset-10-toolchain

scl --list
scl enable devtoolset-10 bash
gcc --version
g++ --version

wget http://ftp.gnu.org/gnu/gcc/gcc-10.2.0/gcc-10.2.0.tar.xz
tar -Jxvf gcc-10.2.0.tar.xz
#进入gcc目录安装
cd gcc-10.2.0
./contrib/download_prerequisites
#编译
mkdir build
cd build

export CC=/opt/rh/devtoolset-10/root/usr/bin/gcc
export CXX=/opt/rh/devtoolset-10/root/usr/bin/g++

../configure --prefix=/usr/local/gcc10 --enable-checking=release --enable-languages=c,c++ --disable-multilib
make -j$(nproc)
make install

sln /usr/local/gcc10/bin/gcc /usr/local/gcc10/bin/cc
chmod +x /usr/local/gcc10/lib64/libgcc_s.so.1
sln /usr/local/gcc10/lib64/libgcc_s.so.1 /lib64/libgcc_s.so.1

2）安装paddle conda
conda create -n paddleocr3 python==3.11
python -m pip install paddlepaddle-gpu==3.2.2 -i https://www.paddlepaddle.org.cn/packages/stable/cu118/
python -m pip install pandas==1.5.3 numpy==1.24.4 paddleocr==3.3.0

3）编译paddle
yum install -y libnccl-2.17.1-1+cuda11.0 libnccl-devel-2.17.1-1+cuda11.0 libnccl-static-2.17.1-1+cuda11.0

conda activate paddleocr3

export CC=/usr/local/gcc10/bin/gcc
export CXX=/usr/local/gcc10/bin/g++
export CUDAHOSTCXX=/usr/local/gcc10/bin/g++
export PATH=/usr/local/gcc10/bin/:$PATH
export LD_LIBRARY_PATH=/usr/local/gcc10/lib64:$LD_LIBRARY_PATH
export NVCC_ARCH_BIN="86;61"

cd /usr/local/install
git clone https://github.com/PaddlePaddle/Paddle.git
cd Paddle
git checkout v3.2.2
git submodule update --init --recursive
mkdir -p build && cd build

vim /usr/local/install/Paddle/third_party/flashattn/csrc/CMakeLists.txt
注释掉以下内容，这里编译时候会到git更新错误
#if(NOT GIT_SUBMOD_RESULT EQUAL 0)

message(FATAL_ERROR "Failed to update Git submodules")

#endif()

export MY_CFLAGS="-mavx -mtune=generic"
export MY_CXXFLAGS="-mavx -mtune=generic"

cmake ..
-DCMAKE_BUILD_TYPE=Release
-DCMAKE_C_COMPILER=${CC}
-DCMAKE_CXX_COMPILER=${CXX}
-DCMAKE_C_FLAGS="${MY_CFLAGS}"
-DCMAKE_CXX_FLAGS="${MY_CXXFLAGS}"
-DWITH_GPU=ON
-DWITH_AVX=ON
-DWITH_NCCL=ON
-DNCCL_INCLUDE_DIR=/usr/include
-DWITH_MKL=OFF
-DWITH_TESTING=OFF
-DWITH_DISTRIBUTE=OFF
-DON_INFER=OFF
-DWITH_PYTHON=ON
-DPY_VERSION=3.11
-DCUDA_ARCH_NAME=Auto
-DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda
-DCUDNN_ROOT=/usr/local/cuda
-DCUDA_NVCC_EXECUTABLE=/usr/local/cuda/bin/nvcc

make -j$(nproc)

ls python/dist
pip install python/dist/paddlepaddle_gpu-3.2.2-*.whl

[passerbya]

编译后.whl安装后运行是没有问题的。编译报malloc_consolidate(): invalid chunk size主要原因还是/lib64/libc.so.6与/lib64/libstdc++.so.6版本不匹配造成的

[YuanRisheng]

没看明白是什么问题，是paddle编译有问题，还是paddleocr使用有问题呢

[passerbya]

没看明白是什么问题，是paddle编译有问题，还是paddleocr使用有问题呢

这是两个问题，第一个是在没有AVX2指令的CPU下运行的时候报“非法指令”，第二个问题是，遇到这个问题后我尝试重新编译，又遇到 #64794 #64649 #45026 的malloc_consolidate(): invalid chunk size问题。