背景

Hadoop 2.4.1預編譯版本自帶的libhadoop.so是在32位機器上編譯上,導致在64位OS上運行時，總出現如下告警：

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

查看libhadoop.so的位數，顯示是32位的：

file libhadoop.so.1.0.0

libhadoop.so.1.0.0: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, not stripped

為徹底解決此問題，自行編譯Hadoop 2.4.1源碼。

編譯過程

安裝相關軟件

yum install -y cmake autoconf automake libtool gcc zlib1g-dev pkg-config libssl-dev openssl gcc g++ make maven zlib zlib1g-dev libcurl4-o

安裝protobuf-2.5.0

1

2

3

4

5

6wget https://github.com/google/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz

tar -zxvf protobuf-2.5.0.tar.gz

cdprotobuf-2.5.0

./configure

make

make install

編譯Hadoop源碼

1

2

3

4

5wget https://archive.apache.org/dist/hadoop/core/hadoop-2.4.1/hadoop-2.4.1-src.tar.gz

tar -zxvf hadoop-2.4.1-src.tar.gz

cdhadoop-2.4.1-src

exportPlatform=x64

mvn package -Pdist,native -DskipTests -Dtar

查看編譯后的本地庫，已經是64位：

1

2

3cdhadoop-dist/target/hadoop-2.4.1/lib/native

file libhadoop.so.1.0.0

libhadoop.so.1.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped

拷貝到Hadoop環境中

1

2

3

4

5cdhadoop-dist/target/hadoop-2.4.1/lib/native

cp * /opt/hadoop/lib/native/

scp * data01:/opt/hadoop/lib/native/

scp * data02:/opt/hadoop/lib/native/

scp * data03:/opt/hadoop/lib/native/

修改環境變量

在etc/hadoop/hadoop-env.sh中增加：

1

2exportLD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native/:/usr/local/lib/

exportJAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH:/opt/hadoop/lib/native/

參考文檔