If you are reading this blog, I assume that you already have Hadoop(v 2.4.1) installed on your OS X machine and that you are bit annoyed by the following error message
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
If you are only planning to use Hadoop on OS X for development procedures, this error should not bother you. For me it was the same case, but I was just annoyed with this warning message and wanted to try building Native libraries from the source code.
Steps to build Native Hadoop libraries
- Download source from GitHub
- git clone git@github.com:apache/hadoop.git
- git checkout branch-2.4.1
- Dependencies
Install cmake and zlib using homebrew package manager
- brew install cmake
- brew install zlib
- Run maven command
- mvn package -Pdist,native -DskipTests -Dtar
On linux machines, the above procedure should be enough, but not for MAC OS X with Java 1.7. So for that you have to go with few more changes.
Issues faced on Building Native libraries On Mac OS X
1.Missing tools.jar
If you are building using Java 1.7, you would see an error talking about missing tools.jar, which is a bug in Maven JSPC. The related Jira issue is HADOOP-9350. The JSPC Plugin expects classes.jar in ../Classes folder, so we create a symlink.
error:
Exception in thread “main” java.lang.AssertionError: Missing tools.jar at: /Library/Java/JavaVirtualMachines/jdk1.7.0_17.jdk/Contents
/Home/Classes/classes.jar. Expression: file.exists()
Solution: Create a symbolic link to trick Java into believing that classes.jar is same as tools.jar
- sudo mkdir
/usr/libexec/java_home
/Classes- sudo ln -s
/usr/libexec/java_home
/lib/tools.jar/usr/libexec/java_home
/Classes/classes.jar
2. Incompatible source code
Some code in Hadoop v2.4.1 is not compatible with Mac system, so need to apply the patch HADOOP-9648.v2.patch and the related Jira issue is HADOOP-10699
error:
[exec] /Users/gaurav/GitHub/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsNetgroupMapping.c:77:26: error: invalid operands to binary expression ('void' and 'int') [exec] if(setnetgrent(cgroup) == 1) { [exec] ~~~~~~~~~~~~~~~~~~~ ^ ~ [exec] 1 error generated.
Solution: Download the Patch from Jira issue HADOOP-10699. Download link-> HADOOP-9648.v2.patch
- git apply HADOOP-9648.v2.patch
- mvn package -Pdist,native -DskipTests -Dtar
Result
hadoop-dist/target/hadoop-2.4.1/lib/native folder should contain the native libraries. Copy them to hadoop-2.4.1/lib/native folder and restart Hadoop cluster.
References
- Native Libraries Guide documentation page.
- Hadoop Git repo
- HADOOP-10699 V2 Patch
- Details about Maven JSPC Issue