In one of earlier blogs I explained how we can build Native Hadoop(2.4.1) libraries on OS X. In the meanwhile Hadoop 2.5.1 was released, so I was curious if now the source code has been patched and building libraries on OS X works out of box. But to my surprise, it still doesn’t work.
So in this blog I won’t go into much details, for that you can check the other blog.
Issues faced on Building Native libraries On Mac OS X
1.Problem in hadoop-hdfs maven module
error:
[exec] /Users/gaurav/GitHub/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test/vecsum.c:61:23: error: use of undeclared identifier 'CLOCK_MONOTONIC' [exec] if (clock_gettime(CLOCK_MONOTONIC, &watch->start)) { [exec] ^ [exec] /Users/gaurav/GitHub/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test/vecsum.c:79:23: error: use of undeclared identifier 'CLOCK_MONOTONIC' [exec] if (clock_gettime(CLOCK_MONOTONIC, &watch->stop)) { [exec] ^
Solution: Download the Patch from Jira issue HDFS-6534. Download link-> HDFS-6534.v2.patch
- git apply HDFS-6534.v2.patch
- mvn package -Pdist,native -DskipTests -Dtar
2.Problems in hadoop-yarn-server-nodemanager maven module
error:
[exec] /Users/gaurav/GitHub/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c:501:33: error: use of undeclared identifier 'LOGIN_NAME_MAX' [exec] if (strncmp(*users, user, LOGIN_NAME_MAX) == 0) { [exec] ^
[exec] /Users/gaurav/GitHub/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c:1266:48: error: too many arguments to function call, expected 4, have 5
[exec] if (mount("none", mount_path, "cgroup", 0, controller) == 0) {
[exec] ~~~~~ ^~~~~~~~~~
[exec] /usr/include/sys/mount.h:384:1: note: 'mount' declared here
[exec] int mount(const char *, const char *, int, void *);
[exec] ^
Solution: Download the Patch from Jira issue YARN-2161. Download link-> YARN-2161.v1.patch
- git apply YARN-2161.v1.patch
- mvn package -Pdist,native -DskipTests -Dtar
Result
hadoop-dist/target/hadoop-2.5.1/lib/native folder should contain the native libraries. Copy them to hadoop-2.5.1/lib/native folder and restart Hadoop cluster.
Gaurav Kohli 
Hello, there – nice catch! Too bad I only found it after finishing my own tutorial, summarizing my efforts to install Hadoop 2.4.0 with native components on Yosemite for my work… Our approach is a bit different though (I changed things manually though, did not know there are official patches – re-creating the patches you’re linked to), so in case you are interested: http://lajosd.blogspot.be/2015/07/building-hadoop-240-on-mac-os-x.html
LikeLike
Thanks for the write-up!
A few questions:
Checkout branch-2.5.1 or release-2.5.1 – what is the difference?
They publish those patches – but it is a difference of what against what? What is the base for changes? I tried checking against 2.5.1 and 2.5.2 and seems like patches do not apply. Is there a way to create a specific patch yourself?
Building 2.6.0 works okay – would natively be compatible to 2.5.x versions?
LikeLike
Thanks a lot for the tips. Got Hadoop built from source for the first time on Mac OS :).
FYI, the second patch YARN-2161.v1.patch didn’t apply cleanly for me. Neither did the v2 patch in the JIRA. Compiled fine without them and seems to be working fine — cluster is up and could run an example job.
LikeLike