In one of earlier blogs I explained how we can build Native Hadoop(2.4.1) libraries on OS X. In the meanwhile Hadoop 2.5.1 was released, so I was curious if now the source code has been patched and building libraries on OS X works out of box. But to my surprise, it still doesn’t work.
So in this blog I won’t go into much details, for that you can check the other blog.
Issues faced on Building Native libraries On Mac OS X
1.Problem in hadoop-hdfs maven module
error:
[exec] /Users/gaurav/GitHub/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test/vecsum.c:61:23: error: use of undeclared identifier 'CLOCK_MONOTONIC' [exec] if (clock_gettime(CLOCK_MONOTONIC, &watch->start)) { [exec] ^ [exec] /Users/gaurav/GitHub/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test/vecsum.c:79:23: error: use of undeclared identifier 'CLOCK_MONOTONIC' [exec] if (clock_gettime(CLOCK_MONOTONIC, &watch->stop)) { [exec] ^
Solution: Download the Patch from Jira issue HDFS-6534. Download link-> HDFS-6534.v2.patch
- git apply HDFS-6534.v2.patch
- mvn package -Pdist,native -DskipTests -Dtar
2.Problems in hadoop-yarn-server-nodemanager maven module
error:
[exec] /Users/gaurav/GitHub/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c:501:33: error: use of undeclared identifier 'LOGIN_NAME_MAX' [exec] if (strncmp(*users, user, LOGIN_NAME_MAX) == 0) { [exec] ^
[exec] /Users/gaurav/GitHub/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c:1266:48: error: too many arguments to function call, expected 4, have 5
[exec] if (mount("none", mount_path, "cgroup", 0, controller) == 0) {
[exec] ~~~~~ ^~~~~~~~~~
[exec] /usr/include/sys/mount.h:384:1: note: 'mount' declared here
[exec] int mount(const char *, const char *, int, void *);
[exec] ^
Solution: Download the Patch from Jira issue YARN-2161. Download link-> YARN-2161.v1.patch
- git apply YARN-2161.v1.patch
- mvn package -Pdist,native -DskipTests -Dtar
Result
hadoop-dist/target/hadoop-2.5.1/lib/native folder should contain the native libraries. Copy them to hadoop-2.5.1/lib/native folder and restart Hadoop cluster.