Tag Archives: hadoop 2.5.1

Building Native Hadoop (v 2.5.1) libraries for OS X

15 Monday Dec 2014

Posted by in Big Data, Hadoop

≈ 3 Comments

Tags

,

In one of earlier blogs I explained how we can build Native Hadoop(2.4.1) libraries on OS X. In the meanwhile Hadoop 2.5.1 was released, so I was curious if now the source code has been patched and building libraries on OS X works out of box. But to my surprise, it still doesn’t work.

So in this blog I won’t go into much details, for that you can check the other blog.

Issues faced on Building Native libraries On Mac OS X

1.Problem in hadoop-hdfs maven module

error:

 [exec] /Users/gaurav/GitHub/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test/vecsum.c:61:23: error: use of undeclared identifier 'CLOCK_MONOTONIC'
 [exec]     if (clock_gettime(CLOCK_MONOTONIC, &watch->start)) {
 [exec]                       ^
 [exec] /Users/gaurav/GitHub/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test/vecsum.c:79:23: error: use of undeclared identifier 'CLOCK_MONOTONIC'
 [exec]     if (clock_gettime(CLOCK_MONOTONIC, &watch->stop)) {
 [exec]                       ^

Solution: Download the Patch from Jira issue HDFS-6534. Download link-> HDFS-6534.v2.patch

  • git apply HDFS-6534.v2.patch
  • mvn package -Pdist,native -DskipTests -Dtar

2.Problems in hadoop-yarn-server-nodemanager maven module

error:

 [exec] /Users/gaurav/GitHub/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c:501:33: error: use of undeclared identifier 'LOGIN_NAME_MAX'
 [exec]       if (strncmp(*users, user, LOGIN_NAME_MAX) == 0) {
 [exec]                                 ^   

 [exec] /Users/gaurav/GitHub/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c:1266:48: error: too many arguments to function call, expected 4, have 5
 [exec]     if (mount("none", mount_path, "cgroup", 0, controller) == 0) {
 [exec]         ~~~~~                                  ^~~~~~~~~~
 [exec] /usr/include/sys/mount.h:384:1: note: 'mount' declared here
 [exec] int     mount(const char *, const char *, int, void *);
 [exec] ^

Solution: Download the Patch from Jira issue YARN-2161. Download link-> YARN-2161.v1.patch

  • git apply YARN-2161.v1.patch
  • mvn package -Pdist,native -DskipTests -Dtar

Result

hadoop-dist/target/hadoop-2.5.1/lib/native folder should contain the native libraries. Copy them to hadoop-2.5.1/lib/native folder and restart Hadoop cluster.

Advertisement