Category Archives: Big Data

Building Native Hadoop (v 2.5.1) libraries for OS X

15 Monday Dec 2014

Posted by in Big Data, Hadoop

≈ 3 Comments

Tags

,

In one of earlier blogs I explained how we can build Native Hadoop(2.4.1) libraries on OS X. In the meanwhile Hadoop 2.5.1 was released, so I was curious if now the source code has been patched and building libraries on OS X works out of box. But to my surprise, it still doesn’t work.

So in this blog I won’t go into much details, for that you can check the other blog.

Issues faced on Building Native libraries On Mac OS X

1.Problem in hadoop-hdfs maven module

error:

 [exec] /Users/gaurav/GitHub/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test/vecsum.c:61:23: error: use of undeclared identifier 'CLOCK_MONOTONIC'
 [exec]     if (clock_gettime(CLOCK_MONOTONIC, &watch->start)) {
 [exec]                       ^
 [exec] /Users/gaurav/GitHub/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test/vecsum.c:79:23: error: use of undeclared identifier 'CLOCK_MONOTONIC'
 [exec]     if (clock_gettime(CLOCK_MONOTONIC, &watch->stop)) {
 [exec]                       ^

Solution: Download the Patch from Jira issue HDFS-6534. Download link-> HDFS-6534.v2.patch

  • git apply HDFS-6534.v2.patch
  • mvn package -Pdist,native -DskipTests -Dtar

2.Problems in hadoop-yarn-server-nodemanager maven module

error:

 [exec] /Users/gaurav/GitHub/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c:501:33: error: use of undeclared identifier 'LOGIN_NAME_MAX'
 [exec]       if (strncmp(*users, user, LOGIN_NAME_MAX) == 0) {
 [exec]                                 ^   

 [exec] /Users/gaurav/GitHub/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c:1266:48: error: too many arguments to function call, expected 4, have 5
 [exec]     if (mount("none", mount_path, "cgroup", 0, controller) == 0) {
 [exec]         ~~~~~                                  ^~~~~~~~~~
 [exec] /usr/include/sys/mount.h:384:1: note: 'mount' declared here
 [exec] int     mount(const char *, const char *, int, void *);
 [exec] ^

Solution: Download the Patch from Jira issue YARN-2161. Download link-> YARN-2161.v1.patch

  • git apply YARN-2161.v1.patch
  • mvn package -Pdist,native -DskipTests -Dtar

Result

hadoop-dist/target/hadoop-2.5.1/lib/native folder should contain the native libraries. Copy them to hadoop-2.5.1/lib/native folder and restart Hadoop cluster.

Advertisement

Building Native Hadoop (v 2.4.1) libraries for OS X

28 Sunday Sep 2014

Posted by in Big Data, Hadoop

≈ 6 Comments

Tags

, , ,

If you are reading this blog, I assume that you already have Hadoop(v 2.4.1) installed on your OS X machine and that you are bit annoyed by the following error message

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

If you are only planning to use Hadoop on OS X for development procedures, this error should not bother you. For me it was the same case, but I was just annoyed with this warning message and wanted to try building Native libraries from the source code.

Steps to build Native Hadoop libraries

  1. Download source from GitHub
  • git clone git@github.com:apache/hadoop.git
  • git checkout branch-2.4.1
  1. Dependencies
    Install cmake and zlib using homebrew package manager
  • brew install cmake
  • brew install zlib
  1. Run maven command
  • mvn package -Pdist,native -DskipTests -Dtar

On linux machines, the above procedure should be enough, but not for MAC OS X with Java 1.7. So for that you have to go with few more changes.

Issues faced on Building Native libraries On Mac OS X

1.Missing tools.jar

If you are building using Java 1.7, you would see an error talking about missing tools.jar, which is a bug in Maven JSPC. The related Jira issue is HADOOP-9350. The JSPC Plugin expects classes.jar in ../Classes folder, so we create a symlink.

error:

Exception in thread “main” java.lang.AssertionError: Missing tools.jar at: /Library/Java/JavaVirtualMachines/jdk1.7.0_17.jdk/Contents
/Home/Classes/classes.jar. Expression: file.exists()

Solution: Create a symbolic link to trick Java into believing that classes.jar is same as tools.jar

  • sudo mkdir /usr/libexec/java_home/Classes
  • sudo ln -s /usr/libexec/java_home/lib/tools.jar /usr/libexec/java_home/Classes/classes.jar

2. Incompatible source code

Some code in Hadoop v2.4.1 is not compatible with Mac system, so need to apply the patch HADOOP-9648.v2.patch and the related Jira issue is HADOOP-10699

error:

     [exec] /Users/gaurav/GitHub/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsNetgroupMapping.c:77:26: error: invalid operands to binary expression ('void' and 'int')
     [exec]   if(setnetgrent(cgroup) == 1) {
     [exec]      ~~~~~~~~~~~~~~~~~~~ ^  ~
     [exec] 1 error generated.

Solution: Download the Patch from Jira issue HADOOP-10699. Download link-> HADOOP-9648.v2.patch

  • git apply HADOOP-9648.v2.patch
  • mvn package -Pdist,native -DskipTests -Dtar

Result

hadoop-dist/target/hadoop-2.4.1/lib/native folder should contain the native libraries. Copy them to hadoop-2.4.1/lib/native folder and restart Hadoop cluster.

References

  1. Native Libraries Guide documentation page.
  2. Hadoop Git repo
  3. HADOOP-10699 V2 Patch
  4. Details about Maven JSPC Issue