• Home
  • Big Data
  • Hadoop
  • Oozie
  • Hive
  • Tez
  • Scala
  • Archives
  • About

Gaurav Kohli

Gaurav Kohli

Tag Archives: mac os

Building Native Hadoop (v 2.4.1) libraries for OS X

28 Sunday Sep 2014

Posted by Gaurav in Big Data, Hadoop

≈ 6 Comments

Tags

hadoop 2.4.1, java 1.7, mac os, native libraries

If you are reading this blog, I assume that you already have Hadoop(v 2.4.1) installed on your OS X machine and that you are bit annoyed by the following error message

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

If you are only planning to use Hadoop on OS X for development procedures, this error should not bother you. For me it was the same case, but I was just annoyed with this warning message and wanted to try building Native libraries from the source code.

Steps to build Native Hadoop libraries

  1. Download source from GitHub
  • git clone git@github.com:apache/hadoop.git
  • git checkout branch-2.4.1
  1. Dependencies
    Install cmake and zlib using homebrew package manager
  • brew install cmake
  • brew install zlib
  1. Run maven command
  • mvn package -Pdist,native -DskipTests -Dtar

On linux machines, the above procedure should be enough, but not for MAC OS X with Java 1.7. So for that you have to go with few more changes.

Issues faced on Building Native libraries On Mac OS X

1.Missing tools.jar

If you are building using Java 1.7, you would see an error talking about missing tools.jar, which is a bug in Maven JSPC. The related Jira issue is HADOOP-9350. The JSPC Plugin expects classes.jar in ../Classes folder, so we create a symlink.

error:

Exception in thread “main” java.lang.AssertionError: Missing tools.jar at: /Library/Java/JavaVirtualMachines/jdk1.7.0_17.jdk/Contents
/Home/Classes/classes.jar. Expression: file.exists()

Solution: Create a symbolic link to trick Java into believing that classes.jar is same as tools.jar

  • sudo mkdir /usr/libexec/java_home/Classes
  • sudo ln -s /usr/libexec/java_home/lib/tools.jar /usr/libexec/java_home/Classes/classes.jar

2. Incompatible source code

Some code in Hadoop v2.4.1 is not compatible with Mac system, so need to apply the patch HADOOP-9648.v2.patch and the related Jira issue is HADOOP-10699

error:

     [exec] /Users/gaurav/GitHub/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsNetgroupMapping.c:77:26: error: invalid operands to binary expression ('void' and 'int')
     [exec]   if(setnetgrent(cgroup) == 1) {
     [exec]      ~~~~~~~~~~~~~~~~~~~ ^  ~
     [exec] 1 error generated.

Solution: Download the Patch from Jira issue HADOOP-10699. Download link-> HADOOP-9648.v2.patch

  • git apply HADOOP-9648.v2.patch
  • mvn package -Pdist,native -DskipTests -Dtar

Result

hadoop-dist/target/hadoop-2.4.1/lib/native folder should contain the native libraries. Copy them to hadoop-2.4.1/lib/native folder and restart Hadoop cluster.

References

  1. Native Libraries Guide documentation page.
  2. Hadoop Git repo
  3. HADOOP-10699 V2 Patch
  4. Details about Maven JSPC Issue
Advertisement

Archives

  • December 2014 (1)
  • September 2014 (1)
  • August 2014 (1)
  • August 2013 (2)
  • July 2013 (1)

Tweet

  • RT @msisodia: The best appreciation that one can get ever! twitter.com/PriyankaJShukl… 2 years ago
  • RT @gchahal: Our culture has accepted two huge lies...The first is that if you disagree with someone, you must hate them. The second is tha… 2 years ago
  • RT @KushaKapila: time taken to ban @kunalkamra88 from several airlines: less than half a day time taken to arrest ABVP's Komal Sharma: sti… 3 years ago

Categories

  • Big Data (2)
  • Hadoop (3)
  • Jira (2)
  • Oozie (1)
  • Play (1)

Info

Amsterdam
Follow Gaurav Kohli on WordPress.com

Tweet

  • RT @msisodia: The best appreciation that one can get ever! twitter.com/PriyankaJShukl… 2 years ago
  • RT @gchahal: Our culture has accepted two huge lies...The first is that if you disagree with someone, you must hate them. The second is tha… 2 years ago

Categories

  • Big Data (2)
  • Hadoop (3)
  • Jira (2)
  • Oozie (1)
  • Play (1)

Archives

  • December 2014 (1)
  • September 2014 (1)
  • August 2014 (1)
  • August 2013 (2)
  • July 2013 (1)

Blog at WordPress.com.

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy
  • Follow Following
    • Gaurav Kohli
    • Already have a WordPress.com account? Log in now.
    • Gaurav Kohli
    • Customize
    • Follow Following
    • Sign up
    • Log in
    • Report this content
    • View site in Reader
    • Manage subscriptions
    • Collapse this bar