IntroductionI maintain several clusters. Most of them are used for testing various versions of Hadoop. I needed a way to automate switching the client side version of java and Hadoop on my laptop.
So, I created a simple system which helps me with that.
|Create a folder for your |
|Create a subfolder for your |
|Extract your distribution of |
Hadoop into its own folder
|tar xvf hadoop-22.214.171.124.tar.gz |
mv hadoop-126.96.36.199 /opt/hadoop/hdp1.2/hadoop
|In the same subfolder, extract |
your version of Java
|tar xvf jdk-6u31-linux-x64.tar.gz -C /opt/hadoop/hdp1.2/|
|Create a symbolic link to |
the java folder
ln -s jdk-6u31-linux-x64 java
|Modify JAVA_HOME in |
|nano hadoop/conf/hadoop-env.sh |
|Modify bin/hadoop||nano hadoop/bin/hadoop |
Comment the following lines:
#bin=`cd "$bin"; pwd`
Add the following lines:
bin=`readlink -f "$0"`
if [ -z "$bin" ]; then
|Add update-alternatives |
update-alternatives --install <link> <name> <path> <priority>
sudo update-alternatives --install /usr/bin/hadoop hadoop /opt/hadoop/cdh3u4/hadoop/bin/hadoop 100
|Install Hadoop configs||Drop your configuration files in the conf directory|
Repeat steps as needed for additional Hadoop clients.
After setting everything up, your folder structure could look something like this:
- java -> jdk1.6.0_31/
Switching Hadoop Distributions
You can switch between hadoop configurations by entering the following:
sudo update-alternatives --config hadoop
You'll then be presented with a menu where you can select which version of Hadoop you'd like to use.
Other Possible Tweaks
- You could store all your java distributions in /opt/java and then point your java symbolic links toward them.
- Use slave links within update-alternatives to handle the java symbolic links