How to Install Hadoop on Ubuntu 22.04

Hadoop is an open-source framework that facilitates the distributed processing of large datasets across clusters of computers using simple programming models. Ubuntu, on the other hand, is a popular Linux distribution known for its simplicity and ease of use. Installing Hadoop on Ubuntu can be a straightforward process if you follow the right steps. In this article, we’ll guide you through the process of install Hadoop on Ubuntu, step by step.

Hadoop is widely used in handling big data due to its scalability and fault tolerance. It consists of two main components: the Hadoop Distributed File System (HDFS) for storage and the MapReduce programming model for processing data in parallel across a distributed cluster.

Ubuntu is a user-friendly Linux OS among different Linux distributions favored by many developers and system administrators. It provides a stable and secure environment for running various applications, including Hadoop.

Installing Hadoop on Ubuntu 22.04

The following are the steps to install Hadoop on Ubuntu operating system:

Step 1: Updating System Repositories

When installing Hadoop on Ubuntu, updating the package lists ensures that you have the latest information about available software packages and their versions. This step is essential to ensure that when you proceed with installing Hadoop and its dependencies, you’re using the most up-to-date package information from the Ubuntu repositories. This helps in avoiding potential issues related to outdated package lists and ensures a smoother installation process.

sudo apt update

Step 2: Installing Java Development Kit (JDK)

The Java Development Kit (JDK) is essential for developing and running Java applications. Hadoop, being a Java-based framework, requires the JDK to be installed on the system to compile and run its Java code. By installing the default JDK package using this command, you ensure that the necessary Java development tools are available on your system, which are required for building and running Hadoop.

sudo apt install default-jdk

When you run this command, it will output information about the version of Java that is currently installed on your system. This information includes the version number of Java along with additional details such as the Java Runtime Environment (JRE) version and the Java Virtual Machine (JVM) version. Verifying the Java version is important to ensure compatibility with Hadoop, as certain versions of Hadoop may require specific versions of Java to function properly.

java -version

Step 3: Downloading Hadoop

After opening the browser, navigate to the official Hadoop website to open the Apache Hadoop website’s releases page. This page typically provides information about the various releases of Apache Hadoop, including the latest stable versions and any relevant release notes or documentation. It’s a resource where you can find the download links for different versions of Hadoop.

Next, copy the link address and execute the following command:

wget https://dlcdn.apache.org/hadoop/common/hadoop-3.4.0/hadoop-3.4.0.tar.gz

By executing this command, you’re instructing wget to download the specified Hadoop distribution package from the provided URL. Once the download is complete, you’ll have the Hadoop distribution archive file available locally on your system. This archive file contains all the necessary files and directories required to install and configure Hadoop on your Ubuntu system.

Step 4: Extracting the Content

The downloaded file with be an archive file where you need to extract its content by executing the below command:

tar -xzvf hadoop-3.4.0.tar.gz

The -x option specifies extraction, the -z option indicates that the archive is compressed with gzip, the -v option enables verbose mode for detailed output, and the -f option specifies the filename of the archive to be processed.

After running this command, the contents of the Hadoop archive will be extracted and available in a directory named hadoop-3.4.0 in the current working directory. This directory will contain all the files and directories comprising the Hadoop distribution, which can then be further configured and utilized as needed.

Step 5: Moving the Extracted Hadoop Directory to the Installation Location

It is a common practice to organize Hadoop installations on Ubuntu systems. Moving the directory to /usr/local/hadoop ensures that Hadoop is installed in a standard location where it can be easily accessed and managed. After running this command, you’ll find the Hadoop installation directory located at /usr/local/hadoop, containing all the necessary files and directories for configuring and running Hadoop on your system.

sudo mv hadoop-3.4.0 /usr/local/hadoop

Step 6: Finding the Installed Java Path

This can be useful for obtaining the directory where Java is installed, which may be necessary for configuring Hadoop.

readlink -f /usr/bin/java | sed "s:bin/java::"

Step 7: Editing Hadoop Environment Configuration File

Next, you need to open the “hadoop-env.sh” file for editing. This allows you to modify environment variables and other configurations as needed to customize the behavior of your Hadoop installation. Using nano as the text editor provides a simple interface for making changes to the configuration file directly from the terminal.

sudo nano /usr/local/hadoop/hadoop-3.4.0/etc/hadoop/hadoop-env.sh

Then write this line inside hadoop-env.sh:

export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64/

By including this command in the “hadoop-env.sh” file, you’re specifying the location of the Java installation that Hadoop should use. This is essential because Hadoop requires Java to be installed on the system and needs to know the location of the Java executable to run. Setting the JAVA_HOME environment variable ensures that Hadoop can locate the Java installation correctly and use it for its operation.

Furthermore, it sets the JAVA_HOME environment variable with the static address pointing to the installation directory of the Java Development Kit (JDK). To provide the dynamic location of the Java installation directory without hardcoding the path, you can execute:

export JAVA_HOME=$(readlink -f /usr/bin/java | sed "s:bin/java::")

This can be useful in scenarios where the location of the Java executable may vary across different systems. The command ensures that the JAVA_HOME environment variable always points to the correct Java installation directory, regardless of the specific path to the Java executable.

If you are not able to find these commands then you can use CTRL+W shortcut key to find this line as well.

Step 8:Verifying Hadoop Installation

Now, the installation of Hadoop is completed that you can verify using the below command:

/usr/local/hadoop/hadoop-3.4.0/bin/hadoop version

By executing this command, you are instructing Hadoop to output information about its version. This includes latest Hadoop installation on Ubuntu such as the version number, build information, and any other relevant metadata. Checking the version of Hadoop installed on the system is important for verifying the installation and ensuring compatibility with other software components or applications that rely on Hadoop.

Reasons to Use Hadoop

1. Scalability: One of the primary reasons to use Hadoop is its scalability. As data volumes continue to grow exponentially, Hadoop provides a scalable solution that can easily accommodate the increasing demands for storage and processing power by adding more nodes to the cluster.
2. Flexibility: Hadoop offers flexibility in terms of data types and sources. It can handle structured, semi-structured, and unstructured data from various sources, including social media, sensor data, and log files, making it suitable for a wide range of use cases.
3. Cost-effectiveness: Compared to traditional data storage and processing solutions, Hadoop is cost-effective. It runs on commodity hardware, eliminating the need for expensive proprietary hardware and software licenses, thus reducing the total cost of ownership for organizations.

Benefits of Hadoop

• Compatibility with Ubuntu: Hadoop is fully compatible with Ubuntu, one of the most popular Linux distributions. This compatibility allows organizations to leverage the benefits of Hadoop while utilizing the stability and security features of the Ubuntu operating system.
• Enhanced performance: By running Hadoop on Ubuntu, organizations can achieve enhanced performance and reliability. Ubuntu’s lightweight design and efficient resource management optimize the performance of Hadoop clusters, ensuring faster data processing and analysis.
• Data management capabilities: Ubuntu provides robust data management capabilities that complement Hadoop’s functionality. With features such as advanced file system support, security enhancements, and seamless integration with cloud services, Ubuntu enhances the overall data management experience for Hadoop users.

Conclusion

Hadoop is a distributed processing technology that enables the processing of large datasets across clusters of computers using simple programming models. It consists of two main components: the Hadoop Distributed File System (HDFS) for storage and the MapReduce programming model for processing. 

The installation process of Hadoop in Ubuntu has also been discussed in this guide. By following these instructions, users can quickly get up and running with Hadoop, unlocking its potential to transform their data workflows and drive insights that were previously unattainable.

FAQ