Jenkins End-to-End CICD Pipeline Project

💡 Introduction

Welcome to the world of DevOps, today we are going to build a Full-fledged Jenkins Pipeline project. In this project, we will get build a Java project based on Maven and host it on GitHub repository which will act as source code for our application. Following that we will build a Jenkins pipeline which will checkout our code from GitHub with help of a webhook configured, use Maven to build the application, SonarQube which will scan the image of our project, and if it passes both stages, we will push it to our Docker Hub account. In case the build or scanning failed a Slack notification will be sent to our Slack account containing all the logs for the failure. After the pushing the image, we will use Shell scripts (not ArgoCD Image updater as it is not commonly used in industry level projects) to update our Docker image for the project and maintain a GitHub repository as manifest repository for following GitOps approach. At the end, we will use ArgoCD to deploy our application on Kubernetes.

💡 Pre-Requisites

Before we jump into building our Jenkins pipeline, let’s make sure we’ve got all the necessary tools and platforms ready to go. Below are the key components you need to have installed and configured on your system:

1. Docker and Docker Hub Account: We would be using docker to create an image of our application and we need to have a Docker Hub account on which we will push our project image. Docker can be installed with a single command:

    sudo apt install docker.io -y
   

2. Jenkins: This is the core tool for our project. Jenkins will automate the entire pipeline. You can install it on your system by following the instructions on the official Jenkins website here.

3. Minikube: We’ll use Minikube to simulate a Kubernetes cluster locally. You can get it installed on Ubuntu by following the steps from their documentation here.

4. ArgoCD: For our GitOps deployment, we’ll be using ArgoCD to handle Kubernetes deployments. You can install ArgoCD on your Kubernetes cluster by following this guide here.

5. Basic understanding of Maven and SonarQube: We’ll be using Maven to build our Java application, and SonarQube to scan the code for quality and security checks. You don’t need to be an expert, but a basic understanding of how these tools work will be helpful. If you need a refresher, check out Maven here and SonarQube here.

💡 Configuring Jenkins and Running the Application Locally

Once you’ve taken care of the prerequisites, it’s time to configure Jenkins and get our pipeline in motion. Start by installing the necessary plugins in Jenkins. These plugins will help integrate our tools and automate the build process:

1. Git Plugin: This allows Jenkins to interact with Git repositories, which is crucial since our project is hosted on GitHub.

2. Maven Integration Plugin: Since we’re working with a Maven-based project, this plugin helps Jenkins execute Maven commands.

3. Pipeline Plugin: This enables us to write and manage our Jenkins pipeline as code.

4. Kubernetes Continuous Deploy Plugin: We’ll use this for deploying our Docker container to the Kubernetes cluster.

You can install these plugins from the Manage Jenkins section under Manage Plugins in your Jenkins dashboard.

The source code for this project is available on GitHub: Jenkins-Zero-To-Hero Project. Now, before deploying this project on Kubernetes, let’s first run it locally using Docker.

Since this is a Maven application, you’ll need Maven installed on your system. You can install Maven on Ubuntu using this command:

sudo apt install maven

To run the project locally, follow these steps:

1. Clone the repository:

    git clone https://github.com/Pravesh-Sudha/Jenkins-Zero-To-Hero.git
   

2. Navigate to the project directory:

    cd Jenkins-Zero-To-Hero/java-maven-sonar-argocd-helm-k8s/spring-boot-app
   

3. Build the Maven project by executing the following command to generate the necessary artifacts:

    mvn clean package
   

4. Now, let’s create a Docker container for the application and run it. Use the following commands:

    docker build -t ultimate-cicd-pipeline:v1 .
    docker run -d -p 8010:8080 -t ultimate-cicd-pipeline:v1
   

After running these commands, the application will be accessible locally at http://localhost:8010. You can open this in your browser to verify that everything is working as expected before deploying to Kubernetes. This will show the following output:

💡 Setting Up Jenkins Pipeline for Kubernetes Deployment

Now that we've tested the application locally, it's time to take things up a notch by deploying it to Kubernetes. To do that, we first need to configure Jenkins and create a pipeline project.

1. Login to Jenkins: Start by logging into your Jenkins server. If you don’t know, here is the link.

2. Create a New Pipeline Project:

   • Go to New Item on your Jenkins dashboard.

   • Enter the project name as "Ultimate-Demo".

   • Choose Pipeline as the project type and click OK.

3. Configure Pipeline from SCM:

   • In the pipeline configuration, select Pipeline script from SCM.

   • For SCM, choose Git.

   • In the Repository URL field, enter the GitHub repository URL for our project: https://github.com/Pravesh-Sudha/Jenkins-Zero-To-Hero.

   • Set the Branch as "main".

4. Define the Script Path:

   • In the Script Path field, specify the path to the Jenkins pipeline file:

   • blob/main/java-maven-sonar-argocd-helm-k8s/spring-boot-app/JenkinsFile.

This pipeline configuration links your Jenkins server to the source code in your GitHub repository, and the pipeline script (JenkinsFile) will define the steps Jenkins takes to build, test, and deploy the application.

💡 Writing the Jenkinsfile

Now that we’ve set up our Jenkins pipeline, let’s dive into the Jenkinsfile, which defines the steps for building, testing, and deploying our application. You can find the Jenkinsfile in the GitHub repo at blob/main/java-maven-sonar-argocd-helm-k8s/spring-boot-app/JenkinsFile.

Let’s break down each stage in the pipeline:

1. Agent Configuration:

   • We’re using a Docker agent with the image 'abhishekf5/maven-abhishek-docker-agent:v1'.

   • The agent mounts the Docker socket from the host to allow the pipeline to run Docker commands.

2. Stage 1: Checkout:

   • In this stage, Jenkins will check out the project source code from GitHub. It’s currently commented out in the script with a simple sh 'echo passed' statement, but the actual command should be:

       git branch: 'main', url: 'https://github.com/Pravesh-Sudha/Jenkins-Zero-To-Hero.git'
     

3. Stage 2: Build and Test:

   • This stage runs the Maven build for the Java project and generates a JAR file.

   • It navigates to the Spring Boot app directory (java-maven-sonar-argocd-helm-k8s/spring-boot-app) and runs:

       mvn clean package
     

   • This command compiles the code and packages it into a distributable JAR file.

4. Stage 3: Static Code Analysis (SonarQube):

   • Here, we use SonarQube for static code analysis to check code quality.

   • The SonarQube server is defined at http://<YOUR-IP-ADDRESS>:9000.

   • Jenkins retrieves the SonarQube authentication token from credentials and then runs the SonarQube scan on the project using Maven.

       mvn sonar:sonar -Dsonar.login=$SONAR_AUTH_TOKEN -Dsonar.host.url=${SONAR_URL}
     

5. Stage 4: Build and Push Docker Image:

   • In this stage, we build a Docker image from the Spring Boot app and push it to Docker Hub.

   • The image tag is dynamically generated using the build number (${BUILD_NUMBER}).

   • After building the image, it’s pushed to Docker Hub using the stored Docker Hub credentials (docker-cred).

6. Stage 5: Update Deployment File:

   • This stage is all about updating the Kubernetes deployment manifest file with the new Docker image tag.

   • It updates the deployment.yml file to use the new image tag, commits the change, and pushes it to the GitHub repository using a GitHub token for authentication.

Before we run this pipeline, we need to install some additional plugins and set up SonarQube:

1. Install the Following Jenkins Plugins:

   • Docker Pipeline Plugin: To allow Jenkins to build and run Docker images.

   • SonarQube Scanner Plugin: To enable SonarQube integration for code analysis.

2. Install SonarQube on Your System: SonarQube needs to be set up locally for our static code analysis. Follow these steps to install it on Ubuntu:

    sudo apt install unzip
    sudo adduser sonarqube
    wget https://binaries.sonarsource.com/Distribution/sonarqube/sonarqube-9.4.0.54424.zip
    unzip sonarqube-9.4.0.54424.zip
    sudo chmod -R 755 /home/sonarqube/sonarqube-9.4.0.54424
    sudo chown -R sonarqube:sonarqube /home/sonarqube/sonarqube-9.4.0.54424
    cd /home/sonarqube/sonarqube-9.4.0.54424/bin/linux-x86-64/
    ./sonar.sh start
   

   Once SonarQube is up and running, you can access it at http://localhost:9000 to configure your project for code analysis. The dashboard will look like this:

The Login and Password for the sonarqube server is admin (for both).

💡 Creating a SonarQube Token and Adding it to Jenkins

Now that Docker and SonarQube are set up on your system, the next step is to create an authentication token in SonarQube and link it to Jenkins. This token allows Jenkins to communicate with SonarQube for code analysis.

Here’s how you can create the token and add it to Jenkins:

1. Create a Token in SonarQube:

   • Open your SonarQube dashboard and go to your Account Settings.

   • Navigate to the Security Tab.

   • Under Generate Token, enter "Jenkinsfile" as the token name and click Create Token.

   • Copy the token that gets generated. Make sure to save it somewhere safe as it won’t be shown again.

1. Add the Token to Jenkins:

   • Head to your Jenkins dashboard, then go to Manage Jenkins.

   • Select Manage Credentials.

   • Under Global Credentials, click Add Credentials.

   • In the Kind dropdown, select Secret Text.

   • Paste the SonarQube token you just copied into the Secret field.

   • In the ID field, enter "Sonarqube".

   • Click Create.

By adding this token, Jenkins will be able to authenticate with SonarQube and perform static code analysis in the pipeline.

💡 Setting Up Minikube and ArgoCD-Operator

Now that we’ve configured SonarQube and Jenkins, it's time to create a Kubernetes cluster using Minikube and install ArgoCD for our deployment process.

1. Create a Minikube Cluster: We’ll start by creating a Minikube cluster. Use the following command to start the cluster with 4GB of memory and the Hyperkit driver:

    minikube start --memory=4098 --driver=hyperkit
   

   Once the cluster is up and running, we can proceed to install ArgoCD.

2. Install ArgoCD Operator: To install ArgoCD in our Minikube cluster, run the following commands:

   • First, install the Operator Lifecycle Manager (OLM) for managing the ArgoCD operator:

       curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.28.0/install.sh | bash -s v0.28.0
     

   • Then, install the ArgoCD operator:

       kubectl create -f https://operatorhub.io/install/argocd-operator.yaml
     

   • Finally, verify the installation by checking the ClusterServiceVersion (CSV):

       kubectl get csv -n operators
     

3. Configure Jenkins Credentials: After installing ArgoCD, we need to set up some credentials in Jenkins for seamless integration:

   • Docker Credentials:

     • Add your Docker Hub username and password to Jenkins.

     • In Manage Credentials, create a new credential of type Username with password.

     • Save it with the ID "docker-cred".

   • GitHub Token:

     • To connect ArgoCD with GitHub, create a Personal Access Token on GitHub.

     • Add this token to Jenkins as Secret Text with the ID "GITHUB_TOKEN".

With the Minikube cluster and ArgoCD in place, and the required credentials set up in Jenkins, we’re ready to move forward with deploying our application to Kubernetes!

💡 Running the Pipeline and Setting Up ArgoCD

Now that we’ve set everything up, it's time to run our Jenkins pipeline. Once the pipeline kicks off, it will execute all the stages we’ve defined in the Jenkinsfile:

1. Pipeline Execution:

   • The checkout stage will pull the latest code from the GitHub repository.

   • In the Build and Test stage, Maven will build the application and create a JAR file, using the mvn clean package command.

   • The Static Code Analysis stage will use SonarQube to scan the code. After this, the project will appear in the Projects tab of the SonarQube dashboard.

   • The Build and Push Docker Image stage will create a Docker image tagged as pravesh2003/ultimate-cicd and push it to the Docker Hub account.

   • In the final stage, Update Deployment File, the Docker image version in the Kubernetes deployment manifest will be updated to the latest one.

Once all stages complete successfully, our Docker image will be available on Docker Hub under pravesh2003/ultimate-cicd, and the Maven build artifacts (JAR file) will be stored locally. (Be patient with the Jenkins Pipeline as you can see it takes me 50 times to run the pipeline successfully.) If you get an error, feel free to leave a comment.

2. Setting Up ArgoCD: Now that we have the Docker image, let’s move on to configuring ArgoCD to handle our deployments:

   • First, create an argocd-basic.yaml file with the following content:

       apiVersion: argoproj.io/v1alpha1
       kind: ArgoCD
       metadata:
         name: example-argocd
         labels:
           example: basic
       spec: {}
     

   • Apply the YAML file to the cluster:

       kubectl apply -f argocd-basic.yaml
     

   • Wait for the ArgoCD pods to be in a Running state by checking:

       kubectl get pods -n argocd
     

3. Accessing ArgoCD Dashboard:

   • Next, get the service details for ArgoCD:

       kubectl get svc example-argocd-server
     

   • To make the ArgoCD server accessible from your browser, change its service type from ClusterIP to NodePort:

       kubectl edit svc example-argocd-server
     

   • After that, use Minikube to create a link to the service:

       minikube service example-argocd-server
     

   • You can also list all Minikube services to find the link to access the ArgoCD server dashboard:

       minikube service list
     

     

     In the above image, you can see the url as http://127.0.0.1:61925, go to it and you will see the following page:

4. Logging into the ArgoCD Server:

   • The default Username for ArgoCD is admin, and the Password is stored in a Kubernetes secret.

   • To retrieve the password, run the following command:

       kubectl edit secret example-argocd-cluster
     

     

   • Find the encoded password value in the secret, then decode it:

       echo "ENCODED_PASSWORD_VALUE" | base64 -d
     

   • Copy the decoded password (don’t copy the % symbol as the last character of password) and use it to log in to the ArgoCD dashboard.

With this setup, we’re now able to access ArgoCD and manage the deployment of our application on Kubernetes from a simple web interface.

After Login into the Argocd, you will see the following screen:/

💡 Creating an Application in ArgoCD

Now that ArgoCD is set up, let’s create a new application for our project. Follow these steps to deploy the application onto the Kubernetes cluster:

1. Create a New Application:

   • In the ArgoCD dashboard, click on the New App button.

   • For the Application Name, enter 'test'.

   • In the Project Name, select default.

   • Set the Sync Policy to Automatic.

   • Under Repository URL, enter:

       https://github.com/Pravesh-Sudha/Jenkins-Zero-To-Hero
     

   • In the Path field, enter:

       java-maven-sonar-argocd-helm-k8s/spring-boot-app-manifests
     

2. Set Deployment Details:

   • Navigate to the Destination tab.

   • In Cluster URL, use the default cluster.

   • Set the Namespace to default.

3. Create the App:

   • Click on Create App to deploy the application onto your Kubernetes cluster. ArgoCD will now monitor and manage the deployment automatically. If you update the image version or make any changes in the Git repository, ArgoCD will roll out these updates to the Kubernetes cluster, ensuring that your application is always in sync with the latest configuration.

💡Conclusion

And that’s a wrap! 🎉 We’ve successfully created an end-to-end Jenkins CI/CD pipeline that builds, tests, scans, and pushes a Docker image to Docker Hub, then deploys it to Kubernetes using ArgoCD. Throughout this project, you got to see how powerful automation can be when integrating tools like Jenkins, SonarQube, Docker, and ArgoCD into your workflow. With this setup, your applications can be continuously delivered and updated in a streamlined, hands-off way.

I hope this guide was helpful! Feel free to experiment with different stages or add more automation to your pipeline. Remember, DevOps is all about continuous improvement, so keep iterating and optimizing your workflow! Good luck, and happy coding! 🚀 For more informative blog, Follow me on Hashnode, X(Twitter) and LinkedIn.

Till then, Happy Coding!!