Prometheus and Grafana Integration

Overview:

The combination of Prometheus and Grafana is becoming a more and more common monitoring stack used by DevOps teams for storing and visualizing time series data. Prometheus acts as the storage backend and Grafana as the interface for analysis and visualization.

Prometheus collects metrics from monitored targets by scraping metrics from HTTP endpoints on these targets.

Purpose of this article:

1. Deploy prometheus and grafana as pods on top of Kubernetes by creating resources Deployment, ReplicaSet, Pods or Services

2. And make their data to be remain persistent.

3. And both of them should be exposed to outside world.

Steps to perform:

Create Dockerfile for Prometeus and Grafana :

Prometheus :

FROM centos
RUN yum install wget tar -y
# Downloading Prometheus's archived file
RUN wget https://github.com/prometheus/prometheus/releases/download/v2.19.2/prometheus-2.19.2.linux-amd64.tar.gz
# extracting the archive into a separate directory
RUN mkdir /promdir
RUN tar -xzf prometheus-2.19.2.linux-amd64.tar.gz
RUN cp -rf  prometheus-2.19.2.linux-amd64/* /promdir
RUN cp /promdir/prometheus /usr/bin
# this config file will serve as config map on k8s
CMD prometheus --config.file=/promdir/prometheus.yml

Grafana :

FROM centos
# Downloading Grafana
RUN yum install wget -y
RUN wget https://dl.grafana.com/oss/release/grafana-7.0.6-1.x86_64.rpm
RUN yum install grafana-7.0.6-1.x86_64.rpm -y
WORKDIR /usr/share/grafana
# providing the entry point
ENTRYPOINT ["/usr/sbin/grafana-server", "--config=/etc/grafana/grafana.ini", "cfg:default.paths.logs=/var/log/grafana", "cfg:default.paths.data=/var/lib/grafana", "LimitNOFILE=10000"]

Now push both the images to DockerHub :

docker build -t user_name/grafana:v1 .
docker push user_name/grafana:v1

Download, Install and start exporters for the targets:

wget https://github.com/prometheus/node_exporter/releases/download/v1.0.0/node_exporter-1.0.0.rc.1.linux-amd64.tar.gz
tar xvfz node_exporter-1.0.0.rc.1.linux-amd64.tar.gz
cd node_exporter-1.0.0.rc.1.linux-amd64
./node_exporter

Creating Deployment, Services, ConfigMap and PVC’s for Prometehus and Grafana :

Deployment :

Grafana :

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
spec:
  replicas: 1
  selector:
    matchLabels:
      env: deploy
  template:
    metadata:
      name: grafana-pod
      labels:
        env: deploy
    spec:
      containers:
        - name: g1
          image: devmohitlive/grafana
          volumeMounts:
            - name: data
              mountPath: /var/lib/grafana
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: grafana-pvc

Prometheus :

apiVersion: apps/v1
kind: Deployment
metadata:
    name: prometheus
    labels:
      env: deploy 
spec:
  replicas: 1
  selector:
    matchLabels:
       env: deploy
  template:
    metadata:
     name: prometheus-pod
     labels:
       env: deploy
    spec: 
      containers:
        - name: promnode
          image: devmohitlive/prometheus
          volumeMounts:
            - name: logs
              mountPath: /promdir
          volumeMounts:
            - name: nodes
              mountPath: /promdir/prometheus.yml
              subPath: prometheus.yml
      volumes:
      - name: logs
        persistentVolumeClaim:
            claimName: prometheus-pvc
      - name: nodes
        configMap:
            name: prometheus-nodes
            defaultMode: 0744

PVC’s :

Grafana :

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: grafana-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 2Gi

Promethus:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
    name: prometheus-pvc
spec:
    accessModes:
    - ReadWriteMany
    resources:
        requests:
            storage: 2Gi

Services:

Grafana :

apiVersion: v1
kind: Service
metadata:
  name: grafana
spec:
  selector:
    env: deploy
  type: NodePort
  ports:
    - port: 3000
      targetPort: 3000
      nodePort: 30004

Prometheus:

apiVersion: v1
kind: Service
metadata:
  name: prometheus
spec:
  selector:
    env: deploy
  type: NodePort
  ports:
    - port: 9090
      targetPort: 9090
      nodePort: 30008

ConfigMap :

kind: ConfigMap
apiVersion: v1
metadata:
  name: prometheus-nodes
data:
  prometheus.yml: |
    global: 
      scrape_interval: 15s
      evaluation_interval: 15s
    alerting: 
      alertmanagers:
      - static_configs: 
        - targets: 
    rule_files:
    scrape_configs:
      - job_name: 'node'
        static_configs:
        - targets : ['192.168.43.55:9100']
      - job_name: 'docker'
        static_configs:
        - targets : ['192.168.43.77:1234']

Kustomization File :

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - configmap.yml
  - prometheus.yml
  - grafana.yml

Creating the Infrastructure :

kubectl create -k .

Resources Created :

Git hub files….

Thank You !!