Install and Configure Prometheus Server and Node Exporter in RHEL

Install and Configure Prometheus Server and Node Exporter

There are 3 importants components to make sure Prometheus is up and running. We need to install and configure

  1. Prometheus Server,
  2. Node Exporter, and
  3. Dashborad which can be PromDash or Grafana.

Step 1 – Install and configure Prometheus Server in RHEL 7

Download Prometheus
Download Prometheus from https://prometheus.io/download/. You can find 3 Prometheus package which is supported for darwin/linux and windows.

$ sudo -s
$ yum install wget -y
$ cd /opt/
$ wget https://github.com/prometheus/prometheus/releases/download/v2.3.2/prometheus-2.3.2.linux-amd64.tar.gz
$ tar -zxvf prometheus-2.3.2.linux-amd64.tar.gz
$ cd prometheus-2.3.2.linux-amd64

If you find following files….

[root@ip-172-31-21-164 prometheus-2.3.2.linux-amd64]# ls -1
LICENSE
NOTICE
console_libraries
consoles
prometheus
prometheus.yml
promtool

This completes the Prometheus installation.

Verify the installation by typing in the following command:

[root@ip-172-31-21-164 prometheus-2.3.2.linux-amd64]# ./prometheus --version
prometheus, version 2.3.2 (branch: HEAD, revision: 71af5e29e815795e9dd14742ee7725682fa14b7b)
build user: root@5258e0bd9cc1
build date: 20180712-14:02:52
go version: go1.10.3

Step 2 – Install and configure Prometheus Node Exporter in RHEL 7

In order to monitor the metrics of your RHEL server, you should install a tool called Node Exporter. Node Exporter, as its name suggests, exports lots of metrics (such as disk I/O statistics, CPU load, memory usage, network statistics, and more) in a format Prometheus understands.

cd /opt/
wget https://github.com/prometheus/node_exporter/releases/download/v0.16.0/node_exporter-0.16.0.linux-amd64.tar.gz
tar -zxvf node_exporter-0.16.0.linux-amd64.tar.gz
cd node_exporter-0.16.0.linux-amd64
nohup ./node_exporter &

Step 3 — Running Node Exporter as a Service
if you want to Running Node Exporter as a Service

$ sudo vi /etc/systemd/system/node_exporter.service

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=root
Group=root
Type=simple
ExecStart=/root/prometheus/node_exporter/node_exporter

[Install]
WantedBy=multi-user.target

# Save the file and exit the text editor.

Reload systemd so that it reads the configuration file you just created.

$ sudo systemctl daemon-reload

Enable it so that it starts automatically at boot time.

$ sudo systemctl enable node_exporter.service

You can now either reboot your server, or use the following command to start the service manually:

$ sudo systemctl start node_exporter.service

Once it starts, use a browser to view Node Exporter’s web interface, which is available at http://your_server_ip:9100/metrics. You should see a page with a lot of text:

Step 4 — Starting Prometheus Server

Enter the directory where you installed the Prometheus server:

$ cd /opt/prometheus-2.3.2.linux-amd64

Before you start Prometheus, you must first understand a configuration file for it called prometheus.yml.

[root@ip-172-31-21-164 prometheus-2.3.2.linux-amd64]# more prometheus.yml
# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
# - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# - "first_rules.yml"
# - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'

# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.

static_configs:
- targets: ['localhost:9090']

Whereas, This creates a scrape_configs section and defines a job called node. It includes the URL of your Node Exporter’s web interface in its array of targets. The scrape_interval is set to 15 seconds so that Prometheus scrapes the metrics once every fifteen seconds.

You could name your job anything you want, but calling it “node” allows you to use the default console templates of Node Exporter. Save the file and exit.

Start the Prometheus server as a background process.

$ nohup ./prometheus > prometheus.log 2>&1 &

You can view the last few lines of the file using the tail command:

$ tail prometheus.log

Once the server is ready, you will see the following messages in the file:

level=info ts=2018-08-21T11:47:38.56459183Z caller=web.go:415 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2018-08-21T11:47:38.569742672Z caller=main.go:543 msg="TSDB started"
level=info ts=2018-08-21T11:47:38.569767659Z caller=main.go:603 msg="Loading configuration file" filename=prometheus.yml
level=info ts=2018-08-21T11:47:38.57069388Z caller=main.go:629 msg="Completed loading of configuration file" filename=prometheus.yml
level=info ts=2018-08-21T11:47:38.570724527Z caller=main.go:502 msg="Server is ready to receive web requests."

Use a browser to visit Prometheus’s homepage available at http://your_server_ip:9090. You’ll see the following homepage.

Step 5 — Verify Prometheus Server
To make sure that Prometheus server is scraping data from Node Exporter, click on the Graph tab at the top of the page. On the page that opens, type in the name of a metric (like up, for example) in the text field that says Expression. Then, press the blue Execute button. Click Graph (next to Console) just below, and you should see a graph for that metric:

More metric can be found from the Node Exporter host such as
http://X.X.X.X:9100/metrics

Prometheus has console templates that let you view graphs of a few commonly used metrics. These console template are accessible only if you set the value of job_name to “prometheus” in Prometheus’s configuration.

Visit http://your_server_ip:9090/consoles/prometheus.html to access the Node Console and click on your server, localhost:9100, to view its metrics.

Step 6 — Installing PromDash Or Grafana…..

Tagged : / / / / /

Install and Configure Grafana in Ubuntu and Debian

Install and Configure Grafana in Ubuntu and Debian

Step 1 – Download & Install Grafana
Download Grafana RPM file RPM for Linux from https://grafana.com/grafana/download?platform=linux

# Ubuntu & Debian
$ wget https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana_5.2.2_amd64.deb
$ sudo dpkg -i grafana_5.2.2_amd64.deb

Step 2 – Understand Grafana Installation details in Ubuntu/Debian

  1. Installs binary to /usr/sbin/grafana-server
  2. Installs Init.d script to /etc/init.d/grafana-server
  3. Creates default file (environment vars) to /etc/default/grafana-server
  4. Installs configuration file to /etc/grafana/grafana.ini
  5. Installs systemd service (if systemd is available) name grafana-server.service
  6. The default configuration sets the log file at /var/log/grafana/grafana.log
  7. The default configuration specifies an sqlite3 db at /var/lib/grafana/grafana.db
  8. Installs HTML/JS/CSS and other Grafana files at /usr/share/grafana

Step 3 – Start the server (init.d service)

$ sudo service grafana-server start

This will start the grafana-server process as the grafana user, which was created during the package installation. The default HTTP port is 3000 and default user and group is admin.

Step 4 – Configure the Grafana server to start at boot time

$ sudo update-rc.d grafana-server defaults
$ systemctl daemon-reload
$ systemctl start grafana-server
$ systemctl status grafana-server
$ sudo systemctl enable grafana-server.service

Step 5 – Grafana server Environment file
The systemd service file and init.d script both use the file located at /etc/default/grafana-server for environment variables used when starting the back-end. Here you can override log directory, data directory and other variables.

Step 6 – Grafana server Log
By default Grafana will log to /var/log/grafana

Step 7 – Grafana Database
The default configuration specifies a sqlite3 database located at /var/lib/grafana/grafana.db.

You can also use MySQL or Postgres as the Grafana database, as detailed on
http://docs.grafana.org/installation/configuration/#database

Step 7 – Grafana configuration
The configuration file is located at /etc/grafana/grafana.ini. Go the Configuration page for details on all those options. You can add following data sources

  1. Graphite
  2. InfluxDB
  3. OpenTSDB
  4. Prometheus

Step 8 – Server side image rendering
Server side image (png) rendering is a feature that is optional but very useful when sharing visualizations, for example in alert notifications.

$ yum install fontconfig
$ yum install freetype*
$ yum install urw-fonts

Step 9 – How to access the Grafana dashboard

http://13.232.27.156:3000/
User name – admin
Password – admin

Tagged : / / / /

Install and Configure Grafana in RHEL 7

Install and Configure Grafana in RHEL 7

Step 1 – Download & Install Grafana
Download Grafana RPM file RPM for Linux from https://grafana.com/grafana/download?platform=linux

# RHEL 7

$ sudo yum install initscripts fontconfig -y
$ wget https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-5.2.2-1.x86_64.rpm
$ sudo yum localinstall grafana-5.2.2-1.x86_64.rpm

Step 2 – Understand Grafana Installation details in RHEL/CENTOS

  1. Installs binary to /usr/sbin/grafana-server
  2. Copies init.d script to /etc/init.d/grafana-server
  3. Installs default file (environment vars) to /etc/sysconfig/grafana-server
  4. Copies configuration file to /etc/grafana/grafana.ini
  5. Installs systemd service (if systemd is available) name grafana-server.service
  6. The default configuration uses a log file at /var/log/grafana/grafana.log
  7. The default configuration specifies an sqlite3 database at /var/lib/grafana/grafana.db

Step 3 – Start the server (init.d service)

$ sudo service grafana-server start

This will start the grafana-server process as the grafana user, which is created during package installation. The default HTTP port is 3000, and default user and group is admin.

Step 4 – Configure the Grafana server to start at boot time

$ sudo /sbin/chkconfig --add grafana-server
$ sudo systemctl enable grafana-server.service
$ systemctl daemon-reload
$ systemctl start grafana-server
$ systemctl status grafana-server

Step 5 – Grafana server Environment file
The systemd service file and init.d script both use the file located at /etc/sysconfig/grafana-server for environment variables used when starting the back-end. Here you can override log directory, data directory and other variables.

Step 6 – Grafana server Log
By default Grafana will log to /var/log/grafana

Step 7 – Grafana Database
The default configuration specifies a sqlite3 database located at /var/lib/grafana/grafana.db. Please backup this database before upgrades.

You can also use MySQL or Postgres as the Grafana database, as detailed on
http://docs.grafana.org/installation/configuration/#database

Step 8 – Grafana configuration
The configuration file is located at /etc/grafana/grafana.ini. Go the Configuration page for details on all those options. You can add following data sources

  1. Graphite
  2. InfluxDB
  3. OpenTSDB
  4. Prometheus

Step 9 – Server side image rendering
Server side image (png) rendering is a feature that is optional but very useful when sharing visualizations, for example in alert notifications.

$ sudo yum install fontconfig -y
$ sudo yum install freetype* -y
$ sudo yum install urw-fonts -y


Step 10 – Browse the dashboard
http://X.X.X.X.:3000/
Username – admin
Password – admin

Tagged : / / / / /

Top 10 Container(Docker) Monitoring Solutions and Tools in 2018

Top 10 Container Monitoring Solutions/Tools in 2018

  1. Native Docker
  2. cAdvisor
  3. Scout
  4. Pingdom
  5. Datadog
  6. Sysdig
  7. Prometheus
  8. Heapster / Grafana
  9. ELK stack
  10. Sensu

Reference

https://rancher.com/comparing-10-container-monitoring-solutions-rancher/

Tagged : / / / /

How to compile and build Gerrit Plugins?

To build Gerrit Plugins from source, you need:

A Linux or macOS system (Windows is not supported at this time)

zip, unzip, wget

$yum install zip -y
$ yum install unzip -y
$ yum install wget -y
$ yum install git -y

Python 2 or 3
This is installed in each RHEL 7 and Ubunutu server by defaul.

Node.js

curl --silent --location https://rpm.nodesource.com/setup_8.x | sudo bash -
OR
curl --silent --location https://rpm.nodesource.com/setup_10.x | sudo bash -
sudo yum -y install nodejs

Bazel

## RHEL/CentOS 7 64-Bit ##
$ wget https://copr.fedorainfracloud.org/coprs/vbatts/bazel/repo/epel-7/vbatts-bazel-epel-7.repo
$ cp vbatts-bazel-epel-7.repo /etc/yum.repos.d/
$ yum install -y bazel

How to Installing Bazel on Ubuntu?
https://docs.bazel.build/versions/master/install-ubuntu.html

Maven

$ cd /opt
$ wget http://www-us.apache.org/dist/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.zip
$ unzip apache-maven-3.5.4-bin.zip
$ mv apache-maven-3.5.4 maven
$ export PATH=$PATH:/op/maven/bin

gcc

$ sudo yum install gcc-c++ make

Now, Bazel in tree driven means it can only be built from within Gerrit tree. Clone or link the plugin into gerrit/plugins directory:

# First become a non-root user

A JDK for Java 8

$ cd
$ wget -c --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u181-b13/96a7b8442fe848ef90c96a2fad6ed6d1/jdk-8u181-linux-x64.tar.gz
$ tar -xvf jdk-8u181-linux-x64.tar.gz
$ export JAVA_HOME=/home/ec2-user/jdk1.8.0_181
$ java -version

Follow for Gerrit.war

$ git clone --recursive https://gerrit.googlesource.com/gerrit
$ cd gerrit 
$ bazel build release

Follow for plugins such as its-jira

$ cd plugins
$ git clone https://gerrit.googlesource.com/plugins/its-jira
$ git clone https://gerrit.googlesource.com/plugins/its-base
$ bazel build plugins/its-jira

The output can be normally found in the following directory:

bazel-genfiles/plugins/its-jira/its-jira.jar

# Some plugins describe their build process in src/main/resources/Documentation/build.md file. It may worth checking.

# Some plugins cane be build using maven as well

Reference

  • https://gerrit-review.googlesource.com/Documentation/dev-bazel.html
  • https://gerrit.googlesource.com/gerrit/
  • https://gerrit-review.googlesource.com/Documentation/cmd-plugin-install.html
  • https://gerrit-review.googlesource.com/Documentation/dev-build-plugins.html
Tagged : / / / /

What is “Install Verified label” in Gerrit?

What is “Install Verified label” in Gerrit?

The Verified label was originally invented by the Android Open Source Project to mean ‘compiles, passes basic unit tests’. Some CI tools expect to use the Verified label to vote on a change after running.

During site initialization the administrator may have chosen to configure the default Verified label for all projects. In case it is desired to configure it at a later time, administrators can do this by adding the following to project.config in All-Projects:

[label “Verified”]
function = MaxWithBlock
value = -1 Fails
value = 0 No score
value = +1 Verified
copyAllScoresIfNoCodeChange = true
The range of values is:

-1 Fails
Tried to compile, but got a compile error, or tried to run tests, but one or more tests did not pass.
Any -1 blocks submit.

0 No score
Didn’t try to perform the verification tasks.

+1 Verified
Compiled (and ran tests) successfully.
Any +1 enables submit.

For a change to be submittable, the change must have a +1 Verified in this label, and no -1 Fails. Thus, -1 Fails can block a submit, while +1 Verified enables a submit.

Additional values could also be added to this label, to allow it to behave more like Code-Review (below). Add -2 and +2 entries to the label.Verified.value fields in project.config to get the same behavior.

As an example, the popular gerrit-trigger plugin for Jenkins/Hudson can set labels at:

  • The start of a build
  • A successful build
  • An unstable build (tests fails)
  • A failed build

Usually the range chosen for this verdict is the Verified label. Depending on the size of your project and discipline of involved developers you might want to limit access right to the +1 Verified label to the CI system only. That way it’s guaranteed that submitted commits always get built and pass tests successfully.

If the build doesn’t complete successfully the CI system can set the Verified label to -1. However that means that a failed build will block submit of the change even if someone else sets Verified +1. Depending on the project and how much the CI system can be trusted for accurate results, a blocking label might not be feasible. A recommended alternative is to set the label Code-review to -1 instead, as it isn’t a blocking label but still shows a red label in the Gerrit UI. Optionally, to enable the possibility to deliver different results (build error vs unstable for instance), it’s also possible to set Code-review +1 as well.

If pushing new changes is granted, it’s possible to automate cherry-pick of submitted changes for upload to other branches under certain conditions. This is probably not the first step of what a project wants to automate however, and so the push right can be found under the optional section.

Suggested access rights to grant, that won’t block changes:
Read on ‘refs/heads/*’ and ‘refs/tags/*’
Label: Code-Review with range ‘-1’ to ‘0’ for ‘refs/heads/*’
Label: Verified with range ‘0’ to ‘+1’ for ‘refs/heads/*’

Optional access rights to grant:
Label: Code-Review with range ‘-1’ to ‘+1’ for ‘refs/heads/*’
Push to ‘refs/for/refs/heads/*’

Reference
https://gerrit-review.googlesource.com/Documentation/config-labels.html#label_Verified
https://gerrit-review.googlesource.com/Documentation/access-control.html#examples_cisystem
https://groups.google.com/forum/#!topic/repo-discuss/FdN29piSmEQ

Tagged : /

What is Enable signed push support in Gerrit?

This options Defaults to false.

This ensure When a client pushes with git push –signed, this ensures that the push certificate is valid and signed with a valid public key stored in the refs/meta/gpg-keys branch of All-Users.

If true, server-side signed push validation is enabled.

Config in gerrit.config – receive.enableSignedPush

Tagged : / / / / /

Importannce of Canonical web url in Gerrit

The canonical web url must be set. Optional base URL for repositories available over the anonymous git protocol. For example, set this to git://mirror.example.com/base/ to have Gerrit display patch set download URLs in the UI. Gerrit automatically appends the project name onto the end of the URL.

By default unset, as the git daemon must be configured externally by the system administrator, and might not even be running on the same host as Gerrit.

Tagged : /

How to backup and restore Gerrit server?

How to backup and restore gerrit server?

There are 3 coponent which should be backed up in gerrit

  1. Repository – According to me best way to backup the repository is to setup a replication with other gerrit hosting tools using gerrit replication plugins. The steps can be find as below;
  2. Gerrit Database
    Depends on the database, you should take the database backup. It can be H2 or mysql….
  3. Gerrit Config
    Rysnc is the best tools to take the entire gerrit site backup.

How to replicate Gerrit repository using replication plugins?

Step 1- Setup Gerrit Server
http://www.devopsschool.com/tutorial/gerrit/gerrit-install-and-configuration.html

Step 2 – Create a Project in Gerrit

Step 3 – Setup a Developement Machine
git clone http://admin@35.154.81.167:8080/a/prj1 && (cd prj1 && curl -kLo `git rev-parse –git-dir`/hooks/commit-msg http://admin@35.154.81.167:8080/tools/hooks/commit-msg; chmod +x `git rev-parse –git-dir`/hooks/commit-msg)

Step 4: Sample Commits to be done
> touch file1.txt;git add .;git commit -m”adding first version”

Step 5: Sample push and submit it
> git push origin HEAD:refs/for/master

Step 6: create it $site_path/etc/replication.config

Content of the files is –
[remote “github”]
url = git@github.com:scmgalaxy/${name}.git

Within each URL value the magic placeholder `${name}` is replaced with the Gerrit project name.

Step 7: Generate a public/private key

> ssh-keygen -t rsa

Step 8: create a “config” under /root/.ssh

Host github.com
User git
IdentityFile /root/.ssh/id_rsa
StrictHostKeyChecking no
UserKnownHostsFile /dev/null

Step 9: Update the public key to github

Step 10: Create a repo in Github.com with same name.

Tagged : / / /