Logstash explained in 5 mins

What is Logstash?
Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite “stash.”

Logstash Benefits

  • Logstash allows you to easily ingest unstructured data from a variety of data sources including system logs, website logs, and application server logs.
  • Logstash offers pre-built filters, so you can readily transform common data types, index them in Elasticsearch, and start querying without having to build custom data transformation pipelines.
  • With over 200 plugins already available on Github, it is likely that someone has already built the plugin you need to customize your data pipeline.

Logstash work in 3 phases….

Phase 1 – When it comes from ingests data from a multitude of sources simultaneously, which includes files, s3,beats, kafka etc. Data is often scattered or siloed across many systems in many formats. Logstash supports a variety of inputs that pull in events from a multitude of common sources, all at the same time.
List of sources from where logstash can ingest the data are as follows;
https://www.elastic.co/guide/en/logstash/current/input-plugins.html

Phase 2 – Next, It Parse & Transform Your Data On the Fly. As data travels from source to store, Logstash filters parse each event, identify named fields to build structure, and transform them to converge on a common format for easier, accelerated analysis and business value. Logstash dynamically transforms and prepare your data regardless of format or complexity

Phase 3 – Last, Logstash stored the parsed data into Elasticsearch, aws,hadoop, Mongodb and go-to output that opens up a world of search and analytics possibilities.Logstash has a variety of outputs that let you route data where you want, giving you the flexibility to unlock a slew of downstream use cases. Some of these are given below;
https://www.elastic.co/guide/en/logstash/current/output-plugins.html

Where you can use the Logstash?

  1. Log Analytics – Ingest un-structured and semi-structured logs generated by servers, applications, mobile devices, and more for a wide variety of applications such as digital marketing, application monitoring, fraud detection, ad tech, gaming, and IoT. Logstash provides plugins to quickly load data from a variety of data sources.
  2. IT Operations Monitoring – Capture server logs and push them into your Elasticsearch cluster using Logstash. Elasticsearch indexes the data and makes it available for analysis in near real-time (less than one second). You can then use Kibana to visualize the data and perform operational analyses like identifying network issues and disk I/O problems. Your on-call teams can perform statistical aggregations to identify root cause and fix issues.
Tagged :

Log Management Tools using splunk and other alternatives

Splunk

Splunk (the product) captures, indexes and correlates real-time data in a searchable repository from which it can generate graphs, reports, alerts, dashboards and visualizations.
Splunk aims to make machine data accessible across an organization and identifies data patterns, provides metrics, diagnoses problems and provides intelligence for business operation. Splunk is a horizontal technology used for application management, security and compliance, as well as business and web analytics.Splunk has over 7,900 customers worldwide.

Graylog2 + Elasticsearch + MongoDB

based on elastic search and Open Source. Graylog2 has powerful real-time text-based search, basic analytics, dashboards, and flexible alerts. Graylog2 is under heavy development. Graylog2 is Java. Both are no real equivalent to SPLUNK, SPLUNK is multiple years ahead, but it’s possible that these tools are also meet your requirement. Graylog2 is an open source log management system. It’s built with Ruby on Rails and MongoDB and uses its own JSON-based log format called GELF (Graylog Extended Log Format).

Apache Flume + Elastic Search + Apache Lucene
Cloudera CTO Jeff Hammerbacher suggests using Apache Flume, an open source log aggregation tool, and Elastic Search, an open source search system based on Apache Lucene, to create an open source log file search system

Fluentd + Elasticsearch + Kibana

Elasticsearch is an open source search engine known for its ease of use.
Kibana is an open source Web UI that makes Elasticsearch user friendly for marketers, engineers and data scientists alike.
Fluentd is an open source data collector, which lets you unify the data collection and consumption for a better use and understanding of data.
By combining these three tools (Fluentd + Elasticsearch + Kibana) we get a scalable, flexible, easy to use log search engine with a great Web UI that provides an open-source Splunk alternative, all for free.

Logstash + Elasticsearch + Kibana

logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). Speaking of searching, logstash comes with a web interface for searching and drilling into all of your logs.It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.setting it up is way more complicated than setting up Splunk, but it offers similar – but not as complete – feature set. Elasticsearch is an open source search engine known for its ease of use. Kibana setting it up is way more complicated than setting up Splunk, but it offers similar – but not as complete – feature set.

Elasticsearch <=> Indexer
Logstash <=> Heavy Forwarder
Kibana <=> Search Head

OpenTSDB
OpenTSDB is an open source database sponsored by StumbleUpon. It’s designed for aggregating and visualizing machine data. It’s built on top of Apache Hbase and boasts the ability to “collect many thousands of metrics from thousands of hosts and applications, at a high rate (every few seconds).”

Sumo Logic
Sumo Logic is a cloud-based log management and analytics service that leverages machine-generated big data to deliver real-time IT insights

Reference – 
http://softwarerecs.stackexchange.com/questions/16/free-equivalent-for-splunk-log-manager
http://docs.fluentd.org/articles/free-alternative-to-splunk-by-fluentd
http://www.infoworld.com/article/2610524/log-analysis/splunk-feels-the-heat-from-stronger–cheaper-open-source-rivals.html

http://kmlbblog.co.in/entries/general/splunk-vs-elk-elasticsearch-logstash-kibana

 

Tagged : / / / / / / / / / / / /

Top 5 Log Monitoring Tools | List of Log Monitoring Tools | scmGalaxy

These days there are various kinds of tools used by professionals in Software industry. From CI tools to Virtualization, to Issues, bug tracking tools to various others tools, the lists goes on and on. Thus, it’s really a tough task for them to select best tools as per their needs. So, today I am going to help System Administrators and operations by providing them the lists of top log monitoring tools.
But before that lets see, what is log monitoring ?
Log monitoring is a process of monitoring and understanding the state of the system, server, network, security devices and different applications and how they are working. Actually when these servers and different applications runs, they generate log files and events like Errors, problems, and more information is constantly logged and saved for analysis as text or binary files in the system. System admin keep reviewing these log files on daily basis so they can understand the condition and functioning of systems and different applications. But reviewing all those log files on daily basis is not an easy task, to do this laborious task log monitor needs to be set on those logs which automatically
monitor, review, analyze and generate reports as per the instructions set by System Admin.
As you saw how helpful this is for system admin but to find a good tool for this process is not an easy task. So, without wasting any time let’s check out the top 5 monitoring tools.
1. Logstash
logstash
Features
  • Extended functionality via plugins.
  • Free and open source.
  • Filters are code.
  • Great integration with other Elastic products.

2. Fluentd
fluentd
Features
  • Handles up to 50,000 messages per second at peak time
  • Data filtering and alerting
  • Apache 2.0 License project
  • Simplify and scale data pipeline management with tags
  • More than 200 community-contributed plugins
  • Real-time monitoring and alerts system
  • Store data in multiple systems
  • Collect and correlate web server access logs and application error logs

 

3. Loggly

loggly
Features
  • Text-based logs from any source
  • Custom source groups
  • Point-and-click trending graphs
  • Automated filters and event parsing
  • Full-system RESTful API to integrate with other applications
  • Unlimited saved searches and users
  • Adaptable interface with multiple views, pages and workspaces
  • Unlimited custom dashboards based on any search
  • Built-in customizable alerts with triggers

4. Graylog2
graylog2
Features
  • Leverages Java, Scala and ElasticSearch technologies
  • Central syslog monitoring
  • Interactive API browser
  • Application debugging
  • Exception monitoring
  • API analytics
  • Intuitive search interface
  • Comprehensive dashboard

5. Splunk
splunk
Features
  • Works with Hadoop & NoSQL
  • Setup standard searches as real-time alerts
  • Trigger automatic responses
  • Scale from a single server to multiple data centers
  • Deployable to on premise sites, hybrid-cloud, and/or private and public cloud based infrastructures
  • Securely make data available
  • Supports various use cases including log consolidation/retention, security, compliance reporting, and more
  • Perform ad hoc queries
So, this is the list my top log monitoring tools, I hope you find this useful and if not than feel free to share your thoughts in comment section below.
Tagged : / / / / / / / / / / / / / / /

Top 10 DevOps Tools which is mostly used by DevOps Engineers | scmGalaxy

top-10-devops-tools
DevOps is an important component for software industry today. Developing and implementing a DevOps culture helps to focus IT results and to save time and money as the gap between developers and IT operations teams closes. Just as the term and culture are new, so are many of the best DevOps tools these DevOps engineers use to do their jobs efficiently and productively. To help you in your DevOps process, we have searched and created this list of DevOps tools which is mostly used by DevOps Engineers in their projects.

1. Chef

devops-tool-chef

Chef is an extremely popular tool among DevOps engineers. From IT automation to configuration management, Chef relies on recipes and resources so you can manage unique configurations and feel secure knowing Chef is checking your nodes and bringing them up to date for you.
Key Features:
  • Manage nodes from a single server
  • Cross-platform management for Linux, Windows, Mac OS, and more
  • Integrates with major cloud providers
  • Premium features available

2. Jenkins

devops-tool-jenkins

 

An extensible continuous integration engine, Jenkins is a top tool for DevOps engineers who want to monitor executions of repeated jobs. With Jenkins, DevOps engineers have an easier time integrating changes to projects and have access to outputs to easily notice when something goes wrong.
Key Features:
  • Permanent links
  • RSS/email/IM integration
  • After-the-fact tagging
  • JUnit/TestNG test reporting
  • Distributed builds
3. Puppet

devops-tool-puppet

Puppet is an open-source configuration management tool. It runs on many Unix-like systems as well as on Microsoft Windows, and includes its own declarative language to describe system configuration. DevOps engineers often rely on Puppet for IT automation. Get a handle on configuration management and software while making rapid, repeatable changes with Puppet.
Key Features:
  • Automatically enforce consistency of environments
  • Works across physical and virtual machines
  • A common tool-chain
  • Support key DevOps best practices, including continuous delivery

4. Ant

 

devops-tool-ant

A Java library and command-line tool, Apache Ant looks “to drive processes described in build files as targets and extension points dependent upon each other.” This build automation tool is one that saves DevOps engineers a great deal of time.
Key Features:
  • Supplies a number of built-in tasks for compiling, assembling, testing, and running Java applications
  • Builds non-Java applications, such as C or C++ applications
  • Pilot any type of process which can be described in terms of targets and tasks
  • Extremely flexible and does not impose coding conventions or directory layouts to the Java projects which adopt it as a build tool

5. Apache Maven

devops-tool-apache-maven

DevOps engineers can manage a project’s build, reporting, and documentation from a central piece of information with Apache Maven. A software project management and comprehension tool, Maven has been a reliable tool for DevOps engineers.
Key Features:
  • Simple project setup follows best practices
  • Easily work with multiple projects at one time
  • Large repository of libraries and metadata that continue to grow
  • Extensible, with the ability to write plugins in Java or scripting languages
6. Logstash

devops-tool-logstash

For open source log processing, search, and analytics, Logstash is a popular tool among DevOps engineers. Because Logstash is licensed under Apache 2.0, you can use it in the way that best suits your needs.

Key Features:
  • Collects, parses, and stores logs for later use
  • Includes a web interface for searching and drilling into all of your logs
  • Ship logs from any source, parse them, timestamp them correctly, index them, and search them

7. Docker

devops-tool-docker

 

An open platform for distributed applications, Docker is an application for DevOps engineers who want to “build, ship, and run any app, anywhere.” With Docker, you can quickly assemble apps from components and work collaboratively.
Key Features:
  • Assemble multi-container apps and run on any infrastructure
  • Compose an app using both proprietary containers and Docker Hub Official Repos
  • Manage all containers of an app as a single group
  • Cluster an app’s containers to optimize resources and provide high-availability
8. New Relic

devops-tool-new-relic

With New Relic APM, DevOps engineers spend less time monitoring applications and more time on building and deploying. A popular, reliable tool, New Relic APM is a great choice for DevOps engineers.
Key Features:
  • Helps in the build, deployment, and maintenance of web software
  • Application monitoring in one place
  • Cross application and transaction tracing
  • Database and availability and error monitoring
9. Gradle

devops-tool-gradle

 

Gradle is a robust tool for automating building, testing, publishing, and deploying software packages and other projects. With the combined power and flexibility of Ant and Maven, Gradle is an open source build automation system which is perfect and very useful for DevOps engineers.
Key Features:
  • Declarative builds and build-by-convention
  • Language for dependency-based programming
  • Structure your build
  • Deep API
  • Multi-project builds
  • Ease of migration
10. Git 

devops-tool-git 

 

Git is a mature, actively maintained open source project originally developed in 2005 by Linus Torvalds, the famous creator of the Linux operating system kernel. Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency.
Key Features:
  • Working offline
  • Fast to Work With
  • Repositories Are Smaller
  • Moving or Adding files
  • Ignore Certain Files
  • Branches
  • Check the Status of Your Changes
  • Stash Branches
  • Cherry Pick Changes from Branches
  • Find version that Introduced a bug using Binary Search
These are the most popular DevOps tools which are used by DevOps engineers or practitioners these days. But to make most out of these tools you need to have proper knowledge of these tools like installation process, implementation process, where to you use, how to use, troubleshooting and much more. So, if you think you need help or training for these tools or for DevOps related helps than we are here to assist you with our industry expertise professionals.
Tagged : / / / / / / / / / / / / / / / / / / / /