FOSDEM 2005

Ethan Galstad

www.nagios.org

February 26th, 2005

Presentation Overview

Presentation (40 minutes)

Background and overview

In-depth information on specific topics

New features in 2.0

Future directions

Question & Answer (10 Minutes)

Slides available online:

http://www.nagios.org/fosdem2005

What Is Nagios?

Open Source (GPL) network monitoring application

Runs on Linux, FreeBSD, Solaris, etc.

Monitors hosts and services on your network

Provides you with an overview of the status of your systems

Notifies you know when things are go wrong

Allows you to resolve problems faster

Provides availability reports for SLAs, etc.

How It Started

Development started in January 1999

Wanted to provide monitoring services to local businesses

No suitable (cost- and feature-wise) commercial or free solution was available

Originally designed to monitor small LAN (~20 servers)

Initially anticipated a small user base (~20 users)

Where It Is Now

Six years and nine major versions later

Top installations monitor thousands of hosts and services

Over 150,000 downloads of latest stable version (1.2)

The Name

Originally released under the name “NetSaint” (netsaint.org)

Name changed to “Nagios” in 2001 for trademark reasons

N.A.G.I.O.S. = “Nagios Ain't Gonna Insist On Sainthood”

Trademark registration for Nagios®

I pronouce it as “nah-ghee-ose”

Correct pronunciation: however you'd like

Design Overview

Designed in a modular fashion

Daemon contains the monitoring logic and coordinates things

CGIs allow users to view status information through web browser

External applications handle the low-level monitoring work

External commands can be triggered to manage alerts, state changes, and monitoring information

Can be integrated with 3rd-party applications fairly easily

Nagios Overview

What can be monitored?

Servers, workstations, printers, routers,etc.

More genericly, anything that:

has or is associated with an address of some kind

is “reachable”

Nagios doesn't know or care about network protocols or addresses

Not limited to monitoring standard network equipment and services

Monitoring subjects/targets

Hosts

Services

Nagios Overview

Hosts

Usually physical objects (server, switches, routers, printers, etc.)

Can have parent/child relationships with other hosts

Provide one or more services

Services

Things associated with or provided by a host

Tangible services (e.g. disk usage, printer toner supply)

Intangible services (e.g. HTTP, SMTP, IMAP, POP3, DNS)

Nagios Overview

Example: Small Network Layout

Nagios Overview

Hosts as viewed by Nagios

Nagios Overview

Services as viewed by Nagios

Plugins

How are hosts and services monitored?

Nagios doesn't understand network addresses, protocols, or services

Nagios passes information about what needs to be checked to external commands (plugins)

Plugins perform the actual checks of hosts and services and return information back to Nagios

What are plugins?

Shell or perl scripts, executables, etc.

Understand network addresses, protocols, services, etc.

Plugins

Plugins

What information do plugins return?

Text output

Shown to users via the web interface, in notifications, etc.

Return code

This is what Nagios uses to determine the status of things

Four possible return codes used to indicate different states

0 = OK

1 = WARNING

2 = CRITICAL

3 = UNKNOWN

Service Checks

Services

Services are the main reason we monitor things

If a host offers no useful services, why would we monitor it?

Service checks

Checks are performed at regularly scheduled intervals using plugins

Multiple service checks can be run in parallel

Four possible states: OK, WARNING, CRITICAL, and UNKNOWN

You can monitor anything you can write a plugin for:

Simple: HTTP, IMAP, DNS, disk usage, etc.

More complicated: Web transactions, mail loop detection, etc.

Service Checks

Host Checks

Hosts

“Containers” for services

Host Checks

As with services, checks are performed using plugins

Checks are performed on-demand after service state changes

Critical decision point: is the host or the service the real problem?

Need timely and accurate information on host status

Three possible states: UP, DOWN, and UNREACHABLE

Can trigger a route verification

Host Route Verification

Why might a host not be UP?

The host is DOWN

The route to the host is blocked by one or more other hosts (UNREACHABLE)

Route verification

Determines whether hosts are DOWN or UNREACHABLE

Can be very time-intensive if network problems are widespread

Why is it useful?

Helpful in determining the “real” cause of widespread problems

Different host states may be acted upon differently by processing and notification logic

Host Route Verification

Logical host relationships - The world according to Nagios

Host Route Verification

A problem is detected with 'Port Status' service on Switch2...

Host Route Verification

Switch2 is checked for problems and found to NOT be UP...

Host Route Verification

Check is propagated upstream to Firewall1, which is also NOT UP...

Host Route Verification

Check is propagated upstream to Switch1, which IS found to be UP

Host Route Verification

Reachability of Switch2 can now be determined - Firewall1 is DOWN and Switch2 is UNREACHABLE.

Host Route Verification

Checks are propagated to children of SWITCH2, which are found to be UNREACHABLE

Host Route Verification

Other children of Firewall1 are checked later, found to be UNREACHABLE

External Application Interfaces

External Commands

-

Passive Checks

-

Event Handlers

-

OCSP/OCHP Commands

-

Performance Data

External Commands

Nagios can accept control commands and data from third-party applications through the external command interface

What can external commands be used for?

Controlling aspects of the monitoring and processing logic

Disabling/enabling host and service checks

Disabling/enabling notifications

etc.

Submitting host and service check results

External Commands

How are commands submitted?

Nagios creates a named pipe (FIFO) at startup, reads the file at regular intervals and processes any new commands that are found

CGIs and 3rd-party applications submit commands to Nagios by writing to the external command file

Passive Checks

Methods for checking hosts and services

Actively

Passively

Active checks

Synchronous in nature

Scheduled and initiated by Nagios

Passive Checks

Passive checks

Performed by external applications

Check results are submitted through the external command file

Asynchronous in nature

Results are processed using the same logic as active checks

Ideal for incorporating security alerts

Passive Checks

Integrating security alerts using passive checks

SWATCH

TCP Wrappers

NMAP

Passive Checks

SWATCH Integration

Generate alerts if bad patterns are found in your logfiles

SWATCH is configured to execute a script when pattern is found

Script sends a CRITICAL alert to Nagios by submitting a passive service check result to the external command file

Passive Checks

TCP Wrappers Integration

Generate alerts if remote machines attempt to access restricted services

hosts.deny is configured to execute a script when access is rejected

Script sends a CRITICAL alert to Nagios by submitting a passive service check result to the external command file

Passive Checks

NMAP Integration

Generate alerts if rogue/banned services appear on your LAN

Cron job runs periodically and initiates an NMAP scan for rogue services

If rogue services are detected, send CRITICAL alert to Nagios by submitting a passive service check result to the external command file

If no rogue services are detected, send an OK alert to Nagios

Event Handlers

What are event handlers?

External commands that are executed during state changes

Can be defined for both hosts and services

Global and host- or service-specific event handlers available

What can they be used for?

Passing state change information to other applications

Restarting failed services

Automatically entering trouble tickets into a helpdesk system

etc.

Event Handler Logic

OCSP/OCHP Commands

What's with the names?

OCSP = “Obsessive Compulsive Service Processor”

OCHP = “Obsessive Compulsive Host Processor”

What are they?

External commands that are optionally executed after:

every service check (OCSP command)

every host check (OCHP command)

What are they used for?

Passing check results to external applications for processing

Performance Data

What is performance data?

Optional metrics returned by plugins after host or service checks

How is performance data processed?

After every host or service check, performance data can either be written to files or passed to external commands

External applications can then process the data (for graphing, etc.)

Advanced Topics

Distributed Monitoring

-

Redundant Monitoring

-

Failover Monitoring

Distributed Monitoring

Useful when:

You need to distribute monitoring load over multiple servers

Remote hosts/services are behind a firewall

What's Involved?

1 central monitoring server

1 or more distributed monitoring servers

Nagios features used in configuration:

Passive checks

OCSP command

Distributed Monitoring

Redundant Monitoring

Purpose:

Backup monitoring server in case primary server fails or goes offline

Ensures notifications are sent out to contacts

Key points:

Slave host is configured with the same set of services as master, plus an extra one to check master Nagios process

Master and slave hosts monitor shared services simultaneously

Only one server (the current master) can send notifications out

Redundant Monitoring

Failover Monitoring

Purpose:

Same as with redundant monitoring, but common services should only be monitored by one server at any given time

Key points:

Master and slave hosts both configured with the same set of services

Slave server doesn't actively monitor services while master is running

Only one server (the current master) can send notifications out

Failover Monitoring

New and Upcoming Features

New Features in Nagios 2.0

-

Plans for Nagios 3.0

-

Long-Term Direction

New Features in Nagios 2.0

Two important additions

Adaptive monitoring

Nagios Event Broker (NEB) API

Adaptive Monitoring

What is it?

Allows you to modify how, what, and when things are monitored and processed on-the-fly

What can it be used for?

Specialized monitoring cases

Possible applications in redundant and failover monitoring

What can be changed?

Global event handler commands

Service and host check and event handler commands, check interval, and max check attempts

Adaptive Monitoring

Event Broker API

What is it?

Method for allowing third-party loadable modules to be integrated with Nagios during runtime

Loadable modules can process Nagios event and status data in real-time

What can it be used for?

Internal processing of performance data

Storing status and event data in external database (MySQL, etc.)

Lots of other applications

Event Broker API

How does it work?

Third-party C/C++ code is compiled as an object module (*.o)

Nagios is configured to use the module

Event Broker API

How it works cont'd

At startup, Nagios loads the module and initializes it by calling the module's nebmodule_init() function

Module performs startup functions, registers for Nagios events using callback functions

Event Broker API

How it works cont'd

When Nagios encounters certain events and processes data, it passes the event or status data to the modules callback function

Module can do whatever it likes with the data passed to it

At shutdown, Nagios calls the module's nebmodule_deinit() function and unloads it

Planned Features for Nagios 3.0

Event broker enhancements

Allow modules to intercept system commands

Allow modules to intercept service checks and perform them internally, rather than let Nagios perform them externally

Spin web interface off as seperate project

Move to using PHP or Perl instead of C for CGIs

Current CGIs retired when new interface is ready

Plugin changes

Allow for longer text output (current limit is approx. 350 chars)

Allow for multi-line output

Maintain backward compatability with current plugins

Long-Term Directions

Through monitoring we have a wealth of information

What can we do with all this information?

Prediction - Being proactive, rather than reactive

Prediction Types

Prediction of usage trends (capacity planning)

Prediction of events

Prediction Challenges

Analyzing data and developing rules

Refining rules over time

“Active observer” problem

Wrapup

All slides available online

http://www.nagios.org/fosdem2005

Nagios support options

Documentation

Mailing lists (2000+ members on the users list)

Commercial support and consulting

http://www.findopensourcesupport.com

Ethan Galstad

2005-01-17T15:01:38

Ethan Galstad