| - | - | NOCOL-
- Design and Internals |
-
| - | - | - |
-
| Overview - ¤ Design principles - ¤ Architecture - User Interfaces |
- - | Overview-NOCOL is a system and network monitoring software which runs on Unix platforms and - monitors reachability, ports, routes, system resources, etc. It is modular in design, and - allows adding new monitors easily and without impacting other portions of the software- in - fact, a large number of the monitors are contributed by various NOCOL users. -The basic design principles behind NOCOL are relatively few:
It might seem strange, but the initial versions of SunNet Manager, CiscoWorks, etc. had - none of the above features when NOCOL was originally written. Most of the commercial - packages required pretty extensive hardware to run, and it seemed like they sacrificed a - lot in order to present a pretty graphical interface. Nocol in contrast could run on a - very low end system, the monitors could be separated from the logging and reporting - machine since they communicated over a network, and the datapoints collected were very - small in number since they were only recorded when the severity of a device/variable - changed. So, the disk space on a machine could vary from 10% to 60% full, and only one - entry is logged since this would all be considered 'normal'. If the disk becomes 80% full, - a 'warning' message is generated and another datapoint is logged. This simple - approach gives amazingly small volumes of data, and yet presents a perfectly comprehensive - (though quantized) report on the variables. -The architecture of nocol itself is very simple- the monitors poll the devices and - assign a threshold to each 'poll' (called an 'event'). These thresholds are user settable - and vary from monitor to monitor (in fact, the rest of the software does not care what is - being monitored and does not store any intelligence about the variables). All intelligence - of the variable being monitored and what conditions are to normal and abnormal is built - into the monitor. -
The monitor would then set:
and invoke a nocol API function. This writes out the current values, etc. to a realtime - data file on disk (this contains the current state of any device/variable) and if the - severity has changed, then this also gets logged to a incremental logging daemon - (noclogd). -All user displays can then display the data from the realtime data directory, whereas - the alarm and notification subsystem gets activated by the incremental 'noclogd' - process. The noclogd process can filter events based on user defined criteria, and invoke - an SMS pager, send email, perhaps even run some automated tests or open a trouble ticket. -This simple architecture has proven to work very effectively in this application. The - base system has not really changed since the software was initially written, but new - monitors, displays, notification software is continually being added without any changes - to the core system. -- The Monitors-The monitors collect variable values and compares to see if it exceeds any of the 3 - thresholds (warning, error, critical- these thresholds are user configurable). This is all - done using the nocol library functions, so in effect, all the monitor needs to do is get - the value for the variable being monitored and read the thresholds from a config file. The - nocol library ensures consistency in the way that the monitoring is processed by the rest - of the system. -Each monitor is unique in the way that it monitors its respective variable. The DNS - monitor needs to make an authoritative DNS query to see if the dns server is configured - properly, the Port monitor needs to connect to TCP or UDP ports to ensure that any - processes are responding properly, and the SNMP monitor needs to monitor snmp variables - using the SNMP protocol. The intelligence about the entity being monitored and how to - monitor it lies strictly in the monitor- the rest of the nocol subsystem is just - expecting a device name, variable and its value. -A fair amount of effort has gone into making the monitors very efficient where possible - in order to allow them to scale to a large number of devices. Connectionless (UDP) - monitors are specially well suited to using the select() system call so that many devices - can be queried at the same time and the monitor then waits for the responses to come in. - The other option was to fork multiple processes with a single parent and each process - monitors one device. However, the level of scalability that could be achieved with the - first method proved to be far more than what could be achieved with the forking method. -To emphasize the above, consider 'pinging' 100 devices with 5 packets each, waiting 1 - second for each response and 1 sec between each packet to the same host. If done serially, - this would take at least 500 seconds for each pass. If we fork multiple processes to do it - in parallel, this would take about 5 seconds, but we would have to fork a 100 processes. - The 'multiping' monitor could send out 1 packet to each of the 100 devices in about - 10 seconds and then listen for the responses to come in- effectively taking about 15 - seconds for the entire pass. -Building this level of 'multi-tasking' is a lot more difficult in the TCP based - services since it would require non-blocking I/O, but it it important to do this for - monitors such as 'portmon'. All of these type of monitors (using select()) are limited by - the MAXFD value (maximum open file descriptors that can be handled by the select() call). -The 'hostmon' monitor is an example of letting the remote hosts being monitored - do the local data collection (i.e. distributing the 'time consuming' part to - hostmon-clients). The 'hostmon' process running on the nocol host simply takes all these - data files and uses them as raw input to the process. -In some cases, the monitors do not need any data other than what is in the nocol data - structure written to disk (the raw data), whereas in others they need to store ancillary, - variable and device specific information in memory. All possible efforts are made to avoid - storing unnecessary data in memory and having 'bloated' monitoring processes. -- User Interfaces-The user interfaces need to display the current state of the devices being - monitored, and this 'current' data is stored on disk (in the 'data' directory). This - allows any number of users and monitors to view the same consistent data, and run only one - set of monitors (unlike some other systems which need a separate monitor for each - display). -The other diversion from traditional network monitoring packages is the displaying of - monitored data using text lines and not a map or other graphical interface. The reason - this approach was taken is that in practical experience, a network diagram was always done - in some 'drawing' tool and the map on the NMS was not updated regularly. Even today, most - network/lan diagrams are maintained in a tool such as Visio, and the NMS graphical - interface is always a 'second' copy. This and being able to view line based data from any - terminal weighed very heavily in favor of a non-graphical user interface. -Netconsole (curses)-Netconsole is a simple Unix 'curses' based TTY interface. It reads the raw data from - disk and formats it for displaying on the screen. It has limited intelligence, and its - method of setting an alarm is when it sees a change in the number of 'down' items. This is - the original user interface and was written to let engineers view the state of the - systems/network over a low speed connection (over which X windows, etc. would not be - feasible). -WebNocol (Web)-Contributed by Rick Beebe, this is a Web based frontend to the datafiles. It - allows running CGI's and troubleshooting listed events and all the other benefits of HTTP. -
This web interface automatically refreshes periodically, plays an audio clip if a site - changes its severity level, etc. A 'status' message can be displayed next to each event - which is inserted by any valid operator. Users are assigned access levels which controls - how much information they can view or edit. -tkNocol (Tcl/Tcl)-This is a client-server application contributed by Lydia Leong. The tkNocol application - connects to the ndaemon process on the host system, and displays the nocol data - in a X-window. -
This interface needs 'tixwish' on the system. Any number of clients can connect - to the simple process (ndaemon) running on the nocol host which sends data to all - the clients periodically. Currently there is no access control configured on the ndaemon - host, so this should be protected by a firewall, but this interface can be extended to add - these features in the daemon if needed. -- Reporting-The monitors in NOCOL generate a 'historical' event (logged to the noclogd - logging daemon) only when the severity of a variable changes (i.e. it goes from warning - to error or from critical back to info. This is done to reduce - the amount of historical data collected and restrict it only to 'relevant' datapoints. - This quantized data storage allows a monitor to poll a device or variable as frequently as - it likes (30 secs, 10 minutes), but it will generate a logging entry only if the variable - crosses one of the thresholds. -This approach of classifying the data into 'bins' reduces the quantity of - historical data significantly. Even though some granularity is lost, statistical analysis - can easily be done on the collected data by using the time interval that a variable - remained in a particular level. -'noclogd' is similar to the Unix 'syslog' process- it allows piping the log message to - an external program or writing to a file based on the monitor name. This forms the - basis of invoking SMS scripts to do paging, sending email, automated insertion into - trouble ticketing systems, auto problem analysis, etc.- the possibilities are virtually - unlimited. -Currently this system writes to flat files, but the data can easily be piped to a - process that writes into a database. Note that the 'current' data that the monitors write - to disk (the raw data which is displayed by the user interfaces) is overwritten in every - pass by the monitor. Hence the size of those files is fixed and does not grow over time. -- Future Work-The package does not interface to any database, and would benefit greatly from storing - the raw (monitored) and noclogd historical information in a database such as MySQL, etc. - This would allow co-relating the various variables being monitored for any device (e.g. - show the state of all variables being monitored for device lan-gw). Graphs and reports - could be charted from the historical noclogd database. -A Java based user interface along the same lines as tkNocol would allow running the - display on any platform and one could build a lot of graphing, reporting functionality - into the gui itself. -The GUI could also support collapsing all the variables for an event in one line, and - only display the variable when it needs to be displayed. -Instead of the current line based GUI, it would be good to be able to display a - map of the network and the devices. The raw data would have no information on coordinates - or drawing, but a separate file could contain all the information necessary to create a - graphical display. As an example, the file could contain coordinates of nodes and edges, - and hierarchical relationships between the devices- the user interface could read this - data and construct the diagram of the network. -In order to make this scalable, it would be useful to allow various NOCOL's to interact - with each other. This is easily doable using the noclogd daemon, since noclogd - can be enhanced to send an event to other noclogd's running on remote hosts. The data can - be isolated and referred to using the 'nodename' to prefix the data/events. -- |
-
| - Vikas Aggarwal - - | -- |
