Like every other software, CORVUS® and NOCTUA® has also certain system requirements. Because CORVUS® and NOCTUA® is free Software, everybody who has enough programming knowledge could be able to port it to other free software systems.

The good news: You don't have to port anything if you already use one of the following LINUX distributions:

For exact versions please take a look into Chapter 2, Installation

Monitoring configurations are stored in databases for faster access and therefore faster reaction time and further more flexible administration of data.

CORVUS® and NOCTUA®is using Django as its database interface so every database which is compatible with Django can be used. The recommended Database is PostgreSQL (mostly due to license issues), another well tested one would be MySQL.

CORVUS® packages are available for following operating systems:

It might be very well possible to get the software up and running on different platforms - but this type of operation is not tested at all.

If you are running one of the supported operating systems add the repository matching your installation to your package management software. Refer to the manual of your package management software how to do that.

There are 2 main and 1 extra repository you have to deal with to install either CORVUS® or NOCTUA®.

Repositories are available for oldstable (1.x), stable (2.x) and for master (devel) versions of above mentioned operating systems. The system running on your machines and the version of CORVUS® or NOCTUA® you want, determine which repository you must use in your package manager.

Based on the above mentioned operating system, repository and desired software version, resulting repositories can be added.

There are two different ways to add new repositories for monitoring software by init.at™ to the operating system. It can be added all at once in one central file or in a repository directory. For each operating system there are special repository directories.

There are two different ways to add new repositories for monitoring software by init.at™ to the operating system. It can be added all at once in one central file or in a repository directory.

deb http://www.initat.org/cluster/DEBs/debian_wheezy/cluster-devel wheezy main
deb http://www.initat.org/cluster/DEBs/debian_wheezy/monit-devel wheezy main
deb http://www.initat.org/cluster/DEBs/debian_wheezy/extra wheezy main
                        

deb http://www.initat.org/cluster/DEBs/debian_wheezy/cluster-2.0 wheezy main
deb http://www.initat.org/cluster/DEBs/debian_wheezy/monit-2.0 wheezy main
deb http://www.initat.org/cluster/DEBs/debian_wheezy/extra wheezy main
                        

deb http://www.initat.org/cluster/DEBs/debian_wheezy/cluster-1.0 wheezy main
deb http://www.initat.org/cluster/DEBs/debian_wheezy/monit-1.0 wheezy main
deb http://www.initat.org/cluster/DEBs/debian_wheezy/extra wheezy main
                        

For ubuntu 12.04 you can also add repositories either in one file, into /etc/apt/sources.list or in repository directory /etc/apt/sources.list.d

deb http://www.initat.org/cluster/DEBs/ubuntu_12.04/cluster-devel precise main
deb http://www.initat.org/cluster/DEBs/ubuntu_12.04/monit-devel precise main
deb http://www.initat.org/cluster/DEBs/ubuntu_12.04/extra precise main
                    

deb http://www.initat.org/cluster/DEBs/ubuntu_12.04/cluster-2.0 precise main
deb http://www.initat.org/cluster/DEBs/ubuntu_12.04/monit-2.0 precise main
deb http://www.initat.org/cluster/DEBs/ubuntu_12.04/extra precise main
                    

deb http://www.initat.org/cluster/DEBs/ubuntu_12.04/cluster-1.0 precise main
deb http://www.initat.org/cluster/DEBs/ubuntu_12.04/monit-1.0 precise main
deb http://www.initat.org/cluster/DEBs/ubuntu_12.04/extra precise main
                    

Debian and ubuntu use an other package manager than CentOS, OpenSUSE or SLES. For that reason, on rpm based operating systems sources.list does not exist. Rather there are a few files for repository management not only one. All relevant repository files stays in the directory /etc/zypp/repos.d/.

[cluster_devel_remote]
name=cluster_devel_remote
enabled=1
autorefresh=0
baseurl=http://www.initat.org/cluster/RPMs/suse_13.1/cluster-devel
type=rpm-md
                    

[monit_devel_remote]
name=cluster_devel_remote
enabled=1
autorefresh=0
baseurl=http://www.initat.org/cluster/RPMs/suse_13.1/monit-devel
type=rpm-md
                    

[init-extra]
name=init-extra
enabled=1
autorefresh=0
baseurl=http://www.initat.org/cluster/RPMs/suse_13.1/extra
type=rpm-md
                

Repository directory is /etc/yum.repos.d/. Place your desired *.repo files inside this directory, do a yum check-update and you are ready to install CORVUS® / NOCTUA®

 [initat_cluster]
autorefresh=1
enabled=1
type=rpm-md
name=initat_cluster
baseurl=http://www.initat.org/cluster/RPMs/rhel_6.2/cluster-devel

[initat_extra]
autorefresh=1
enabled=1
type=rpm-md
name=initat_extra
baseurl=http://www.initat.org/cluster/RPMs/rhel_6.2/extra

[initat_monit]
autorefresh=1
enabled=1
type=rpm-md
name=initat_monit
baseurl=http://www.initat.org/cluster/RPMs/rhel_6.2/monit-devel                       
                    

Before installing packages make sure to remove the following conflicting packages:

Install the package containing the at daemon before attempting to install CORVUS® packages.

from the newly added repositories. These packages pull in all the necessary dependencies to use CORVUS®. Ignore the output from the package post install scripts on how to populate the database. Setup the following processes to be started at your default runlevel:

Refer to the documentation of your Database System on how to create users and databases.

The database access data is stored in /etc/sysconfig/cluster/db.cf , a sample file is provided under /etc/sysconfig/cluster/db.cf.sample . If you want to connect via local socket leave DB_HOST empty. Fill in the user and database information.

Every daemon and process from CORVUS® is using this file to gain access to the database. The File has to be readable for the following system entities:

A typical set of rights would look like

-rw-r----- 1 wwwrun idg 156 May 7 2013 /etc/sysconfig/cluster/db.cf
                

Nearly every aspect of CORVUS® and NOCTUA® is administrated via the webfrontend, so the next steps after the initial database setup are done there. The following system processes are needed to access the webfrontend:

In order to check if these processes are running simply issue the command /opt/cluster/sbin/check_scripts.py --system memcached nginx uwsgi-init which should give an output similar to

Name       type   status
------------------------
memcached  system running
nginx      system running
uwsgi-init system running
                

The Webfrontend for NOCTUA® can be accessed via http://SERVERNAME/cluster or by http://IP_ADDRESS:18080/cluster/

The Webfrontend for CORVUS® can be accessed via http://SERVERNAME/cluster or by http://IP_ADDRESS:8080/cluster/

For booting nodes you have to have an NFS server up and running. The entries in your /etc/exports file will be added automatically by the cluster software.

Following steps have to be done to run a KVM libvirt/qemu virtual machine with preinstalled CORVUS®/NOCTUA®

If your machine is setup correct, only you have to do is to start the virtual machine and have fun with monitoring.

In the menu area you'll find submenus, buttons, date, time and user section.

CORVUS® offers some additional menus:

Figure 3.2.  Menu buttons

Buttons and submenus for NOCTUA®

Figure 3.3.  Menus

Menus for NOCTUA®

In the tree area you can find your device group tree and associated devices. Located on top, there is a searchfield and 2 buttons.

Figure 3.4.  Devicetree

NOCTUA®

Figure 3.5.  Devicetree

NOCTUA®

Figure 3.6.  Selection buttons

Buttons to select, deselect or toggle selection

All the configurations and input takes place in the main area. According to the selected or preselected devices and settings, corresponding page appears.

Figure 3.7.  Possible main area

One possible view of main area after select some devices in "device network"

The cluster server information button shows two overview tabs, one tab with information about definied cluster roles and one with information about server

One Server checked

Inside this tab, there is a table showing following information:

Server information

Instance

Name of service

Type

Type of service {node, server,system}

Check

Kind of Check

Installstatus

Status if service is installed or not

Version number

Versionnumber of installed service

Processnumber

Number of processes started

Memory usage

Displays memory usage as number and as statusbar

Action Buttons

Button to apply action to the services

Figure 3.8.  Cluster server information

Backgroundinformation about running or stopped services

The permission system is divided into several parts which covers certain functions. Some permissions depend on other permissions, or in other words, chainpermissions. The more permissions user get the more powerfull they can act. The user "admin" or "superuser" is the most powerfull user. Admin have all possible rights and permissions.

Below is a list with permissions and what their functions are.

The permission level defines what can be done by users. In combination with the permission itself, administrators are more flexible in assigning rights and permissions to user or to groups.

Below are 4 main permission levels which can be assigned.

So far, your server is ready for package installation. Also the clients/nodes have to be prepared for package installation.

The main configuration file for package-server is /etc/sysconfig/package-server. It content should be self-explanatory and looks like this:

The main configuration file for package-client is /etc/sysconfig/package-client. It content should be self-explanatory and looks like this:

If your system do not provide some packages you really want to install, there is an other way to go. In this special case you can either download fitting binary packages from external sources and place it in the right directory or you can compile and build your own package from sourcecode.

Environment variables for setting up RMS can be found under /etc/

To get RMS working it is not enough only to install the package, you must also edit some config files and build the SGE part manually. Below step by step how to install RMS will help you installing RMS and run the required services.

Even it should be obvious, before you are able to install RMS make sure you already installed noctua and its dependencies.

COMMING SOON ...$$

RMS overview provides 4 tabs. Not only for displaying information but also to control jobs. There are a couple of green buttons on the bottom of overview page to hide or unhide columns.

Running jobs

The first tab of RMS overview displays current running jobs in the grid engine. You can get some background informations like jobids, owner, runtime or nodelist of each job. On the right side there is an action button to delete or force delete running jobs.

Figure 6.1.  RMS running jobs

Current running jobs with disabled nodelist column

Waiting jobs

The second tab of RMS overview displays the current waiting jobs. This are jobs waiting in the SGE queue for execution. Among other infos, it shows the "WaitTime", "Depends" and the "LeftTime".

Figure 6.2.  RMS waiting jobs

Current waiting jobs

Done jobs

The third tab of RMS overview displays done jobs and specific columns like "ExitStatus", "Failed" or "RunTime".

Figure 6.3.  RMS done jobs

Done jobs

Nodes

The fourth tab of RMS overview displays the nodes itself. You can enable or disable queues or, if it exists, display graphs of choosen nodes.

Figure 6.4.  RMS nodes

Node overwiew

Commands the SGE provides are:

Common way to submit jobs to the cluster is to use grid engines "q" commands. Assumed that your cluster configuration is correct, running jobs on cluster is as easy as running jobs on local machines.

Following steps have to be done to transfer jobs to queue:

To activate SNMP discovery for one device, simply select the checkbox Enable perfdata, check IPMI and SNMP. To get this checkbox, either select your device and left click the home icon on top, or double click the device.

To reach SNMP scan, go to Base Device network.

There are no SNMP schemes yet in the settings window. Now perform a SNMP scan with left click on the orange update network button.

Figure 8.2.  Auto discover SNMP

Button to auto discover network

It appears a SNMP setting window, where you are able to adjust some basic settings.

Figure 8.3.  SNMP scan settings

SNMP scan settings

Depending on your network size and structure, it takes some time to get complete SNMP data tree, apply filter and algorithms to it and write the extracted data into the database.

After performing SNMP scan, you will get some new network confic entries for the scanned device.

Figure 8.4.  Before SNMP scan

Network config for one device before scan

Figure 8.5.  After SNMP scan

Device network config after successful SNMP scan

That way you automatically get a couple of netdevices with according names, values, MAC addresses, MTU values, speed, etc... without to invest much time or manpower. A very handy and timesaving tool for administrators.

If we go back and choose the Google maps tab, we notice a red Flag onto the google map and also two new buttons, an icon and category name appeared beside the map.

The blue locate button zooms the map in. With the green add location gfx button you are able to upload user image maps in two steps:

Once you named your new location graphics, a new modify button appears. Use the button to upload user images.

Figure 9.3.  Advanced location setting

Advanced location settings

Of course you can add even more than just one user image, so you can create a stepwise zooming from google map to detailed server room photographs.

Figure 9.4.  Three user images added to location

Zoom levels with according user image maps

Figure 9.5.  Concepts of zoom levels with multiple image maps

Zoom levels with according user image maps

CORVUS® allows you also to edit uploaded images with the preview and enhance button.

Following self-explanatory buttons are accessible if you want to edit your uploaded image for quality reasons.

Figure 9.6.  Accessible buttons to modify user images.

Zoom levels with according user image maps

Following editing buttons are integrated:

Once you have created new location categories and added some photos or images, you can easily add device livestatus to it.

Select all devices you wish to add and click either the home button or use selection button.

Navigate to the Location tab select the checkbox and left click on the location category. It appears a show location map button on the right side with some informations about the image and a small preview of it. Push the button to show the image map.

Now you can place your livestatus burst on the right place at the image by clicking on the set button.

Figure 9.7.  Adding livestatus to image maps

After placing livestatus burst on the right place left click on the lock button to prevent the livestatus burst from moving.

Use the remove button to remove livestatus burst from image.

Select your desired device and choose the livestatus view to display livestatus burst on imagemap. If there are more than one assigned location map, there will be tabs for each image map.

Figure 9.8.  Adding livestatus to image maps

Checks will be defined individual with fixed options and bound on specific devices. These checks are always specific, that means to change one option of the check is the same as to change the whole check.

Checks are defined globally as Parameterized check and bound on devices. These checks are not specific, that means to change one option of the check it is enough to change the parameter of it.

There are 10 devices, 7 of them should be checked on port 80 and 3 of them on port 8080:

Solution fixed method:

You have to set up two different checks, one check with option set to port 80 (-p 80) and one check with option set to port 8080 (-p 8080).

Solution parameterized method:

You have to set up only one check with parameterized options (-p $PORT_NUMBER). Now you are able to modify the port option parameter to every desired value without changing the check itself.

For some reason we will create checks with 5 different warning values.

Solution fixed method:

You have to set up five different checks with five different warning option values. If there are even 10 different values you have a lot to do because you need to create 10 different checks.

Solution parameterized method:

You have to set up only one check with parameterized warning option value and change the parameter for each of the five different warning values. If there are also 10 different warning option values you only have to change the warning option parameter for each device instead of renew the check.

RRD stands for Round Robin Database and is a special designed database structure to collect data circular. That means that the database must be setup for the right amount of data which should be collect.

For that reason there are following advantages and disadvantages:

But the monitoring software takes care for this details so you have not to agonize about it.

To collect data and draw graphs in CORVUS® and NOCTUA® there are more services appropriate for. Lower figure illustrates how they work together and how the dataflow between each other is.

Dotted parts are still in progress but will be very soon implemented into CORVUS® or NOCTUA® because of better data flow distribution, less read/write access and therefor less load on the server.

Datatransfer should only takes place if rrd-graphs will be requested.

Figure 12.2.  RRD graph cycle

Cycle of rrd graphic dataflow

To makes rrd-graphing work, the rrd-grapher service and collectd-init service must already run.

Select one ore more devices you want RRD graphs for and click either on the "house" button in top menu or on the green "use selection" button below the top menu.

If you only need RRD graphs for one device, just click on the device name in the device tree view.

In both cases there will be some new tabs displayed, one of them named Graphs

The rrd tree is, similar to the device tree, a overview. The difference is that a parent object can consists of one or more child objects. For instance the parent object mem contains 4 child objects, avail, free, icsw and used.

Figure 12.9.  Tree parents and childs

Showing tree parents und children

Figure 12.10.  Aggregation of 3 devices

Data aggregation of 3 devices

For parent groups it makes sense to summarize some graphs. This kind of summarization is called aggregation in CORVUS® and NOCTUA®. Best way to get group information or to get overview of a cluster is to use aggregation. Aggregation is more complex than simple addition of data. It manages interferences and calculates values the most effective way to display sums as realistic as possible.

Above graph shows the 15 minutes single load value for 3 devices (pink, purpur and green) and a combined sum graph (brown) of all device graphs.

Figure 12.11.  Compound memory graph

Compound memory graph with stacked graphs.

Compound view can be found on top of the monitoring data tree. It combines several data (for example load, cpu, processes, memory and io) on one multigraph, no need to select n graphs.

An other advantage of compund graphs is stacking. Some graphs are more significant if values are displayed stacked.

Above figure explains stacking in context of memory graphing.

In this example you can see straightaway the parts of memory usage in relation to available memory.

Critical errorlogs will also be delivered by mail. So you do not have to check your logs permanent, you will be notified by mail if there are critical errors.

Setting for recipient of errorlog mails is stored in /etc/sysconfig/logging-server.

Another configuration file for mail notification is /etc/sysconfig/meta-server.

Replace the given mailaddress in the line containing TO_ADDR= with your desired mail address.

# from name and addr
FROM_NAME=pythonerror
#FROM_ADDR=localhost.localdomain
# to addr
TO_ADDR=mymail@gmail.com
# mailserver
MAILSERVER=localhost
            

After editing the logging-server configuration file, the logging-server daemon must be restarted:

rclogging-server restart

The new configuration take effect after restart logging-server daemon.

Figure 14.1.  Bad looking fonts

ugly looking fonts due to wrong font setup

If you get something like in the picture above, you have to install fetchmsttfonts (OpenSUSE) and ... (debian) package.

This is a server internal error, likely the server can't find some files. Take a look into /var/log/nginx/error.log for detailed error message.

For some reason the webserver nginx doesn't run. Start it manually, for example with "service nginx start"

Please wait a moment till database connection is active and reload the page. If you still get this message after waiting a time you have to start uwsgi-init, for example with "service uwsgi-init start"

For some changes in your configuration, especially network configuration, you have to rebuild config (cached, RC) first. If your config is stored in cache, you have even to rebuild config (refresh)

Most likely the discovery-server is not installed or discovery-server service is not running. Make sure the discovery-server is installed and running. Run the check_cluster.sh command and look for "discovery-server". If it is not running, start it either by commandline rcdiscovery-server start or via the webfrontend under cluster server information

Figure 14.2.  discovery-server not defined in routing

Errormessage

An other possible reason for that malfunction could be disabled discovery server config for your monitoring server. To enable it select your monitoring server device, navigate to the config tab and select the discovery server settings.

Figure 14.3.  Discovery-server settings

Enabled discovery server config

Sometimes complex network topology slows down display output in firefox. This issue affects firefox up to version 31.0. Reason is likely bad javascript interpretation on firefox side. If you get bad graphic display performance, try to use another browser e.g. chromium or google chrome™.

Sometimes it could be useful to reset a user password. For example if someone forget the password or something else goes wrong.

A short guide how to reset a login password by direct access to the database via clustershell follows:

From now you are able to login with your new password.

If you must wait long time while pending upload and the infolabel "Please wait..." is shown after upload image with add location gfx button, reload the page to resolv this issue.

Figure 14.4.  Please wait ...

"Please wait..." message after image upload

Some vnc-server tends to break correct mouse pointer handling in virtual desktop. To get back correct mouse pointer, log out of your session and back in again.