Like every other software, NESTOR® has also certain system requirements. Because NESTOR® is open source Software, everybody who has enough programming knowledge could be able to port it to other open source software systems.

The good news: You don't have to port anything if you already use one of the following LINUX distributions:

For exact versions please take a look into Installation Chapter.

Monitoring configurations are stored in databases for faster access and therefore faster reaction time and further more flexible administration of data.

NESTOR® is using Django as its database interface so every database which is compatible with Django can be used. The recommended Database is PostgreSQL (mostly due to license issues)

MySQL database is not supported any more.

There is no public access to our repository directories, therefore first, you have to contact us to get a valid LOGINNAME and PASSWORD.

After receiving you access data you are able to use below mentioned repositories.

To install the software on your operating system there are two ways to go.

You can either automatically install the software by downloading and running our install_icsw.py install script or manually by adding the below listed repositories with your individual access data and use your package manager as usual.

The automatically installation via script is well recommended because it is very comfortable and handles most of installation scenarios.

In order to use the install script, first you have to contact us to get your individual access data.

The script does following:

After receiving your individual access data do following steps to install the software manually:

There are two main repositories you have to deal with to install the software.

Repositories are available for stable (2.5) and for master (devel) versions of above mentioned operating systems. The operating system running on your hardware and the version of the software you want, determine which repository configuration you must use for your package manager.

Based on the above mentioned operating system, repository and desired software version, resulting repositories can be added.

There are two different ways to add new repositories for monitoring software by init.at ltd. to the operating system. It can be added all at once in one central file or in a repository directory. For each operating system there are special repository directories.

For ubuntu 12.04 you can also add repositories either in one file, into /etc/apt/sources.list or in repository directory /etc/apt/sources.list.d

Debian and ubuntu use an other package manager than CentOS, OpenSUSE or SLES. For that reason, on rpm based operating systems sources.list does not exist. Rather there are a few files for repository management not only one. All relevant repository files stays in the directory /etc/zypp/repos.d/ .

[cluster_devel_remote]
name=cluster_devel_remote
enabled=1
autorefresh=0
baseurl=http://LOGINNAME:PASSWORD@www.initat.org/cluster/RPMs/suse_13.1/icsw-devel
type=rpm-md
                

[cluster_devel_remote]
name=cluster_devel_remote
enabled=1
autorefresh=0
baseurl=http://LOGINNAME:PASSWORD@www.initat.org/cluster/RPMs/suse_13.1/icsw-2.5
type=rpm-md
                

Repository directory is /etc/yum.repos.d/ . Place your desired *.repo files inside this directory, do a yum check-update and you are ready to install the software.

[initat_cluster]
autorefresh=1
enabled=1
type=rpm-md
name=initat_cluster
baseurl=http://LOGINNAME:PASSWORD@www.initat.org/cluster/RPMs/rhel_6.2/icsw-devel
                

[initat_cluster]
autorefresh=1
enabled=1
type=rpm-md
name=initat_cluster
baseurl=http://LOGINNAME:PASSWORD@www.initat.org/cluster/RPMs/rhel_6.2/icsw-2.5
                

Before continuing the server installation it is worth to say something about the database because it is one of most important parts of the software. It contains all settings, configurations, users and much more in one single database. This is the reason why it is easy to migrate or backup data and this is why it lowers the monitoring effort in comparison to a simple icinga installation.

possible choices for DB engine: psql, sqlite
DB engine (psql)               : 
DB host (localhost)            : 
DB user (cdbuser)              : 
DB name (cdbase)               : 
DB passwd (bTMZPUYMiR)         : 
DB port (5432)                 :
                

CREATE USER cdbuser LOGIN NOCREATEDB UNENCRYPTED PASSWORD 'my_password_123';
CREATE DATABASE cdbase OWNER cdbuser;
                    

local   cdbase          cdbuser                         md5
host    cdbase          cdbuser         127.0.0.1/32    md5
host    cdbase          cdbuser         ::1/128         md5
                

dsn is 'dbname=cdbase user=cdbuser host=localhost password=bTMZPUYMiR 
port=5432'
connection successful
                

-rw-r----- 1 wwwrun idg 156 May 7 2013 /etc/sysconfig/cluster/db.cf
                    

The Webfrontend for your server can be accessed via

http://SERVERNAME/cluster or by http://IP_ADDRESS:80/cluster/

The Webfrontend for your server can be accessed via

http://SERVERNAME/cluster or by http://IP_ADDRESS:80/cluster/

In case you run setup_noctua.sh script manually, the portnumber will be rewritten to 18080. So you can access the web front-end by this url:

http://SERVERNAME:18080/cluster or http://IP_ADDRESS:18080/cluster

Also you can use your server localhost alias for accessing the front-end:

http://localhost:18080/cluster or http://localhost:18080/cluster

After added desired repositories for your operating system it is time to install the software packages itself and configure it. There are three main packages you have to install to get a basic server running:

These packages contains all necessary services, binaries, libraries and dependencies for a clean and proper Installation.

If you also want to get access to the server by web GUI, you additionally need to install the nginx-init package and run the nginx-init http server.

For SUSE operating systems a installation command should look like following one:

zypper ref; zypper install icsw-server icsw-client icsw-dependencies

While accessing the repositories you will be prompted for valid username and password. Type in your received access data on the terminal and continue installation.

icsw setup command
                

Following steps have to be done to run a KVM libvirt/qemu virtual machine with preinstalled NESTOR®;

Finally if your machine is setup correct, only you have to do is to start the virtual machine and have fun with monitoring.

You have to convert our virtual machine image for kvm/qemu into an suitable format for Vmware.

icsw service status

Get more information about possible flags with: icsw --help

One of the most common flag is -v. This shows additionally the version number of each package like shown below.

Figure 3.1.  icsw status command

An other common flag is -a. With this flag, the script shows additional information:

Take a look into our command reference to learn more about icsw command.

The main command, which can be used to manage the cluster components, is the icsw command. The following table present the frequently used examples of the icsw tool:

If you never have applied licenses to your server then you are running an unlicensed software version. To remind you of this fact you will see notification messages on some pages, for example on the login page and also on the dashboard after succesfull login.

Figure 3.2.  Unlicensed notification

Also you can check your license state by navigating to SessionLicense.

There you will see three different drop down windows:

Figure 3.3.  License overview

Shipped Licenses are keyfiles containing information about licensed features, license period and license parameter. Each license is associated to one specific Cluster ID. A keyfile can contain one or more license packages assigned with one or more cluster IDs.

Figure 3.4.  License package

The license package dropdown window shows you an content overview of uploaded keyfiles.

Keyfiles containing more than one single license package will be displayed in separate tabs inside of the License package dropdown window.

You can't expand any dropdown menu by left clicking on the arrow beside the dropdown except the Upload license file one. Use the Choose File button to select your valid license keyfile. After selecting your keyfile its name will be displayed on the right side of the button:

Figure 3.5.  License overview

Push the upload button to integrate your valid license keyfile into the server and activate aquired enterprise features.

After uploading your valid license keyfile to the server, your license overview imediately will be updated and shows your purchased licenses. Generally, if you want to see your license overview navigate to SessionLicense to display the license status.

Figure 3.6.  icsw status command

There are four different periods or states of licenses. Dependent of the period or the state a licensed feature is working or not.

Figure 3.8.  License states

The second factor which decides if a license is valid or not is the parameter limitation. Dependent of purchased parameter amount and of used parameter the license can be valid, in grace time or expired.

Figure 3.9.  License states

Licenses can be violated by exceeding the license time period or by exceeding one of the license parameters limitation. In case of exceeding the license parameter limitation also a grace time period starts which is totaly independent of the license time period grace time.

Figure 3.10.  License states

As relative complex software NESTOR® use some well known frameworks and technology we absolutely have to mention.

Parts which both software packages needs to works properly.

Monitoring in context of hard- and software will be practiced to get information about specified systems.

NESTOR® consist of many different parts and services. Each of these services perform a specific set of tasks in the cluster. Most of these services are network enabled and listen on a specific port for commands. The following list tries to give an overview about the most important parts.

The general idea of NESTOR® is simple. Create an image, a kernel and a set of configuration files to be used by your nodes and distribute them to the nodes. The distribution is done via PXE. NESTOR® enables you to describe the node specific configurations in Python.

The database where all the configuration data of NESTOR® installation is stored is generally referred as the "clusterdatabase".

Throughout this document we might refer to:

Server side scripts .  Are generally services or scripts that are running on the clusterserver. Most of them need database connectivity to function properly.

Node side scripts .  On the other hand, node side scripts are daemons or scripts that are generally run on a node of the cluster. Node side scripts don't require access to the cluster Database.

The graphic shows just the part between database and webfrontend, only a small but essential part of monitoring.

Figure 4.1.  Webfrontend and database dataflow

Dataflow with help of django and django REST framework.

In the menu area you'll find submenus, buttons, date, time and user section.

NESTOR® offers some additional menus:

Figure 5.2.  Menu buttons

Buttons and submenus for NESTOR®

Figure 5.3.  Menus

Menus for NESTOR®

In the tree area you can find your device group tree and associated devices. Located on top, there is a searchfield and 2 buttons.

Figure 5.4.  Devicetree

NESTOR®

Figure 5.5.  Devicetree

NESTOR®

Figure 5.6.  STC Selection buttons

Buttons to select, deselect or toggle selection

Alternativ there is an one button selection method.

Figure 5.7.  S/C Selection button

One button selection, either select all or deselect all

All the configurations and input takes place in the main area. According to the selected or preselected devices and settings, corresponding page appears.

Figure 5.8.  Possible main area

One possible view of main area after select some devices in "device network"

The cluster server information button shows three overview tabs, one tab with information about definied cluster roles, one tab with information about server results and one with information about the server itself.

One Server checked

This is the only one tab inside of server information, which allows you to control something. You are able to control services as you do by command line.

Following information will be displayed:

Server information

Instance

Name of service

Type

Type of service {node, server,system}

Check

Kind of Check

Installstatus

Status if service is installed or not

Version number

Versionnumber of installed service

Processnumber

Number of processes started

Memory usage

Displays memory usage as number and as statusbar

Action Buttons

Button to apply action to the services

Figure 5.9.  Cluster server information

Backgroundinformation about running or stopped services

All preselections are done in the sidebar device tree Section 5.2.2, “ Sidebar device tree (2) ”. Select groups or devices and click in top menu on desired submenu to access .

Figure 6.1.  Submenu method

Device network for preselected devices as result

Submenu method is recommended for working with multiple devices.

Following areas can be accessed by the submenu method:

The method by home button is more general. Again preselection of devices or groups takes place in sidebar device tree but afterwards instead of choosing a submenu, this time we click on the home button.

As result we get some useful tabs for each device.

Figure 6.2.  Home button method

Device configurations for preselected devices as result

Home button method is recommended for working with multiple devices.

Following areas can be accessed by the home button method:

By direct click on the device name in sidebar device tree we get similar overview tabs like by home button method but only for one single device.

Direct method is recommended for working with single devices.

For graphing there is the requirement first to select devices, then select the wanted graphs and finally draw this graphs for selected devices. The graph preselection will remain unaffected if you change the device preselection. After changing device preselection you must push the Apply button. Same is true for drawing graphs.

Figure 6.3.  Two-stage selection

Preselected devices and preselected graphs results in graphs

Most common auxiliary function is the filter input field on top of device trees or configuration tables. Most simple usage of this filter field is to insert some text string or number to filter for. If you do so for example in sidebar device tree then only matching devices will be selected.

^node

node[0-5]

[0-9]$

\d$

Further information on regular expressions filter can be found in the world wide web by looking for javascript regex.

Figure 6.4.  Input field filter for device configurations

Displays all device configuration entries beginning with base, even they are not selected.

An other auxiliary function in handling with tables are the show and hide buttons on top of tables. With this buttons you can easily show or hide specific table columns.

Figure 6.5.  Show and hide buttons for table columns

Device tree with hidden Password column.

To ease displaying of longer lists and to avoid to much page scrolling there is also a simple pagination function built into the software. With pagination we are able to limit entry output on pages to a specific number. Only the choosen numer of etries will be displayed, the other entries, if there are some, will be divided on separate pages which can be accessed via the page button.

Figure 6.6.  Pages and entries per page

Shows 20 entries per page, divided into three pages

Last but not least we'd like to mention the column sort function. It also could be very useful to display only desired data.

Not all columns provide this sort function, but most of them do. The function will be toggled by clicking onto the column name. If the function is activated there is a small triangle left beside the column name pointing with its tip either in top direction for ascending or in bottom direction for descending sorting. If no triangle is visible sorting function is deactivated.

Sorting method is:

Figure 6.7.  Sort column

Activated ascending sorting marked with a small black triangle pointing in top direction.

Our developer created a reversion function not only to display what changes were done but also to go back in change history to desired state and drop changes which were done afterwards. History reversion can be found in top menu at ClusterHistory.

The reversion function is based upon the central database (default is PostgreSQL) so in principle every change written into the database can be reversed. Normally there are lot of different data stored in the database to ensure every component works fine but it makes no sense to provide the reversion feature to all of this collected data. For normal user and administrators it is completely enough to revert changes which were done via the web front-end.

For example three new user were added in User Management. Like shown below, system history lists all relevant database entries for each of them.

Figure 6.9.  System history

Database excerpt for User data

Now lets suppose there was a typo in one of these names e.g Lucy was changed to Luci. If we take a look into the system history under User we get exactly this change displayed in an diff-like style.

Figure 6.10.  System history

Displayed diff in system history after changed user name

If you are not only satisfied to display changes but also really want to go back to an earlier version there is the revert to this version button.

For example, someone changed directory paths in an script located at BaseConfigurations and you would like to display that changes, simply navigate to the History tab of Modification window to get a list containing all applied changes up to now.

Next step to do is to mark your desired version out of the list. Now, you can either apply reversion by clicking on the Modify button or just switch to the script editor to check how the script looks like after reverting.

Figure 6.11.  Reversion overview

Typical colored diff output

We can see all changes at a glance in above shown figure.

The permission system is divided into several parts which covers certain functions. Some permissions depend on other permissions, or in other words, chainpermissions. The more permissions user get the more powerfull they can act. The user "admin" or "superuser" is the most powerfull user. Admin have all possible rights and permissions.

Below is a list with permissions and what their functions are.

The permission level defines what can be done by users. In combination with the permission itself, administrators are more flexible in assigning rights and permissions to user or to groups.

Below are 4 main permission levels which can be assigned.

So far, your server is ready for package installation. Also the clients/nodes have to be prepared for package installation.

The main configuration file for package-server is /etc/sysconfig/package-server. The content should be self-explanatory and looks like this:

The main configuration file for package-client is /etc/sysconfig/package-client. It content should be self-explanatory and looks like this:

If your system do not provide some packages you really want to install, there is an other way to go. In this special case you can either download fitting binary packages from external sources and place it in the right directory or you can compile and build your own package from sourcecode.

Environment variables for setting up RMS can be found under /etc/

To get RMS working it is not enough only to install the package, you must also edit some config files and build the SGE part manually. Below step by step how to install RMS will help you installing RMS and run the required services.

Even it should be obvious, before you are able to install RMS make sure you already installed NESTOR® and its dependencies.

RMS overview provides 4 tabs. Not only for displaying information but also to control jobs. There are a couple of green buttons on the bottom of overview page to hide or unhide columns.

Running jobs

The first tab of RMS overview displays current running jobs in the grid engine. You can get some background information like jobids, owner, runtime or nodelist of each job. On the right side there is an action button to delete or force delete running jobs.

Figure 9.1.  RMS running jobs

Current running jobs with disabled nodelist column

Waiting jobs

The second tab of RMS overview displays the current waiting jobs. This are jobs waiting in the SGE queue for execution. Among other infos, it shows the "WaitTime", "Depends" and the "LeftTime".

Figure 9.2.  RMS waiting jobs

Current waiting jobs

Done jobs

The third tab of RMS overview displays done jobs and specific columns like "ExitStatus", "Failed" or "RunTime".

Figure 9.3.  RMS done jobs

Done jobs

Nodes

The fourth tab of RMS overview displays the nodes itself. You can enable or disable queues or, if it exists, display graphs of choosen nodes.

Figure 9.4.  RMS nodes

Node overwiew

Commands the SGE provides are:

Common way to submit jobs to the cluster is to use grid engines "q" commands. Assumed that your cluster configuration is correct, running jobs on cluster is as easy as running jobs on local machines.

Following steps have to be done to transfer jobs to queue: COMMING SOON

To install all above mentioned and required services on SUSE systems, use the following command line: zypper ref; zypper in dhcp-server tftp nfs-kernel-server xinetd

Use the common operating system package manager for other distributions.

It is not enough only to install the needed services, they also have to be configured properly to make a NESTOR® cluster installation possible.

DHCPD_INTERFACE="eth1"
                

cluster-server.py  -c write_dhcpd_config  -D authoritative:True

Also the following necessary NESTOR® packages have to be installed first in order to get a running cluster installation setup.

Install all needed packages with an one line command like done for system packages. To do it on SUSE systems, use the following command line:

zypper ref; zypper in icsw-server

Create a new partition table under ClusterPartition overview. Important aspect here is the Nodeboot check box explained further down.

Figure 11.1.  Partition overview table

Figure 11.2.  Create partition

Insert a name for the partition, and, if you want, insert also a partition description. Two problems will be immediately displayed on the web front-end. In fact no defined discs and no present mountpoint. To solve this problems do following:

Please pay attention to the naming of discs which orientates on UNIX conventions and therefore used the common devicename standard like: /dev/sd

The first part /dev/ represents the device tree. The second part sd represents a mass storage device like HDD od SSD. To create a disk simply insert a letter between a (first registered device) and z (last registered device).

Do not insert any partition number to the device name. This will be done in the create partition dialogue.

After creating a disk, create partition button will be visible.

Make sure to set the nodeboot flag of the partition.

Next, we need a LINUX™ kernel. It will be handled like a template kernel for the nodes. There are two common ways to create a kernel. You can either compile a completely new kernel or use an existing one.

system target directory /opt/cluster/system/tftpboot/kernels does not exist
                    

The third component you absolutely need to run a cluster and install nodes over network is an image. It contains all needed software packages for your node installation. We need a basic image which will be served to the local nodes and represents a pattern for all node installations. Also here we have to choose if we want to install all image components manually or if we want to be assisted by a script named make_image.py.

In order to booting nodes over PXE we have to define two separate networks. The first network is intended for the boot procedure of the node. The second network will be used for the node installation over NFS and later in productive state.

We need this two different networks both as setting in NESTOR® and as real hardware network settings.

Below figure shows you a typically NESTOR® network setting for one single cluster server and two different networks. Of course, there are three networks if we also count the local area network but for our purpose we concentrate only on the class B networks beginning with 172.

Figure 11.3.  Two different networks

One boot and one prod(uction) network defined

Also important is, that your cluster server really has the same hardware network settings. In our example, we use two network devices(eth1 and eth1:prod) providing our two different networks for booting and productive and additional the local area network (eth0). So we get following ifconfig -a output:

eth0      Link encap:Ethernet  HWaddr 32:34:03:32:32:DD  
          inet addr:192.168.1.239  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::3034:3ff:fe32:32dd/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:71510 errors:0 dropped:10 overruns:0 frame:0
          TX packets:10156 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:7598935 (7.2 Mb)  TX bytes:5595522 (5.3 Mb)

eth1      Link encap:Ethernet  HWaddr 52:54:00:DE:96:95  
          inet addr:172.17.1.1 Bcast:172.17.255.255  Mask:255.255.0.0
          inet6 addr: fe80::5054:ff:fede:9695/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:179865 errors:0 dropped:0 overruns:0 frame:0
          TX packets:193583 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:17279193 (16.4 Mb)  TX bytes:306664808 (292.4 Mb)

eth1:prod Link encap:Ethernet  HWaddr 52:54:00:DE:96:95  
          inet addr:172.16.1.1  Bcast:172.16.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
            

Adding nodes

The device network settings are also very important to boot nodes over network. That are the settings which must be done over web front end. Each node needs following device network settings:

• One network device

• Two IP addresses, each for boot and prod

• Peering connection to the cluster server

For a better understanding there are already two nodes (node, node2) with a complete correct setup. Now we have to setup node3 the same way.

Lets start with adding a new node. In web front end, navigate to BaseDevice tree. Create a new device, create optional a device group before, and name it. Essential settings here are:

1. Bootserver

   This should be the machine our nodes boots from

2. Root passwd

   This will be your node root password, if leaved empty the default password will be init4u

   Note

   Root passwd can be set not only for first installation of nodes but also for every boot procedure.

After created a new node device, navigate to BaseDevice network to create a new netdevice, assign two IP addresses to it and set the peering.

Figure 11.4.  Comparison of complete (node, node2) and incomplete (node3) settings

Figure shows node3 without any netdevice, IP address and peer

1. Click on the Create new button and create a new netdevice.

   In hardware tab type in the MAC address of your device. This is the only way your node can be identified by your cluster server because it has no IP at booting time.

   Also type in your network driver, for example virtio_net if you want to boot a virtual machine with this driver.

   Mark the inter device routing checkbox in the basic settings tab. After creation of your new netdevice

   Figure 11.5.  New created netdevice for node3

   Node3 still without assigned IP address and peer

   
   

2. Create two IP adress for your device, each for boot network and for prod network by click on the create new button beside of your network device and choose IP from the drop down menu.

   Type in your desired IP adress and choose the right network.

3. Last but not least you have to create a peer or network topology connection to make sure the node is connected with the cluster server.

   Like above, click the Create new button, but this time choose network topology connection to make a link between node3 and cluster_server.

An other important point in context of cluster setup is the right device configuration. Here, we enable the NESTOR® cluster software, especially the cluster server device, to act as an cluster server. Analogue is true for the nodes.

You have to enable at least following device configurations for the server:

Also you have to enable node configuration for each node

Figure 11.6.  Table mode

Configuration for cluster server and nodes

Figure 11.7.  List mode

Configuration for cluster server in List mode view

After all packages are installed, configured and are running, it is time for the very funny part of NESTOR®. To control nodes before and after installation there is a tool called Clusternodeboot. Nodeboot enables you to control your nodes in every way. You can also display log lines for each node or with the macbootlog button for all nodes. It is possible to control one single node or to control all nodes at once. Nodeboot allows you to pick one of the following:

Nodeboot provides in addition to above choices a soft control button, an hard control button, an action button and a log button.

Figure 11.8.  Overview of all existing nodes

Nodeboot with two selected devices, ready for global actions

Use the "STC" global action buttons or the "sel" buttons to select nodes for modifications or for actions.

Possible soft controls for nodes are:

Possible hard controls for nodes are:

Content of the modify button depends on weather a header button is selected or not. For all selected header buttons the modify popup window looks like this:

Figure 11.9.  Nodeboot modify popup window

Logging is also possible with nodeboot. To display your nodes logging messages simply click on the show button on the right side.

Figure 11.10.  Opened logging table

An other useful logging view is macbootlog.

Figure 11.11.  Macbootlog

In case of something goes wrong the logging-server writes its logs under /var/log/cluster/logging-server/[HOSTNAME]/.

Denotation of log files and subdirectories is related to the service which writes the log. For example if the meta-server can not start some service it will write its log into the directory

If you want to see background information for package-installation on some nodes the file you have to check is package-client. Analogue this is true for server side, this time the filename is package-server

Files called *.bz2 are compressed logging backup files.

Critical errorlogs will also be delivered by mail. So you do not have to check your logs permanent, you will be notified by mail about critical errors.

Setting for recipient of errorlog mails is stored in /etc/sysconfig/logging-server.

Another configuration file for mail notification is /etc/sysconfig/meta-server.

Replace the given mailaddress in the line containing TO_ADDR= with your desired mail address.

By uncommenting and editing the line beginning with #FROM_ADDR= you are able to set the sender "From" name of recived emails.

# from name and addr
FROM_NAME=pythonerror
#FROM_ADDR=localhost.localdomain
# to addr
TO_ADDR=mymail@gmail.com
# mailserver
MAILSERVER=localhost
            

After editing the logging-server configuration file, the logging-server daemon must be restarted:

icsw service restart logging-server

The new configuration take effect after restart logging-server daemon.

A very handy command to read out logfiles is icsw logwatch. Logwatch makes it possible to display logs for different services and daemons at once. Even if you don't know in which file the logs are written to you are able to watch it. Thats the reason why logwatch is an allround tool for logging.

As usual for log files they have a typical output format.

Without any parameter logwatch.py displays the last 400 lines of logging messages for all services writing logfiles. With icsw logwatch -n 20 you can limit output lines to the last 20.

A very useful parameter for icsw logwatch is --system-filter. This flag restricts the output to one single daemon (service) e.g

displays only log messages related to an rrd (daemon) service.

With the -f flag it is possible to view logs in realtime. Use

to output appended data as the file grows. Try icsw logwatch --help to list all possible options.

In case of malfunction it is very likely that the portnumber will be written into a logfile or apears in the web front-end. To find out which service or process causes the error we have to know which service communicates on which port. Following table shows you a little summary of common services and their communicating ports.