Sunday, August 28, 2011

The Network Administrator - monitoring your network from the cloud

After three months of coding the Network Aministrator is ready to go. In this post I would like to describe some of its features. Since the project is now open to the whole community, I will also write about a major tasks for the next few months, where your help would be very appreciated.

The idea



As some of you may know, the name for this GSoC project was "Network Administrator to the Cloud Land". The title outlines the basic concept of the NA, which is creating monitoring tool that will work in a computing cloud as a web application. This idea seems to be very simple, however a product like that hasn't yet been created (correct me if I'm wrong). But why should we care about developing such an application? Well, computing clouds give us cutting edge scalability and reliability--the features that we are looking for while creating a monitoring tool, right? Moreover, web interfaces provide us with high availability and usability at the same time. In Network Administrator we are using these qualities and we take monitoring software to the next level.

The basic concepts



Now I would like to present you some of the basic terms and concepts we're using in NA.

Event

An Event is a single information about what happened on a server. Description of event contains a message (and its shorter version), type of event (e.g. "WARNING"), time stamp, name of network protocol, source host and optionally some additional data serialized as a JSON [1]. Based on these data we should know everything about an event: when, where and what happened.

Host and Network

Every event has its source--a host, which is represented by a name, IP address (both v4 and v6) and optionally a description. Administering a large number of hosts may be difficult, but you can manage them easier by aggregating them into a network with unique name. [2]

Report

Reports are a basic tool for regular monitoring of a host or network. A report lets us to see events in a specific period of time, and export them into user-friendly format (e.g. PDF). [3]

Web API

One of the greatest features of the Network Administrator is its RESTful API, which allows external applications to report events and to read selected data. Authorization is provided by the xAuth mechanism, which is a slightly modified OAuth technology. For encryption we are using SSL, which is a standard for this kind of services. Data format chosen for this API is JSON. [4]

Plugins

Monitoring a network is a very ambiguous term, therefore monitoring tool should be a highly extensible software, to let user easily define what data he wants to see and how. With the basic knowledge of Python and Django template language you can create a plugin that will show additional data about events or networks. Or it can show the same data but using different representation, like a chart. There are lots of possibilities now, and there will be much more of them in the future.

Implementation details



Django is a powerful and mature web framework, so I had no doubts to use it in the Network Administrator. However, this project is different than any other application I've done before--it is entirely based on NoSQL backend. Using non-relational database we can create system that is quicker and lighter. Besides, it's much easier to change data model during development process on database that is horizontally scalable. Unfortunately Django doesn't support NoSQL backends by default. But there is a cool project called Django-nonrel [5], created to meet this requirement. Its authors claim that the latest release is stable. Indeed it works well and I think we can use it in production environment.

As I wrote above the web API is a crucial part of the NA. It was implemented based on Piston framework [6]. Using this simple Django application we can easily create RESTful services by defining handlers for GET, PUT, POST and DELETE methods. It also provides nice authentication backends, e.g. OAuth.

For generating reports we are using Geraldo Reports--another Django application [7] with lots of nice features.

Since the Network Administrator was designed to work in a computing cloud, I had to test it in one of those. The Google AppEngine seemed to be the perfect choice. Why? It is completely free to deploy there startup application like this, it has great documentation and it provides services like e-mail, cron, database viewer etc. Thanks to Djangoappengine [8] app I could easily set up NA on Google's servers.

Charms of using NoSQL



One of the most important lesson I learnt during development of the NA was to use NoSQL databases. At one moment I had to forget about JOINs, many-to-many relations support, full-text search etc. Now I have to admit it was quite amazing experience, that learnt me to focus on every single query I write.

Review of the most important features



Now I would like to show you some of the most important features of the Network Administrator.

Dashboard--your work starts here


Dashboard--your work starts here


Browsing and searching events




Managing a host




Setting up alerts




Managing a network




Report exported to a PDF





TODO



Hoping that some of you may be interested in contributing to the project, I wrote a short TODO for the next few months. It's not a complete list of tasks but it shows major priorities for the Network Administrator.

Name

Does anybody here believe that "the Network Administrator" is a final name for this project? No way! It should be short and catchy if we want to efficiently promote this idea.

Testing and refactoring

We should write much more tests for all of project's applications. I'm also aware of need to spend more time on code refactoring.

Graphic design

The goal is to create a very good-looking interface with a cutting-edge usability. It's the easiest way to show people that our project pretends to be the high end web tool. To reach this goal we have to find (hire?) a professional graphic designer.

Usability and new features

Regardless of the plan to create a new layout, we should think about new features that would make this tool more usable and user-friendly. It's easy to extend the NA with plugins so we just have to think: how to make NA a monitoring tool that we would like to use?

How to start with the Network Administrator



To start just register a new accout at http://ns-dev.appspot.com/user/register/. After pressing "Register account" button you should receive a mail with an activation link. Click it and that's it--you can log in! Then set up your servers to report events to your account [9]. Now you can monitor your network the way you never did!


[1] http://dev.umitproject.org/projects/na/repository/revisions/master/entry/netadmin/events/models.py
[2] http://dev.umitproject.org/projects/na/repository/revisions/master/entry/netadmin/networks/models.py
[3] http://dev.umitproject.org/projects/na/repository/revisions/master/entry/netadmin/reportmeta/models.py
[4] http://dev.umitproject.org/projects/na/repository/revisions/master/entry/netadmin/webapi/handlers.py
[5] http://www.allbuttonspressed.com/projects/django-nonrel
[6] https://bitbucket.org/jespern/django-piston/wiki/Home
[7] http://www.geraldoreports.org/
[8] Djangoappengine authors are the same guys who are responsile for Django-nonrel project: http://www.allbuttonspressed.com/projects/djangoappengine
[9] In the next post I'll show you step by step, how to set up NA with the latest Network Inventory

No comments:

Post a Comment