Umit project

Hi folks. During this Google Summer of Code 2011 I have been developing OpenMonitor Cloud Aggregator. It was a good experience, as I learned to work with new tools, like GoogleAppEngine, Protobuf, Django, and the work was developed in a team.

was over on August 22nd, but not for us here at Umit Project. Though we've send the code written up to the firm pencils down date, we kept working and working on our projects and some other organizational tasks that a program like GSoC requires from us. And we will still be cranking throughout the months to come, as we've got a lot to do with all these awesome projects that were developed during this summer.

Now, I'm thrilled to announce you the successful projects! Please, join us on congratulating these students and spreading the word about their projects!

• Alan (Zhongjie Wang) developed the OpenMonitor desktop agent. It is a neat piece of software that will scan the internet from where user is connected, and provide our Cloud Aggregator with real time data about censorship, throttling and network availability in general. Useful for us to map in realtime what governments or ISPs are blocking around the globe as well as figure possible major internet outages caused by accidents or catastrophes.
• Angad Singh had a challenging undertaking this Summer, while trying to get traceroute to work on Android. The result now features the Mobile Umit Network Scanner that will soon be released for FREE at the Android Market.
• Diogo Pinheiro managed to create the OpenMonitor Cloud Aggregator, which is the piece that puts together all the real time data we get from our OpenMonitor desktop and mobile agents, and shows it on a very nice map with detailed description on the incident.
• Dragos Dena developed the Umit Network Inventory Next Generation, inspired by the awesome software of the same name developed by Guilherme Polo and that we currently distribute along with Umit Network Scanner. This new breed isn't really meant to replace the original project, but to allow for users to bring network monitoring and inventory to a whole new level allowing for users to create alarms, customize them, extend monitoring agents and best of all: integrate with our cloud based Network Administrator, that I'll describe below.
• Gaurav Ranjan managed to touch and improve four different and very complex projects we have here at Umit Project: Umit Network Scanner, Zion, RadialNet and UMPA. His contributions improved these projects will a bunch of new features and have now made all of them capable of handling IPv6 requests. We'll soon be releasing a new version of Umit Network Scanner featuring all these improvements.
• Piotrek Wasilewski developed the Network Administrator, which is a cloud based system that allows for easily monitoring hosts status remotely. It is a very cool piece of software and is very extensible, easy to use and easy to deploy. Your days of trying for weeks to install that huge and complex monitoring system are over.
• Zubair Nabi is the wizard that poped out of his hat the OpenMonitor mobile agent for Android. He managed to design an impressive interface and a very high quality software for helping out the world to monitor internet availability. If you have an Android, stay tunned: this will soon feature the Android Market for FREE.

After three months of coding the Network Aministrator is ready to go. In this post I would like to describe some of its features. Since the project is now open to the whole community, I will also write about a major tasks for the next few months, where your help would be very appreciated.

The idea

As some of you may know, the name for this GSoC project was "Network Administrator to the Cloud Land". The title outlines the basic concept of the NA, which is creating monitoring tool that will work in a computing cloud as a web application. This idea seems to be very simple, however a product like that hasn't yet been created (correct me if I'm wrong). But why should we care about developing such an application? Well, computing clouds give us cutting edge scalability and reliability--the features that we are looking for while creating a monitoring tool, right? Moreover, web interfaces provide us with high availability and usability at the same time. In Network Administrator we are using these qualities and we take monitoring software to the next level.

The basic concepts

Now I would like to present you some of the basic terms and concepts we're using in NA.

Event

An Event is a single information about what happened on a server. Description of event contains a message (and its shorter version), type of event (e.g. "WARNING"), time stamp, name of network protocol, source host and optionally some additional data serialized as a JSON [1]. Based on these data we should know everything about an event: when, where and what happened.

Host and Network

Every event has its source--a host, which is represented by a name, IP address (both v4 and v6) and optionally a description. Administering a large number of hosts may be difficult, but you can manage them easier by aggregating them into a network with unique name. [2]

Report

Reports are a basic tool for regular monitoring of a host or network. A report lets us to see events in a specific period of time, and export them into user-friendly format (e.g. PDF). [3]

Web API

One of the greatest features of the Network Administrator is its RESTful API, which allows external applications to report events and to read selected data. Authorization is provided by the xAuth mechanism, which is a slightly modified OAuth technology. For encryption we are using SSL, which is a standard for this kind of services. Data format chosen for this API is JSON. [4]

Plugins

Monitoring a network is a very ambiguous term, therefore monitoring tool should be a highly extensible software, to let user easily define what data he wants to see and how. With the basic knowledge of Python and Django template language you can create a plugin that will show additional data about events or networks. Or it can show the same data but using different representation, like a chart. There are lots of possibilities now, and there will be much more of them in the future.

Implementation details

Django is a powerful and mature web framework, so I had no doubts to use it in the Network Administrator. However, this project is different than any other application I've done before--it is entirely based on NoSQL backend. Using non-relational database we can create system that is quicker and lighter. Besides, it's much easier to change data model during development process on database that is horizontally scalable. Unfortunately Django doesn't support NoSQL backends by default. But there is a cool project called Django-nonrel [5], created to meet this requirement. Its authors claim that the latest release is stable. Indeed it works well and I think we can use it in production environment.

As I wrote above the web API is a crucial part of the NA. It was implemented based on Piston framework [6]. Using this simple Django application we can easily create RESTful services by defining handlers for GET, PUT, POST and DELETE methods. It also provides nice authentication backends, e.g. OAuth.

For generating reports we are using Geraldo Reports--another Django application [7] with lots of nice features.

Since the Network Administrator was designed to work in a computing cloud, I had to test it in one of those. The Google AppEngine seemed to be the perfect choice. Why? It is completely free to deploy there startup application like this, it has great documentation and it provides services like e-mail, cron, database viewer etc. Thanks to Djangoappengine [8] app I could easily set up NA on Google's servers.

Charms of using NoSQL

One of the most important lesson I learnt during development of the NA was to use NoSQL databases. At one moment I had to forget about JOINs, many-to-many relations support, full-text search etc. Now I have to admit it was quite amazing experience, that learnt me to focus on every single query I write.

Review of the most important features

Now I would like to show you some of the most important features of the Network Administrator.

Dashboard--your work starts here




Browsing and searching events




Managing a host




Setting up alerts




Managing a network




Report exported to a PDF

TODO

Hoping that some of you may be interested in contributing to the project, I wrote a short TODO for the next few months. It's not a complete list of tasks but it shows major priorities for the Network Administrator.

Name

Does anybody here believe that "the Network Administrator" is a final name for this project? No way! It should be short and catchy if we want to efficiently promote this idea.

Testing and refactoring

We should write much more tests for all of project's applications. I'm also aware of need to spend more time on code refactoring.

Graphic design

The goal is to create a very good-looking interface with a cutting-edge usability. It's the easiest way to show people that our project pretends to be the high end web tool. To reach this goal we have to find (hire?) a professional graphic designer.

Usability and new features

Regardless of the plan to create a new layout, we should think about new features that would make this tool more usable and user-friendly. It's easy to extend the NA with plugins so we just have to think: how to make NA a monitoring tool that we would like to use?

How to start with the Network Administrator

To start just register a new accout at . After pressing "Register account" button you should receive a mail with an activation link. Click it and that's it--you can log in! Then set up your servers to report events to your account [9]. Now you can monitor your network the way you never did!


[1] http://dev.umitproject.org/projects/na/repository/revisions/master/entry/netadmin/events/models.py
[2] http://dev.umitproject.org/projects/na/repository/revisions/master/entry/netadmin/networks/models.py
[3] http://dev.umitproject.org/projects/na/repository/revisions/master/entry/netadmin/reportmeta/models.py
[4] http://dev.umitproject.org/projects/na/repository/revisions/master/entry/netadmin/webapi/handlers.py
[5] http://www.allbuttonspressed.com/projects/django-nonrel
[6]
[7]
[8] Djangoappengine authors are the same guys who are responsile for Django-nonrel project: http://www.allbuttonspressed.com/projects/djangoappengine
[9] In the next post I'll show you step by step, how to set up NA with the latest Network Inventory

Like the websites list, a service can be selected to view scan details.

Engineering the ICM Mobile Agent this summer, I encountered a number of challenges which were both exciting and nerve-wrecking at the same time. Working on a mobile platform and application places one in a unique situation because one does not have access to unlimited resources and power, and various code libraries. As a result, one needs to be mindful of various run-time quirks that can significantly reduce the efficiency and usability of the app. In this post I am going to retrace my path through the course of the summer and highlight three significant hiccups that I faced.

So, Google Summer of Code 2011 came to an end a few days ago. I hope it was a very fun summer for all GSoC students out there. For me, at least, it sure was. I got to work at a project that caught my attention in the pre-application period as having the potential to be a very educative experience for me. And it was, not only because of the project itself, but because of the great community here at the Umit Project. I would like to thank Adriano, Luis and of course my mentor Rodolfo for being helpful and supportive during this summer.

I have learnt a lot of useful stuff: network programming (and also Twisted Matrix with it), a better understanding of Python programming, packaging and installation procedures and of course, some improved debugging skills with all of it:-). But most important for me, being a very disorganized person, I learnt to organize my time much better working at such a project.

As you may have noticed from the title, I worked at Network Inventory: New Generation, a platform to monitor hosts in your network trough asynchronous events sent by those hosts. The NI:NG is composed of 2 main components:

• Network Inventory Agent. A daemon/service that is installed on the host that should be monitored and sends out notifications when something happens. It offers an modular design so the users can add new monitoring functionality based on their needs (or better said, on what is needed to be monitored on that host).
• Network Inventory Server. Also a daemon/service that receives notifications from the agents installed on the monitored hosts. It stores those notifications and provides an interface so other applications can view them. Besides receiving notifications from agents, it also provides support for SNMP Traps and it can even be extended to support new monitoring protocols.

Hi, everyone. The Google Summer of Code 2011 has just reached the end. We've had a quite exciting summer this year because of the project we are incubating , the Internet Connectivity Monitor. Along with the worldwide voices against Internet censorship, ICM will be the sentry in the front line to discover any kinds of Internet connectivity problems.
ICM Desktop Agent is the client side software running on user's computer. It supports Windows, Linux, and Mac OS platform. We develop this software using Python, Twisted and PyGTK. The functionality of the desktop agent is running in background and doing connectivity testing against some websites and services. The connectivity testing tasks are assigned by the aggregator. It also allows user to manually control the behavior of the agent. Users will have an overall view of the connectivity status of current region and the whole internet though the connectivity map, which is based on Google Map and OpenStreetMap. And also, they can see the testing results and the connectivity statistic data in localhost.
We are making ICM as a global monitoring network, so the agents have the capacity of connecting to each other and sharing their knowledge, mainly are reports. The desktop agents forms a two-tier network. Super desktop agents are the first tier, and normal desktop agents are the second tier. The super agents are the ones more powerful and stable, which could be a bridge between the aggregator and the normal agents when the normal agents couldn't reach the aggregator. And also they can provide some balancing services to the normal agents. We are trying to build this network robust, and eliminate the island phenomenon among peers.
Now the ICM Desktop Agent is near it release date. We are looking forward to its roll-out.
Through this summer's coding work, I've learnt a lot of Python programming skills, and have gotten familiar with the python modules and Twisted framework. I also learnt how to write asynchronous networking programs with Twisted. That's a bit challenging, and it's quite different from the programming patterns I've used before.
Some screenshots:

JUnit test suites in Android constitute a separate Android project that is deployed alongside the application under testing [1]. Users often make the mistake of adding external libraries as dependencies to both projects which leads to java.lang.VerifyError failure caused by the conflicting external library. In case of Eclipse, the standard solution is to add the external library as dependency to only the main project and then export it to the test project [2]. But unfortunately, this solution is not applicable when Maven is used to run the test suite [3].

In this post, we are going to build upon my previous post on using Maven as a build manager for Android projects.