Monday, July 30, 2007

Network Inventory updates

I've been working on UMIT Network Inventory mainly and since July 16th many things have been done to make it better ;)

Like I said before, Timeline was being integrated with NI and now it is ready to use.

What things have changed since two weeks ago:
  • Select between "Yearly View", "Monthly View", "Daily View" and "Hourly View";
  • Filter settings are kept, before, changing graph mode or something else would reset it;
  • TLBarDisplay (an "extra" widget for Timeline) now has a nice color transition effect;
  • Changes listing have been reworked, so it was possible to use it inside Timeline;
  • More ways for grabbing data;
  • A lot of several minor (and some not so minor) things have been done, to make it possible to Timeline work with all new changes.
What you cannot do on current Network Inventory yet:
  • Configure something to remove data older than N days;
  • A more flexible data range setting;
  • Search for changes over time*;
* You cannot search for changes, but you can search for several other things that will return results that includes changes over time.

There are probably many other things, I just pointed something that I considered most needed.

You can view Latest UMIT Network Inventory screenshot

Tuesday, July 17, 2007

SUDO, Pardon!

First, this will be short. Previous post was long enough to make you bored probably :)

Now, to the point. On my last post I said it was all sudo's fault for not creating Scheduler control file correctly, and I was wrong.

I wasn't even looking at that problem now, I was continuing with Timeline integration with rest of umitInventory and umitDB but for some unknown reason I just wanted to fix that strange bug.

From the previous post: "Running with sudo at console causes the Scheduler to not create a control file at user home dir, but it runs fine and everything that it needs to do is done, except that it can't be stopped by the controller since it didn't create the control file."

But what was happening was: I've done a function that checks if a process is running at all, on UNIX, it uses kill for that. So, it was working like this: send kill to the pid with signal 0, if it throws an OSError exception the process is not running... WRONG!

Throwing an OSError doesn't mean the process is not running, I needed to check errno also. An OSError with errno equals 3 ('ESRCH') means the process isn't running (errno 3 is 'No such process'), but since I ran the process with sudo it was returning errno equals 1 that means 'Operation not permitted' and then it was returning False and the control file would get deleted and I would think that the control file wasn't created at all.

Now this bug is fixed at least =)

And again, forgive me SUDO.

Monday, July 16, 2007

Network Inventory and Co., The Journey


I've been developing my project since accepted students were announced and many things have been created, changed, recreated, fixed, re-fixed, tested, retested, and still being tested of course =)
So, I will be writing about what is already done and some other things, so when I read this I will know what I need to do :)

Scan Scheduler

Scan Scheduler has been developed to schedule scans (but it may schedule anything).
  • Key features:
    • Multi-platform. Actually, this is the main reason to do this Scheduler;
    • Uses cron format for defining scheduling time[1];
    • PyGTK GUI;
    • Save scans output to a directory;
    • Send scans output through email;
    • Add scans to the Network Inventory. This is another main reason for developing this Scheduler;
    • Based on scheduler profiles[2].

[1] Almost the same cron format is being used (';' instead of ','), but this will be changed.
[2] Each "Scheduled Scan" has an unique profile name that identifies it, and one or more "Scheduled Scan" may use a "Scheduling Profile" (this defines when scans should run) and also has an unique name that identifies it.

  • Some "extra" info and development time of Scan Scheduler:
    • Cron parser was done already, Adriano did it.
    • Scheduler Profiles was Adriano's idea and I though and still think it is good enough to keep it;
    • Took around 1 week (a bit less) to develop Scheduler core and Scheduler GUI;
    • Created a "SMTP Account Editor" GUI for creating and managing SMTP schemas to be used inside "Scan Scheduler Editor" for sending emails.
    • After this one week, I've improved the Scheduler controller several times but still has one main "bug" left [3] and:
    • This is almost ready for use except for:
      • [3] As you may know, some nmap scan options requires root to execute. So, I'm still looking for a way on how to start it as root inside UMIT. I've tried using gksu, but then Scheduler will use config files inside root home. Running with sudo at console causes the Scheduler to not create a control file at user home dir, but it runs fine and everything that it needs to do is done, except that it can't be stopped by the controller since it didn't create the control file.

        Strangely enough this isn't an issue for win32 :p

Network Inventory
Why write yet another Network Inventory ?

Before the official start (28th May), I've been thinking about how to make a nice Network Inventory. I did some research and for my surprise, most of "Network Inventories" were more like a Software and Hardware Inventory that gathered information from local network using some win32 protocol that I forgot the name now (sorry).

After some search I found PBNJ, nice! At least someone thinks the same way I do about what a Network Inventory should be. But (maybe I could be wrong and I'm sorry for that) it is just impossible to visualize and organize a historic using PBNJ alone, since it doesn't offer a GUI or anything else. Also, I think it would be way too hard to handle many changes in a small to medium network with it. PBNJ author seems to be a good person (I just talked a bit with him on IRC asking for permission to talk about PBNJ for a lecture that I talked a bit about my project, and he was very nice), so I hope he doesn't get mad if he reads this.

Other project that I know that does a lot more than I pretend to do in this summer (winter for me) is Splunk, it handles everything you throw at it. Their team have very nice people, talked with them for the same reason mentioned before.

So, UMIT Network Inventory comes to help you understand your network(s) situation, with a nice GUI that includes unlimited Historic per device, Search, Scheduler and an eye-candy and useful Timeline, and is Open Source of course.

What was done to make Network Inventory exist ?

Before the official start I saw that current UMIT database schema would be no help for doing an Inventory. So a new schema needed to be done, flexible enough to handle every piece of nmap xml output [4] and some other things related to the Inventory.

And there was a new schema! Nice! It has evolved and it is on a very good stage now. So, I created a python package for handling this database. This database consists of 30+ tables, and a lot of triggers for forcing foreign key integrity (since sqlite doesn't enforce it). So, what this packages does up to the moment:
  • Inserts XML nmap output into this database;
  • Store/Retrieve any piece of data;
  • Store/Retrieve things related to Inventories;
  • Retrieve combined data;
  • Performs search;
  • Store/Retrieve Inventory Changes;
  • Update Inventory Changes;
  • Grabs Inventory Changes by timerange (just started but works already).
With this new database, it was possible create and maintain several Inventories.

[4] It is still missing some features from latest nmap versions, but this is easily extendable.

The never-ending journey: The Network Inventory GUI

Man.. let me tell you a thing, since I was announced as an accepted student, I've written around 35.000+ lines of code, and "throw away" more than half (right now on my branch there is 15k+ of my code).

The Timeline widget have been written and rewritten several times, always getting better (at least I think) till actual stage. All previous versions never worked with real data, this is, they were never really integrated within Network Inventory. Only this latest version is finally working with real data, and finally it seems I found a good Timeline solution, that accept data in a very flexible format and it is very easy to add new graph "types" to it (right now it supports line graph and area graph).

Other key-piece of Network Inventory GUI was the Changes List and Changes Diff (Changes Diff is an improved version of current DiffCompare found in UMIT). Before current Interface, I've tried several things to create a nice historic visualization (I ended up deprecating some of them even before making them to the repository).

After all, I think it was really important to try several things to see what goes better. Also, I can't count this as a time waste since it helped me a lot to improve the interface, to improve my thinking on how to handle Inventory changes and some other things.

It has been a really good journey, and is still being. More features to come, especially for Timeline, very soon.

Ok, so what Network Inventory does right now ?

This text has gotten long enough, so, here comes the features list:
  • Inventory creation/editing;
  • Network and Host Discovery (using nmap);
  • Timeline for navigating through changes in a better way (right now it supports just yearly view);
  • Archives any xml nmap output (if you know some that breaks it, send me a copy or at least report the error please);
  • Handles an unlimited historic for each device found in scans;
  • Perform "scan diff" for ports, extraports, fingerprint, osclasses, os match;
  • Displays a list with short changes description per device;
  • Shows a more complete set of changes when something is selected at "list with short changes description";
  • Searches for ip, hostname, ports, services, MAC, fingerprint, os match and os classes data;
Expect a much better integration with Timeline and the rest of Network Inventory for the following day/week(s).

If someone wants to follow current development, you will need to checkout my repository at sourceforge using svn.

That was it, thanks for reading ;)

Monday, July 09, 2007

Nmap 4.22SOC1 with Umit Released!

A brand new alpha version of Nmap with some work done by Google Summer of Code students has been released[1], and Umit is been send along with Nmap and other students improvements. We're hopefull to have more people trying Umit, reporting more bugs[2] and giving us more feedback!

If you're curious to know what Umit's students are working on, join us on trying our software and checkout our repository. The student's works are separated by diretories inside the branch dir.
  • bass_boy (Rodolfo) - The UmitWeb! Run and point your browser to localhost:8059. Login: user1, Password: 123
  • ggpolo (Guilherme) - Network Inventory and Scan Scheduler. Execute python umit -i
  • hildon (Adriano) - Porting of Umit interface to Maemo
  • indep_features (Adriano and Frederico) - Independent features and bug hunting.
  • joao (João) - UmitMapper. Run a scan, and take a look on the shinny new tab that shows João's radial map of your network.
  • k0p (Luís Bastião) - Umit Interface Editor. A better way to customize your Umit's profile and wizard interfaces. Run umitInterfaceEditor/
  • max (Max) - NSE Facilitator. An easier way to play and create your NSE Scripts.
  • pavel (Pavel) - Nmap Python Wrapper. A module intended to let you do things like: import nmap; nmap.run_scan("localhost")
  • umitdb (Guilherme and João) - A new Database schema and module for Umit. Better arrangement, and other stuffs improved.

The only thing we're missing is you trying it out, and having fun with us!


[1] - Nmap 4.22SOC1 Release announcement -
[2] - Umit's Bug Tracker -