Friday, August 31, 2012

Inter-peer P2P routing in OpenMonitor - Overview

   Hi all. This is Narendran Thangarajan from India.
   In GSoC 2012, OpenMonitor system has been revamped and empowered in various ways. I am happy to have made a contribution through the inter-peer routing implementation. Let me give an introduction into the technology and then explain briefly how it fits into the OpenMonitor system.

Why did we need P2P routing?

   Before GSoC 2012, OpenMonitor was more of a centralized system. All the peers and super peers were connected to the central aggregator and they relied heavily on this central aggregator for authentication, receiving tests, sending reports etc. But what if the central aggregator itself was blocked due to censorship? Or if the aggregator is facing a minor outage? Thats when the need to modularise the OpenMonitor system arises. If the responsibilities of the aggregator could be delegated to super peers seamlessly, then the tight dependency of the aggregator's availability could be eliminated. 
   Thus we looked into different alternatives on how to make our OpenMonitor system more scalable and available. Listed below are the most adopted DHT based P2P routing algorithms currently.
  1. Content Addressable Network
  2. Chord
  3. Pastry
  4. Tapestry
  5. Kademlia
   We decided that Kademlia Routing algorithm is the best suited P2P logic for our scenario. Lets make a quick visit to Kademlia algorithm now.

Kademlia :


   To understand Kademlia, first we must get to know about Distributed Hash Tables. DHT is the technique powering BitTorrent tracker concept. As it sounds, a DHT network consists of a hash table, distributed across multiple nodes in such a manner that the impact of a node suddenly leaving the network is minimal. 
Distributed Hash Tables
   When you search for any content (or 'value') on this hash table, a hash function maps it to a unique key, which is mapped to a node ID. Kademlia takes utmost O(log(n)) searches, in a network containing n nodes, until it finds the node where the required data is present.
   Each node in Kademlia is assigned a Node ID which is unique in a P2P network at any instant of time. Besides being unique, the node ID serves another purpose of routing. Two nodes are considered to be nearby if the XOR value between two node IDs is small. Difference between two node ID values is considered to be the distance between those two nodes. Each node has a 128 bit routing table which can store a list of node information. Each bit stores information in the form of <IP Address, Port, GUID number> using which node lookup becomes fast and easy.
   Kademlia or 'Kad' is a concept. There are various implementations of the Kad concept in various languages. To list a few
  • TomP2P
  • Libcage
  • Maidsafe
  • Entangled
   We first started off with Maidsafe-DHT. I was able to cross compile the library to accommodate with Python based desktop agent and the Android (Java) based mobile agent. But one of my cool fellow Umit developers Dai found that Maidsafe was not completely compatible with iOS. So we had to move to an alternative. During the initial stages, we also found that Maidsafe was huge. Too huge to be running as a binary on a thin client like the mobile phone. So we chose "libcage" implementation created by Yuuki Takano. Its a C++ based P2P routing framework with NAT traversal. Libcage was originally done as an implementation of DTUN concept proposed by Takano. You can find the source code of this project here. Considering Maidsafe, libcage was much simpler and lightweight, enabling it to be run even under stern resource constraints.

Cross Compiling Libcage for Python :

As I said, libcage is written in C++. In order for a program written in C++ to be used by a python program, the C++ library should be wrapped to run smoothly on the destination platform. Wrapping C++ modules to work with python could be enabled in the following ways - SWIG (Simplified Wrapper and Interface Generator) and Boost Python. Finally we chose Boost Python to enable cross compiling libcage library to be used by C++ programs. Here are the steps :
  1. Clone libcage from https://github.com/ytakano/libcage.git
  2. Create your own mylogic.cpp file using the libcage libraries. Use the examples present in /examples folder.
  3. Add the following module to your mylogic.cpp file to make the methods visible to Boost Python.
BOOST_PYTHON_MODULE(libcagepeers){
def("python_method1",actual_method1);
def("python_method2",actual_method2);
}
  1. Next, create a shared library using "g++ -shared -Wl,-soname,libcagepeers.so.1 -o libcagepeers.so getpeers.cpp" command. Make sure you link the necessary libraries, say boost, libcage and crypto. Executing this command creates the necessary libcagepeers.so shared library, which will act as a python module as well. 
Thus, we can just insert a "import libcagepeers" line into our python code and start using the shared library.

Cross Compiling Libcage for Android :

Android NDK is little complex when compared to Android SDK. But learning it is proportionately more rewarding. Here are the steps to make libcage usable with your Android app. Be warned, the official version of NDK doesn't support certain features like wchar, wstring etc. which are often used in the libraries involved. So I made use of Crystax NDK.
  1. Create a "jni" folder under the root directory of your project.
  2. Cross compile all the required libraries like boost, event and crypto using android toolchain. 
    1. Create an independent toolchain using make-standalone-toolchain.sh command.
    2. Set the environment variables 
export SYSROOT=/home/sunshadow/Documents/android-ndk-r7-crystax-5.beta2/platforms/android-4/arch-arm
export CC=arm-linux-androideabi-gccexport CXX=arm-linux-androideabi-g++
// For instance to cross compile libevent library. 
sudo ./configure --build=arm --target=arm-linux-androideabi --enable-shared
sudo make

sudo make install
Now you must find the required static libraries in the .libs folder. Move the libevent*.a and libevent*.la libraries to our jni folder in android project's root directory. Repeat the same steps for boost, crypto and libcage. After this step, its exactly what we need to do for Android NDK projects ie. Create an Android.mk file that specifies the C and C++ flags, include folders, libraries to be linked and the source files to be compiled. When implementing our mylogic.cpp file, we must wrap the function to be exposed to Android using JNI_EXPORT <return_type> JNICALL <method_name>.

With respect to OpenMonitor, we also made a few changes in the libcage library, to make certain underlying data structures like the routing table and the listening port visible to the python and android layer. This is the overview on the background processes involved in making a P2P routing library named libcage to work with the desktop agent and the mobile agent of OpenMonitor.

Cheers :)


Sunday, August 26, 2012

Open Monitor Desktop Agent 2012



Hi everybody,
      I am Tianwei, from China. I am a student of Google Summer of Code 2012.The GSoC 2012 has just reached the end! I have a wonderful summer with UMIT. Let’s have a look what we have done this summer.
1.    Login Form
When the users open desktop agent, a login form will pop up for the uses to input username and password.

About the settings button, the user can set the url of the aggregator cloud.


When the user input wrong username/password, the alter window will pop up.

2.    Preference Window
a)     Tests Page
In the Test Page, the user can select and delete the Test Sets, but the website test is necessary, when the user try to remove all tests, the website is still there.



At the same time, the update setting and HTTP Throttled Test Switch is here.
b)     Advanced Page
Task assign, task scheduler, report uploader and test fetcher can be configured. The current desktop agent support English and Chinese, later more language will be added.

c)      Peer Info Page
Some information about peer id, test set version, cloud URL are in this page. The users can add super peer information(ip and port) by manual.



d)     General Page
In this page, the users can configure the startup, auto login, desktop notification. Version details can show us.

e)      Update Page
Automatically update or manually update can be selected. The desktop will check software update information in startup stage.

f)       Feedback Page
Service suggestion or website suggestion can be sent to aggregator.

3.    Logs Window

4.    Event Window

5.    Dashboard Window
a)     Capacity Item
This page contains communication grade, service tests result and other peer information.

b)     Sent Details

c)      Task Details

d)     Connection Items

e)      Reports

6.    About Window

7.    Bug Report Window
When the desktop agent meet a crash, it will produce a crash report window which contains bug information and it will be sent to our website.


8.    Software update Window



In the startup process, when the desktop agent check new version , it will show the icon in right-top position.
9.    Super Peer Behavior  
a)     Without GUI

b)     How to set super peer in desktop agent

Of course, the aggregator side should mark this peer is super peer.
c)      When the super peer starts up, it will connect the peers in his peer database.

d)     How the desktop agent add super peer by manual
In Preference Page -> Add Super Peer -> restart the desktop agent. The desktop agent will try to connect super peer and get necessary token information without aggregator.



e)      Send report to super peer

f)       Get task from super peer

g)     End connection

10. Desktop Agent ICON
In Ubuntu, when you input icm-agent, it will show our Open Monitor Desktop Agent!

Monday, August 20, 2012

How To Test Open Monitor iOS Agent



1. Login Form

If you are the first time to use this app, or are not logged in, a login form will pop up for you to input username and password:

  
If you don't have an account, you can tap on the "Register" button to get one:

 

2. Website Test List


If you are logged in, you should see a website list like this:
   
These are the websites tests this agent will perform every 10 min by default.
You can tap on the "Refresh" button in the up-right corner to perform the tests manually.
After a while, you will see the list becomes like this:
Those green dot on the left side of each website means this website is accessible, otherwise it's red. 
Tap on the website, you shall see the detailed status of the website, including URL, status code, date performed, etc.

3. Service Test List

The service test tab is similar to website test tab:
  

4. Map Tab:

This tab show the global evens:


Tap on the "Refresh" button, the app will fetch the events from the aggregator, and show it on the map.
Tap on the pin on the map, the detailed information will show.


5. Settings

In this tab, you can logout, suggest website, suggest service, and check new tests manually.


Website suggestion:
Feedback of website suggestion:
Service suggestion:
Feedback of service suggestion:

6. Preferences:

You can set some preferences in the system settings app, including auto login, host and update interval:




You can set the aggregator host manually:
There are many update interval choices you can choose from:
  
"10 min" means the app will automatically perform connectivity tests every 10 minutes.