Friday, May 25, 2012

Cross-compilation of a static Nmap with OpenSSL for Android

Intro

I am Gorjan Petrovski and this summer I’ll be your Android Network Scanner developer as part of the Google Summer of Code 2012 program :) . In short, my main goal is to build a good frontend for an advanced network scanner for Android which will use Nmap as a backend, but its architecture will be designed so that we can use other scanners too. My proposal describes the goals of my project in more detail. 

Before I head on to code an Android frontend for Nmap, first I needed to cross-compile Nmap for Android, and this blog post documents my experiences. I decided to cross-compile Nmap as a static executable as the most reliable option. Another purpose of this text is to help people which want to build a static Nmap executable with OpenSSL for themselves; If you just need the binaries, please head on to the excellent manual which this post is based on where you can download them:

So, if there’s already a cross-compiled version of Nmap and a manual, why this blog post, you might ask. The simple reason are some problems I encountered as a beginner with Android NDK while trying to cross-compile OpenSSL libs as static and link it to Nmap. Namely, I found some OpenSSL repos on github which compiled the libs as shared, modified them, but afterwards I could not link them to Nmap because of missing static dependencies (libz and libdl). This is why I decided on building OpenSSL and the other libs from the Android source code. 

Building the static libs (libdl, libcrypto, libssl) for Android

It might seem that building the source code of Android for getting these libs is an unnecessary, long and arduous process, but I would highly recommend it to any beginner working with Android NDK, as it will clear up quite a lot of things. 

I don’t plan to guide you through the Android build process because it is well documented and there are people who will help you who are far more experienced than me. I’m only going to give you a few tips which I found helpful:

I highly recommend that you build the whole Android source code to make sure everything is in order with your build environment. After you set up your build environment you can continue with building the modules:

I'm going to suppose that the directory where you've downloaded the Android source code is called ANDROID.

The modules by default are configured to be built as shared libraries, so what I've done is change the Android.mk in their adequate directories so they'll be build as static libraries. Below are the links for the diffs of the libraries:


After you've patched the respectable libraries/directories with the diffs, you can build the modules by cd'ing to ANDROID and running:

$ make clean; make libdl libssl libcrypto

Tip: Don't forget to run $ . build/envsetup.sh to set the environment variables you're gonna need.

If you followed the instructions carefully the modules should be built. You can find the libraries in ANDROID/out/target/product/generic/obj/STATIC_LIBRARIES/ in their respectable libXXX_intermediate directories (ex. for libcrypto the library is libcrypto_intermediates/libcrypto.a). 

Building Nmap

If you compiled the libraries you are now ready to build Nmap. For this you'll need to download and install the Android NDK and Android SDK. I've found that it's most helpful if you expose the directories of the binaries to the PATH environment variable. For example, if you have Android NDK version r8 and it's extracted to your home dir, add this line to your .bashrc script:
export PATH=$PATH:~/android-ndk-r8:~/android-sdks/tools:~/android-sdks/platform-tools

Get Nmap from my github repo (it is version 5.61TEST4 with an additional directory - android):
$ git clone https://github.com/mogi57/nmap-5.61TEST4-android nmap
Copy the ANDROID/external/openssl/include dir to the nmap/android/openssl dir:
$ mkdir -p nmap/android/openssl
$ cp -a ANDROID/external/openssl/include nmap/android/openssl/

and copy the compiled static libraries to the nmap/android dir:
$ cp ANDROID/out/target/product/generic/obj/STATIC_LIBRARIES/lib*_intermediates/lib*.a nmap/android

Edit the NDK variable in the nmap/android/Makefile script to point to the location of your Android NDK and make sure that /tmp is writable for your user because the standalone toolchain will be copied there. After that you're ready to compile Nmap:
$ cd nmap/android; make

...and you should have your nmap static binary ready and compiled in your nmap folder.

Preparing and deploying Nmap to your device

You should still be in your nmap/android dir, and run:
$ make pack

After that plug your Android device in and run:
$ adb shell mkdir /sdcard/opt
$ adb push nmap /sdcard/opt/nmap
$ adb mkdir /data/data/umit.nmap
$ adb mkdir /data/data/umit.nmap/nmap

But you can't run it from /data so you need to:
$ adb cat /sdcard/opt/nmap/bin/nmap > /data/data/umit.nmap/nmap/nmap
$ adb chmod a+x /data/data/umit.nmap/nmap/nmap

... and you can run nmap from adb or a terminal emulator on your phone by running:
# /data/data/umit.nmap/nmap/nmap 127.0.0.1

Happy Hacking!

If you're interested in Nmap for Android keep yourself updated with this blog, as I’ll publish builds of new versions and I'll probably automate this process even more as I continue to build newer versions of Nmap. It's going to be a fun summer.

Monday, April 30, 2012

Build and Run MaidSafe-DHT on Mac OS X

1. Install command line tools for Xcode
Xcode -> Preferences -> Downloads -> Components, Click “Install” behind “Command Line Tools”.

2. Install MacPorts
Download dmg from http://www.macports.org/install.php, and install.

3. Install Boost with MacPorts
sudo port install boost

4. Install gcc-4.6 with MacPorts
sudo port install gcc46

5. Making and Installing MaidSafe-Common
See: https://github.com/maidsafe/MaidSafe-Common/wiki/Developer-Build-Instructions
VERY important: to make cmake to use gcc 4.6:
$ export CC=/opt/local/bin/gcc-mp-4.6
$ export CXX=/opt/local/bin/g++-mp-4.6
Then:
$ cd MaidSafe-Common/maidsafe_common_lib/build/OSX/Release
$ cmake ../../..
$ cmake --build . --target install

6. Making and Installing MaidSafe-Transport
See: https://github.com/maidsafe/MaidSafe-Transport/wiki/Developer-Build-Instructions
$ cd MaidSafe-Transport/build/OSX/Release
$ cmake ../../..
$ cmake --build . --target install

7. Making and Installing MaidSafe-DHT
See: https://github.com/maidsafe/MaidSafe-DHT/wiki/Developer-Build-Instructions
$ cd MaidSafe-DHT/build/OSX/Release
$ cmake ../../..
$ cmake --build . --target install

8. Run it!
We need to run at least k (defaults to 4) nodes. The first node must be run with -f (--first_node) flag. The other nodes should be run with “-b bootstrap_contacts”.
Now we can use the available commands to store and retrieve values.

Friday, April 27, 2012

Kickstarting Android Development


Here are some tips for those beginning Android development, specially at Umit Google Summer of Code '12/


0. Its important that you develop your application on a real phone. The emulator works good for most of the simple tasks but you will not get a real understanding of the performance of your application. Also, its important to test against the real world WiFi and 3G network. Networking is not the best parts of the emulator. You dont want to waste time on that.

1. Use the MOTODEV studio - its a customized Eclipse specially for Android development. It has built in support for both SDK and NDK. You can get it by creating an account here.

2. If you are writing native code and have segfaults or crashes, you can get the stack traces from /data/anr/traces.txt For applications already out in the market, you could use remote-stacktrace to collect the logs from the users.

3. Build your app for Android 2.2 and up. But make sure you test it on Gingerbread and Ice Cream Sandwich as well.

4. Just for the kicks, grab the android source code, and build it on your machine.

5. Android devices are getting really powerful - think 1.5GHz dual-core ARMv7 devices as becoming everyday phones. So you can actually think of porting any popular Linux tool to Android. There are lots of tools that Android is still missing.

6. If you want to do automated testing, you can write shell/perl scripts and run it from your computer that calls various activities (intents) in your app using adb. You can pass parameters to intents and build a complete testing suite for all your activities. More information on this coming soon.

7. Build fast, Release fast. Users are the best testers.

8. Use the inbuilt tcpdump to view raw packets.

9. To profile your application use the Android traceview 

Nmap for Android

We are planning to use Nmap as a major component for the Android Network Scanner. With last year's Google Summer of Code, I had started writing my own host discovery and port scanning methods. Since Nmap covers almost all functionalities that we want in a Network Scanner, it is a better idea to integrate Nmap into the Network Scanner.

Previously, I was using a cross-compiled binary using the method described on the Nmap website by Vlatko Kosturjak.
Here is another script for cross-compiling Nmap for Android. It downloads the arm toolchain from Sourcery, Nmap source and builds Nmap using the arm compiler. Currently tested on Ubuntu 11.04.

Wait for the next release of the Android Network Scanner this summer!

P2P Routing in OpenMonitor : GSoC 2012

As April approaches, the adrenalin level rises in every students’ body for one reason - The Google Summer of Code aka GSoC. GSoC is getting bigger every year and I am proud to be part of it this year.

My name is Narendran Thangarajan and I am implementing the P2P routing algorithm for the OpenMonitor project as part of GSoC 2012. This blog will give a short description of the process I went through to make it to coding for OpenMonitor project through GSoC 2012.

I chose Umit as my organisation. Why? Take a look at their list of interesting projects and you would understand the reason. Among all these projects, the relatively new "OpenMonitor" project is my favorite. I loved their idea. As it sounds, its a Monitor for the Internet. It keeps track of all the connections on the Internet and has the ability to trace back the reason of a network blockage or blackout during crises. Simply put, it frames a nervous system of the Internet by implementing a peer-to-peer topology of nodes scattered around the globe. And that’s totally awesome! I really wanted to be part of this wonderful project. So, when did I start to get involved? It was last year (2011) May. I worked with Luis A. Bastiao Silva, who is a mentor at Umit. He was really, really helpful. He guided me right from basics of git and helped me out at every stage of the project. I hacked a bit on Umit Network Scanner, Packet Manipulator and OpenMonitor then. I even made a talk on Umit Network Scanner at Pycon India 2011.

Then came the news of GSoC 2012. And Umit made its way to the selected organisations list this time too! My joy knew no bounds to know that I had the chance to work on my favourite project as part of the legendary Google Summer of Code.

I studied the code of Desktop Agent and Mobile Agent of OpenMonitor and I found that the inter-peer routing was in a prototype-like stage and it really needed to be tweaked up for further enhancements to happen. Thus I started discussing the problem with Zubair Nabi, Luis and Adriano and finally settled with implementing a Kademlia based P2P implementation for the inter-peer communication among the desktop agents, mobile agents and the aggregator. We initially planned to make our own implementation from scratch in Python, but later planned to use C++ Kademlia libraries and writing wrappers in Python and Java.

And after all the discussions, I was able to fine tune my project proposal and submit my proposal. The list of accepted proposals were out by GMT 1900 hours on April 23rd, 2012 on the Google melange site and I was glad to see my name on the list. Now, I am going to work in one of the coolest projects in the open source world as part of the Google Summer of Code. I am really proud and happy to be at this place.

Monday, April 23, 2012

Selected students for Google Summer of Code 2012

This year, Umit Project have a record amount of proposals covering several of our proposed (a not proposed) ideas!

Google is being extremely generous with Umit, and allowed us to accept 9 students this year. Seriously, it felt like Christmas time! THANK YOU GOOGLE!

Please, join us welcoming the 9 brilliant students that will be coding like crazy this summer:


Narendram Thangarajan (India) - OpenMonitor Inter-Peer routing mechanism

Narendram was so committed to this GSoC, that he just started working on code since after sending his proposal. I've got the feeling that if we rejected him, he would still do this work no matter what! Of course he more than earned his slot, and he'll be working on adapting the Desktop and Mobile agents to communicate using a DHT protocol.

Tianwei Liu (China) - OpenMonitor Desktop Agent

Tianwei will be taking the Desktop Agent to the next level, and guarantee it works on all our platforms, shows useful connectivity stats for our users and that we have a strong release by the end of this Summer, as well as releasing an alpha pretty soon!

Zhiwei Dai (China) - OpenMonitor iOS Mobile Agent

If you were sad you couldn't run the Mobile Agent because you own an iPhone, iPod Touch or iPad you now have a reason to rejoice! iOS users will soon be able to report and verify internet connectivity in their regions with the same ease of use of the android version, but with the Apple mojo ;-) Zhiwei will make sure this agent will also support the DHT protocol so it can communicate with all other nodes and the aggregator.

Orçun Avsar (Turkey) - OpenMonitor Aggregator

This is how we call our main OpenMonitor site, from where people can freely check on internet connectivity status in anywhere on the globe. Orçun is a got an extremely important task of making sure aggregator is scalable, resilient and useful to our users. He's also going to invest on developing new maps and perspectives on connectivity issues so that the world can get an accurate perspective of what is going around the world.

Gautam Bajaj (India) - OpenMonitor Android Mobile Agent

Zubair managed to develop an awesome Android Mobile Agent for us last gsoc, and now he'll be mentoring Gautam to make an even better version of that agent and provide a nice interface for users to easily check on connectivity in their region without having to resort to the aggregator. Gautam was a successful Umit Summer of Code student from last year, when he managed to pull off the Mobile Sniffer for Android from scratch!

Alin Predoi (Romania) - Site Status

Whoever runs a company, with a site and several servers knows how hard it is to keep track of availability, and making sure that eventual (or even frequent) site or service crashes don't drive away users with a bad impression. Site Status is an attempt to provide a site with an availability status page just like the one on twitter and google app engine, and help users defer traffic to that page whenever site goes down so that users don't face an ugly error page and understand that site is going through difficulties and is going to be back in a few. That's going to be all about Alin's Summer this year, and I'm sure he's going to deliver a stunning app for Google AppEngine that people will love to use with their websites.

Amit Pal (India) - Network Administrator Improvements

Amit is another of these students that doesn't care if they're in or out of GSoC. He's just so eager to contribute, that he can't hold his fingers. He was another successful Umit Summer of Code student from last year, and even after summer he just kept contributing code non-stop. Hey Amit, I think you're going to have a lot of fun with Network Administrator this year, but make sure you find some time to get a tan, ok?

Gorjan Petrovski (Macedonia) - Umit Network Scanner for Android Improvements

Last year, Angad managed to deliver a very nice network scanning app for Android and we even published to Google Play. It is being a reasonable success so far, with nice amount of downloads, and no complains so far! Gorjan, will bring this app to the next level, by cross compiling Nmap for Android and revamping the interface to make you feel like you're scanning from your desktop. Make sure you don't miss this handy and useful tool in your Android!

Serdar Yigit (Turkey) - Packet Manipulator / Network Scanner

The Umit Network Scanner and the Packet Manipulator are the "senior" apps in our organization, and there is being a while since last release. Serdar is going to put an end to this fast and will make sure we deliver some good new features and release a stable version of each of these applications during his Summer of Code.


Welcome to the team guys! We're looking forward for a great Summer with all of you!




Tuesday, March 27, 2012

Google Summer of Code Student Application is on!

If you're looking for some adrenaline this Summer developing some Open Source Software for a big Open Source Organization, have this software published by them and make money in the process, this is your chance!

Summer of Code is a program that Google runs since 2005, and it gathers real world Open Source developers to mentor students to develop Open Source softwares during Summer while making US$ 5,000.00 in the process. Not to mention the awesome t-shirt all students get, as well as some nice Google swags and a beautiful participation certificate issued by Google in student's name.

Sounds like a great boost to your resume!

If you're up for it, consider applying for Umit Project! We definitely have the coolest ideas out there, and a nice and welcoming team!

But hurry! Student application deadline is April 6th 19:00 UTC!



Friday, March 16, 2012

Monday, March 05, 2012

Recent updates

Among our most recent updates and developments at Umit Project:

Some Ohloh factoids about Umit Project...







Sunday, February 26, 2012

Google Summer of Code Mentor Summit 2011 Photo!

Hey Guys! I've found the 2011 GSoC Mentor Summit big photo, with all mentors. Umit is there! :)
I know that the Summit has occurred in october 2011, but I'm posting it here anyways. Better later than never.