NICER Off-The-Grid Living: 2016

Tuesday, March 1, 2016

Mapping The New (Serval) World

Humans tend to understand stuff better that they can see, stuff that they can visualize. Having a good map when entering new terrain was and is always important. This is not only true for the real world but also for the digital world. Even without knowledge of the real position of nodes one can still visualize how every thing is connected either through traceroute information or what is present in our routing table.

For this project lets assume the following network setup with 9 nodes involved:

Network Topology

My first approach was to simply parse the output of servals route print command, use pythons networkx library and plot it all with matplotlib:


Visualization using matplotlib

Worked quite nice, was very easy to implement but is really kind of limited when interacting with the graph and also has quite some dependencies. Nevertheless the script can be found here.

While playing around with networkx I found out that it can also output any graph as JSON. This can then be easily parsed by JavaScript libraries such as D3js in your browser. After writing a short prototype in python and testing it I added a route json command to serval to directly output JSON data, eliminating the need for python.

To visualize everything I just merged the official force-directed sample with another sample from the internet plus some minor modifications to prettify stuff and was done with it.

Putting all these pieces together - assuming servald has route json support and is already running plus route-network.html is in the current directory:

$ servald route json > route.json
$ python -mSimpleHTTPServer

Now use a webbrowser and go to http://localhost:8000/route-network.html and you're done!

Visualization using D3

Of course it would be cool to have this route data available as a restful call provided by serval. Then we could easily integrate it into any (Web-)GUI and maybe someone will develop a more sophisticated visualization where one can copy'n'paste SIDs or send MeshMS to specific nodes! Wouldn't that be great?! :)

The scripts, patches, installation instructions and sample data can be found in the serval-vis repository.

Tuesday, February 16, 2016

One Proxy To Rule Them All

Not all client/server applications make sense in a peer-to-peer environment or over a delay-tolerant network. Sometimes data is just too big or node specific to be shared over servals rhizome - think various local community websites - or doesn't make sense over DTN - like an SSH connection.

It would be way too time consuming porting these "legacy" applications to serval and giving them native MSP support. A quick and easy way to still use them is already there in servald. It supports the msp listen and msp connect commands that act similar to netcat.

Manual MSP

So lets get a local webserver into serval mesh:

root@n9:/tmp/pycore.35289/n9.conf# cat index.html
<html><body>test</body></html> 

root@n9:/tmp/pycore.35289/n9.conf# python -mSimpleHTTPServer
Serving HTTP on 0.0.0.0 port 8000 ...

Use a new terminal for testing and setting up the forwarding:

root@n9:/tmp/pycore.35289/n9.conf# curl http://127.0.0.1:8000
<html><body>test</body></html> 

root@n9:/tmp/pycore.35289/n9.conf# servald id self
1
sid
C39309B71A19274E4947A99EE721DA0638D3E8596C2EA2821D0F10DEA5F22729

root@n9:/tmp/pycore.35289/n9.conf# servald msp listen --forward=8000 80

That's it!

So now lets try to reach this site through msp from another node.

root@n4:/tmp/pycore.35289/n4.conf# servald msp connect C39309B71A19274E4947A99EE721DA0638D3E8596C2EA2821D0F10DEA5F22729 80
GET / HTTP/1.0

HTTP/1.0 200 OK
Server: SimpleHTTP/0.6 Python/2.7.6
Date: Tue, 16 Feb 2016 12:25:43 GMT
Content-type: text/html
Content-Length: 36
Last-Modified: Fri, 12 Feb 2016 12:48:04 GMT

<html><body>test</body></html>

So it works! But you don't want to setup each and every connection manually - Software should automatically map SIDs to serval and take care of connection setup and routing. Usually these kind of problems are solved by using a proxy as it is done by the tor project for example.

Automatic Connection

The SOCKS5 protocol has a nice feature that makes our life much easier. It can not only operate on a given IP address to relay a connection but also be given a FQDN and resolve it itself!
So we need a proxy server that intercepts all domain names and check if it is a valid SID - 64 bytes all hex. Easy enough, next we need to invoke msp connect with the supplied SID and port, relay stdin and stdout from servald to the connected TCP session and we're done.

A quick proof-of-concept socks5-serval was put together with a modified version of python-socks5, the changes were minimal and the result works quite good, it can still be used for regular IP connections as well as with FQDNs.

Here it is in action:

root@n4:/tmp/pycore.35289/n4.conf# python python-socks5-serval/socks5.py

Open another terminal to test the proxy server.

root@n4:/tmp/pycore.35289/n4.conf# curl --socks5-hostname localhost:1080 http://C39309B71A19274E4947A99EE721DA0638D3E8596C2EA2821D0F10DEA5F22729
<html><body>test</body></html>

It worked! In our other terminal window where the proxy is running some debug output should also appear:

root@n4:/tmp/pycore.35289/n4.conf# python python-socks5-serval/socks5.py
[INFO:] Got one client connection
[INFO:] Checking atyp: 3
[INFO:] Client wants to connect to C39309B71A19274E4947A99EE721DA0638D3E8596C2EA2821D0F10DEA5F22729:80
[INFO:] Starting transform thread
[INFO:] Starting serval mode C39309B71A19274E4947A99EE721DA0638D3E8596C2EA2821D0F10DEA5F22729
[INFO:] Starting Resending
[INFO:] Client quit normally

Instead of checking for an 64 byte hex string to identify SIDs one could also say that clients must use the TLD .serval to identify a serval address. Then a proxy would only have to check the TLD to be sure but on the other hand 64 byte hex is not a FQDN so we should be safe.

SOCKS5-serval was just a quick'n'dirty hack and is not really production ready but it demonstrates quite well how endless the possibilities for serval meshs are and how our beloved regular applications can continue to work in a full serval-routed network.

Tuesday, February 2, 2016

SIM Sala Bim - MilSim meets Network Emulation

We are currently developing our own network-simulation-emulation mix with complete flexible movement patterns for all nodes, qemu backends and various networks types. The whole system is called MiniWorld and has support for walking and visualizing everything on OSM maps. When talking about different scenarios and creating a scenario editor that got me thinking... A long time ago I used to play a military simulation game called Operation Flashpoint that came with a good scenario editor and good modding support. Just recently the successor, Armed Assault 3, got released for linux.. a perfect excuse to install the game on my workstation!

After fiddling around with it a bit I wrote a short script to periodically log all coordinates of "players". This script must be triggered automatically at the beginning of the mission to record all positions.

Settings for trigger

We then wrote a parser for the generated rpt file and fed these coordinates live into the MiniWorld backend, skipping our own movement code. Next step was to get a screenshot of the map, integrate it into Leaflet to visualize player movement outside of the game itself which was pretty straight forward except for some coordinate conversion problems :)

For demonstration purposes I used the MiniHattan map and created a short "emergency scenario". There are five different people involved, all having some device running an instance of serval. Each device has a fixed radius for wireless transfers, the question was is a drive-by enough time to transfer files? What size? Tests like these can easily be done here because AI, multiplayer, cars, drones and stuff like a physics engine are already built into the game, we could even have depleting batteries modelled in our backend and turn off some of the devices after some time.

Easy Scenario Editor

The whole system in action can be seen in the following video (no audio):

Monday, January 25, 2016

Hop Hop Hop Hop STOP!

While playing around with serval in various different network setups in core-network I found some odd behaviour. Getting to run serval to run in the simulator is describe in this article. In this blog post I am going to describe what I have observed.

The network setup

A simple chain of ServalNodes connected pair-wise, each machine with two interfaces - one neighbour on each interface.

N1 <-> N2 <-> N3 <-> ... <-> N17 <->N18

Chained serval setup in core-network.

The problem(?)

To see how the routing and mdp based services perform we did some testing. First connections were verified by using mdp ping.

root@n1:/tmp/pycore.46656/n1.conf# /home/meshadmin/serval-dna/servald mdp ping F40716A16538D25EA01134139056F02D23F9246583012048B4DBA4BBB46A594E
MDP PING F40716A16538D25EA01134139056F02D23F9246583012048B4DBA4BBB46A594E: 12 data bytes
F40716A16538D25EA01134139056F02D23F9246583012048B4DBA4BBB46A594E: seq=1 time=194ms hops=64 ENCRYPTED SIGNED
F40716A16538D25EA01134139056F02D23F9246583012048B4DBA4BBB46A594E: seq=2 time=198ms hops=64 ENCRYPTED SIGNED
F40716A16538D25EA01134139056F02D23F9246583012048B4DBA4BBB46A594E: seq=3 time=184ms hops=64 ENCRYPTED SIGNED
^C--- F40716A16538D25EA01134139056F02D23F9246583012048B4DBA4BBB46A594E ping statistics ---
3 packets transmitted, 3 packets received (plus 0 duplicates, 0 ignored), 0.0% packet loss
round-trip min/avg/max/stddev = 184/192.000/198/5.888 ms (3 samples)

So n18 is reachable from n1, slow ping but for 17 hops it is still okay.
If we try to do the same with an mdp trace, after all there might be different network paths, we end up with the following:

root@n1:/tmp/pycore.46656/n1.conf# /home/meshadmin/serval-dna/servald mdp trace F40716A16538D25EA01134139056F02D23F9246583012048B4DBA4BBB46A594E
Tracing the network path from 8410960D885656669C1B4C4AA56E4339B171E9285A254A1863A80FF7F483A141 to F40716A16538D25EA01134139056F02D23F9246583012048B4DBA4BBB46A594E
INFO: Local date/time: 2016-01-25 11:58:43 +0100
INFO: Serval DNA version: START-3478-g8e223b5
ERROR:network_cli.c:317:app_trace() overlay_mdp_send returned -1, Timeout waiting for reply to MDP packet (packet was successfully sent).

Trying the same on n17 - only 16 hops away - we get the following:

root@n1:/tmp/pycore.46656/n1.conf# /home/meshadmin/serval-dna/servald mdp trace 614222015D22DBCD65FD79B8C311E12DDE6CDF086E71B0D9CD9746AFCBE02E71
Tracing the network path from 8410960D885656669C1B4C4AA56E4339B171E9285A254A1863A80FF7F483A141 to 614222015D22DBCD65FD79B8C311E12DDE6CDF086E71B0D9CD9746AFCBE02E71
0:8410960D885656669C1B4C4AA56E4339B171E9285A254A1863A80FF7F483A141
1:D6A5E4F4B6EAFE3A22B82742280322695601F2C5C9DF8A8AD6C5067F04E1E139
2:D158F542C7001C55350C67A6B975B08C2B8D2A63A60EC32FEC1712D2206F5C32
3:75A61C0DF5A46BEEEEB28F18C956ACA8C696DED4CBD4C4FE31A54AE2CB26D077
4:424B43C7E6CC80BEEE761F100217824CCEF201ECEA55D14B7F4B205FFA4FF630
5:2E0FCBB04D6C5DDE8D145206A786683164A4443E0AD214E79F69D79EFA1B1D0D
6:1B669680A1555490BC271881362C443E43131ADCB94BA285F6BA1B3B71DC9853
7:9CBAFAB66A0808573ED0FCA42376BC6B527AB39DEB89FFB344EE9381BD105936
8:B9B737D8CFFD5FBFDC5A20505159EC2798D60C4CC4F39B8B0AB6D4240B931C20
9:2AD0422C040079FF320C934D2E98DC9D154896CA1E492FBB3DF720FCFE7FD009
10:BA5288F511246B8C5D77B12B78C6D46DA92D1B28008F57CDAA9F278B9EF54C2B
11:C42A672FB4D19AA59DE47200CC581220CEC36CBE935F8FEF46027CA2604CA072
12:9A3B340EAF09679FF5FCAF720418DA7E2D959018A2C55227EB95C9928B53A52C
13:8EF8016470B665CED119B3CB2F76406952567E2DC16C74AC352B831F1614DB3C
14:57E8A11D20C8970733A5A0776FEA370C0440136D7D7873C1255A26A13924D155
15:79314E660CE67A6DD19E5CAB209CE619EA973A13387D8FE32A35685C1971796E
16:614222015D22DBCD65FD79B8C311E12DDE6CDF086E71B0D9CD9746AFCBE02E71
17:79314E660CE67A6DD19E5CAB209CE619EA973A13387D8FE32A35685C1971796E
18:57E8A11D20C8970733A5A0776FEA370C0440136D7D7873C1255A26A13924D155
19:8EF8016470B665CED119B3CB2F76406952567E2DC16C74AC352B831F1614DB3C
20:9A3B340EAF09679FF5FCAF720418DA7E2D959018A2C55227EB95C9928B53A52C
21:C42A672FB4D19AA59DE47200CC581220CEC36CBE935F8FEF46027CA2604CA072
22:BA5288F511246B8C5D77B12B78C6D46DA92D1B28008F57CDAA9F278B9EF54C2B
23:2AD0422C040079FF320C934D2E98DC9D154896CA1E492FBB3DF720FCFE7FD009
24:B9B737D8CFFD5FBFDC5A20505159EC2798D60C4CC4F39B8B0AB6D4240B931C20
25:9CBAFAB66A0808573ED0FCA42376BC6B527AB39DEB89FFB344EE9381BD105936
26:1B669680A1555490BC271881362C443E43131ADCB94BA285F6BA1B3B71DC9853
27:2E0FCBB04D6C5DDE8D145206A786683164A4443E0AD214E79F69D79EFA1B1D0D
28:424B43C7E6CC80BEEE761F100217824CCEF201ECEA55D14B7F4B205FFA4FF630
29:75A61C0DF5A46BEEEEB28F18C956ACA8C696DED4CBD4C4FE31A54AE2CB26D077
30:D158F542C7001C55350C67A6B975B08C2B8D2A63A60EC32FEC1712D2206F5C32
31:D6A5E4F4B6EAFE3A22B82742280322695601F2C5C9DF8A8AD6C5067F04E1E139
32:8410960D885656669C1B4C4AA56E4339B171E9285A254A1863A80FF7F483A141

Conclusion

16 hops are more than enough, nobody should ever need more... well, apparently ping can do more, didn't test other mdp based services. Delay tolerant services such as rhizome are obviously not effected and perfectly spread data around.

Rhizome spreading data.

Running serval in core-network simulator

For testing purposes it can be really helpful to run your software in a simulator with various different network setups. Here is a short introduction how to use serval with core-network.

Running serval wifi-mesh mixed with static wired nodes.

Installation

Installation of core-network in Ubuntu (here 14.04) is straight forward:

sudo apt-get install core-network

That's it!
For other platform consult the official installation instructions on the core homepage.

The next step is getting serval running as a core-network service. Therefore we need of course serval itself. In this tutorial I am logged in as user meshadmin and have compiled servald with "./configure --prefix=/home/meshadmin/serval-conf".

A set of sample configuration files can be found on github, detailed instructions are following here.

First we need to enable the loading of custom services in /etc/core/core.conf by adding the following line:

custom_services_dir = /home/meshadmin/.core/myservices

In this directory we need to put a small python script describing our service. The contents of this serval.py should look something like this:

''' serval service.
'''

import os

from core.service import CoreService, addservice
from core.misc.ipaddr import IPv4Prefix, IPv6Prefix

class ServalService(CoreService):
    ''' servald as a service.
    '''
    # a unique name is required, without spaces
    _name = "ServalService"
    # you can create your own group here
    _group = "Mesh"
    # list of other services this service depends on
    _depends = ()
    # per-node directories
    _dirs = ("/home/meshadmin/serval-conf/etc/serval","/home/meshadmin/serval-conf/var/log", "/home/meshadmin/serval-conf/var/log/serval", "/home/meshadmin/serval-conf/var/run/serval", "/home/meshadmin/serval-conf/var/cache/serval","/home/meshadmin/serval-conf/var/cache/serval/sqlite3tmp","/home/meshadmin/serval-conf/var/cache/serval/blob")
    # generated files (without a full path this file goes in the node's dir,
    # e.g. /tmp/pycore.12345/n1.conf/)
    _configs = ('/home/meshadmin/serval-conf/etc/serval/serval.conf', "mesh-start.sh", )
    # this controls the starting order vs other enabled services
    _startindex = 50
    # list of startup commands, also may be generated during startup
    #_startup = ('/home/meshadmin/serval-dna/servald start',)
    _startup = ('bash mesh-start.sh',)
    # list of shutdown commands
    _shutdown = ('/home/meshadmin/serval-dna/servald stop')

    @classmethod
    def generateconfig(cls, node, filename, services):
        ''' Return a string that will be written to filename, or sent to the
            GUI for user customization.
        '''
    if filename == "/home/meshadmin/serval-conf/etc/serval/serval.conf":
            cfg = "debug.rhizome=true\n"
            cfg += "debug.verbose=true\n"
        cfg += "interfaces.0.match=eth*\n"
        cfg += "interfaces.0.socket_type=dgram\n"
        cfg += "interfaces.0.type=ethernet\n"
    elif filename == "mesh-start.sh":
        cfg ="#!/bin/sh\n"
        cfg +="/home/meshadmin/serval-dna/servald start\n"
        cfg +="sleep $[ ( $RANDOM % 10 ) + 1 ]s\n"
        cfg +="for i in `ifconfig | grep \"inet addr:10.\" | cut -d\":\" -f 2 | cut -d\".\" -f1,2,3`\n"
        cfg +="do\n"
        cfg +="/home/meshadmin/serval-dna/servald scan $i.255\n"
        cfg +="done\n"
    else:
        cfg = ""
        return cfg

# this line is required to add the above class to the list of available services
addservice(ServalService)

This service starts a new instance of servald, pre-configured to use any ethernet interface available and starts a discovery scan on all local interfaces after a random sleep. Each instance gets its own fresh copy of the instance directory! Core uses ethernet devices for all types of networks, even WiFi - there configuring servald for all ethernet interfaces is enough.

Note: You need to edit the paths according to your setup!

To use this service it needs to be added to the __init__.py file in the myservices directory:

__all__ = ["sample", "serval"]

That is basically it but to make life easier here are a few helpers I also use.
Adding to the standard node configurations is done by editing ~/.core/nodes.conf. This is where I added the following two entries:

7 { ServalNode pc.gif pc.gif {DefaultRoute ServalService} netns {simple serval node} }
8 { ServalMesh mdr.gif mdr.gif {OLSRORG ServalService} netns {simple serval mesh node} }

The first entry, ServalNode, creates a simple node only with our serval service and a default route enabled - classic laptop/pc icon. ServalMesh is a pre-configured node with serval and standard OLSR as a mesh routing protocol - using the standard router icon.

Note: You need olsr installed prior to using it here (sudo apt-get install olsrd) and change the IP config in core to use a /24 or something like that instead of /32 because otherwise serval won't find any peers despite the fact that olsr routing works and you can ping (icmp not mdp!) all addresses.

For live inspection defining custom widgets is also very helpful. This can be done by editing ~/.core/widgets.conf and adding a few lines like these:

15 { {serval peer count} {/home/meshadmin/serval-dna/servald peer count} }
16 { {serval id self} {/home/meshadmin/serval-dna/servald id self} }
17 { {serval rhizome list} /home/meshadmin/serval-tests/listfiles }
18 { {serval mesh scan} /home/meshadmin/serval-tests/mesh-scan.sh }

The first widget displays the number of peers currently known of the inspected serval instance while the second entry shows the SID of the selected node. Widgets 17 and 18 both use helper scripts that can be found here and display the number of files in rhizome store and trigger a new neighbour scan.

Note: Again change the paths to fit your machine.

The last and final step after adding all these files is to restart the core-network service.

sudo service core-daemon restart

Now you are ready to go!

UPDATE: Changed github repo links for core-serval and serval-tests to new location.