Copyright (c) 2014 cozybit Inc.
All rights reserved


INTRODUCTION
############

distro11s is a quick and dirty ephemeral roll-your-own linux distribution based
on Debian that adds packages and a suitable kernel to play with mesh
networking.  It is "ephemeral" because all of the work is meant to be pushed
upstream regularly.  So most folks who want to use mesh will pull their code
from upstream, not from us.  But we need a coherent way to track and test the
latest and greatest.  This document explains all the steps needed to build
distro11s.  For details on the supported target boards, see the README files in
the board/ directory.

distro11s is typically deployed uniformly on several identical nodes.  Each
node needs its own name and IP addresses.  To save developers from having to
maintain special DHCP configurations for assigning these, distro11s uses a
simple host number.  The host number is the only board-specific configuration
value, and it lives in /etc/distro11s-hostnumber.  It is typically set by an
installer that increments his own counter every time he successfully installs
distro11s on a target.  It is perfectly legal for /etc/distro11s-hostnumber to
not exist.  But when it does exist, it influences various system settings, most
notably static IP addresses and the hostname.  See distro11s.sample.conf for
more details.

SAFETY AND SECURITY
###################

It should be noted that little effort has been made to make distro11s safe and
secure.  Building it does require super user access for all kinds of unsavory
tasks such as formatting file systems, blowing away the file system staging
directory, etc.  When distro11s wants to do something as the super user, it
generally warns you and requires your OK.  But it is entirely possible that a
poorly or hastily written distro11s config file could result in blowing away
your entire root file system.  Seriously.

It should also be noted that some features (such as the SSHFS automount) are
handy for developers, but have inherent security flaws.  That feature in
particular adds a key with no password to your authorized_keys file
automatically!  If you were to enable this feature and distribute the distro11s
file system, anybody with that file system and a link to your laptop could log
in to your development machine!  Seriously!  You will be warned at build time
about this sort of stuff.  But seriously!  Be careful.

BINARY RELEASES
###############

If you have a binary release, you must untar it as sudo because it contains a
rootfs that has some files that belong to root.  Yeah, not so pretty.  Sorry.
Anyway, at that point, follow the board-specific README for your binary to
learn how to deploy it to your hardware.

SETTING UP YOUR DEVELOPMENT SYSTEM
##################################

We build distro11s on Ubuntu 11.04.  Here's how:

1. Install the packages that you'll need to build the distro.  Note that this
   list of packages is what's required to build all packages for any
   conceivable board.

   $ apt-get install python-dev uml-utilities git debootstrap autoconf libtool flex bison qemu

   $ wget -O /usr/share/qemu/pxe-rtl8139.bin \
     "http://svn.savannah.gnu.org/viewvc/*checkout*/trunk/pc-bios/pxe-rtl8139.bin?root=qemu"
   $ wget -O /usr/share/qemu/pxe-e1000.bin \
     "http://svn.savannah.gnu.org/viewvc/*checkout*/trunk/pc-bios/pxe-e1000.bin?root=qemu"

   $ sudo apt-get remove gnu-fdisk

2. Create a configuration file for your build.  This is just a bunch of bash
   variables that influence the build.  The recommended way to create a config
   file is to copy distro11s.sample.conf to distro11s.conf and edit it.  It's
   loaded with comments, so read it for more details.

   If you want to support simultaneous builds for different boards, you can
   have a different config file for each board.  Just set the DISTRO11S_CONF
   environment variable to the config file that you want to use.  You may wish
   to set the DISTRO11S_SRC variables in different config files to keep the
   per-package builds separate.  DISTRO11S_OUT, on the other hand, should
   co-exist fine for different boards.

FETCHING THE CODE
#################

This is a distribution.  Accordingly, it is made up of many packages.  To fetch
all of the packages into your $DISTRO11S_SRC directory, say:

$ ./scripts/fetch.sh

BUILDING THE CODE
#################

To build all of the packages, say:

$ ./scripts/build.sh

Note that all targets use debbootstrap to prepare the basic rootfs and must be
built as root.  You will be warned if/when this happens.

ADDING A NEW SUPPORTED BOARD
############################

Create a directory in board/ with the name of your board.  Consider cloning the
board directory for a similar target.  Note the board/<board>/<board>.conf
file.  It contains board-specific overrides that can be used to tweak the
build.

PROVISIONING DISTRO11S
######################

The provisioning.sh script allows you to provision distro11s onto a hard drive.
In order to do this, please follow the next steps:

 - Get a hard drive and connect it to your dev machine.
 - Get its device name possibly from fdisk -l or dmesg (e.g., /dev/sdb)
 - Run the next command:
   $ ./scripts/provisioning.sh -d <devicename>
 - Install the hard drive back on the target

In order to make the installation process easier, a USB Installer can be
created using this script. This USB installer will allow you to install
distro11s by just plugging the USB key into the target. To create a USB
installer, please follow the next steps:

 - Connect your USB device to your dev machine (2Gb minimum)
 - Get its device name possibly from fdisk -l or dmesg (e.g., /dev/sdb)
 - Run the next command:
   $ ./scripts/provisioning.sh -d <device> -i
 - Unplug the USB device and connect it to the TARGET
 - Boot from the USB, and the installation process will begin once is booted
 - Once distro11s is installed, the target will reboot, loading distro11s

This USB installer approach is convenient when installing distro11s on many
machines.  Note, however, that data throughput between the development machine
and a USB drive varies dramatically.  In some cases, copying the half GB or so
of data may take 30 minutes or even longer.  Consider using a
higher-performance external USB harddrive.

TIPS AND TRICKS
###############

-- Many parts of the distro11s build system use stamp files in the
   out/<board>/stamps/ directory.  This way, scripts can be run multiple times
   without repeating very time-consuming steps that might be unnecessary.  To
   force various parts of the build to execute again, delete the appropriate
   stamp file.  Inspect the contents in the stamps directory for more details.

-- The stamp system is not very sophisticated at this time.  It's mostly used
   for the release procedure, which is a complete clean build.  As you develop,
   you may find it easier to hop around the DISTRO11S_SRC directory manually
   invoking make/make install (or whatever).  Alternatively, you can invoke the
   build script with FORCE_BUILD=1.  For example, if you want to execute ALL of
   the build steps for the kernel (including coping the config over!) you can
   say:

   FORCE_BUILD=1 ./scripts/kernel.sh

-- Copmile / test cycle in qemu:

	1. make a change
	2. rebuild your kernel, 'FORCE_BUILD=1 ./scripts/kernel.sh'
	3. push your new modules to your running qemu, './scripts/push -u root@${DISTRO11S_HOST_IP}'
	4. test.

-- Avoid using sudo.  But if you must (e.g., mount, debootstrap, etc.), use the
   root_check function before invoking sudo.  This alerts the user.

-- If you're adding a new build script, try to make it repeatable.  That is, if
   I blow away all of the stamp files and invoke your build script, it should
   do the right thing.

-- Make a minor change to a package and just want to push your changes to a
   node that you already provisioned?  Try this:

   $ FORCE_BUILD=1 ./scripts/<package>.sh
   $ ./scripts/push.sh root@ipaddr-of-target

-- If you want to save yourself the trouble of entering your password every
   time you ./scripts/push.sh, you must set up the root user
   _on_your_dev_machine_ to have a password-less ssh key, and you must
   ssh-copy-id that key over to each device to which you wish to push.sh.  The
   reason is that push.sh rsyncs as sudo.

-- Got some operation you want to perform on each node?  Consider using the
   foreach.sh script.  You'll have to set the DISTRO11S_NUM_NODES variable in
   your config file, then you can say something like:

   ./scripts/foreach.sh './scripts/push.sh [-u root@$HOSTNAME.local]'
   
   or
    
   ./scripts/foreach.sh 'ssh root@${HOSTNAME}.local "<any_command>"'

   Your little bit in single quotes will get executed once for each node.  The
   vars that are set by the foreach.sh script (currently) include:

   HOSTNUM: The hostnumber

   IP: The IP address of the node formed by adding his hostnumber to
   DISTRO11S_STATIC_IP.

   HOSTNAME: The hostname of the node formed by adding his hostnumber to
   DISTRO11S_HOSTNAME.

   Note: you can customize the range of nodes you would like to sync by setting
   DISTRO11S_START_NODE and DISTRO11S_END_NODE in your distro11s.conf.

   Note: to connect to qemu with gdb, while inside gdb type:
   (gdb) target remote ${DISTRO11S_HOST_IP}:1234

-- If you are working on an older machine that does not support hardware
   virtualization, you will have to disable kvm:

diff --git a/board/qemu/qemu.sh b/board/qemu/qemu.sh
index 020daf1..0b7f730 100755
--- a/board/qemu/qemu.sh
+++ b/board/qemu/qemu.sh
@@ -27,7 +27,7 @@ ${QEMU} -kernel ${KERNEL} -hda ${ROOTFS} \
        -append "root=/dev/sda combined_mode=ide console=ttyS0" \
        -nographic -net nic,model=e1000 -net tap,ifname=${IFNAME},script=no \
        -gdb tcp::1234 \
-       -enable-kvm -smp 2
+       -no-kvm -smp 2