A typical pattern that is used when running systems is to have multiple environments. There's the production environment, which handles the “real” data or traffic, and usually a staging system that consists of “test” or fake data that has no impact on what happens in the real system.

This staging environment is useful because it lets you test changes to your software or configuration in an environment that is pretty close to what it looks like in production, giving your confidence that your changes won't break anything in the real world.

It would be great if we had one for our sensor network - that way we could test changes to any part of the network, and see if they would work in production. This would be useful to test gateway or database changes for example.

• would be useful for testing
• good educational experience being able to walk through the whole process
• one outcome from setting up the staging environment - writing a document that tells the story of what we had to do to set it up (useful for others to read later on)

In the past, folks have sorta set something up like this - the “test” gateway that's running on a raspberry pi. That's a great test platform, and allows the firmware and software team to test things, but the only difference is it's not something that's running all the time. A true staging environment should be running all the time, and should be free for anyone to use for testing whenever they need to.

Here is a starting list of tasks - this can be added to later on:

• Go through tasks, put it onto a google doc for collaboration / documentation
• Get a new xbee and usb-serial adapter for use as a coordinator
• Configure the new xbee with the proper parameters and a new PANID for the staging env
• Plug in the new xbee to scelserver-1
• Create a new postgres database for staging
• Create new database credentials for staging
• Create the actual tables required for the gateway in the staging database
• Create a new gateway instance on scelserver-1
• Create a new systemd service for the staging gateway
• Setup a stubbed weatherbox node that transmits fake samples to the new staging xbee
• Create a grafana dashboard to check on the status of the staging network

We currently use systemd to run the production gateway. Systemd is primarily used by us to manage the gateway process, and make sure that it starts up when the system reboots and well if the process crashes somehow.

More details:

• https://en.wikipedia.org/wiki/Systemd

Simple Example

Here's a simple example to provide a bit more context.

Let's say I had a server sitting somewhere that I wanted to run a website on. I can use python's SimpleHTTPServer to do that. This is a pretty simple, one command webserver that will serve up files in a directory so they're accessible in a browser.

So let's create a directory and run our webserver:

kluong@kserver:~$ mkdir test-webserver
kluong@kserver:~$ cd test-webserver/
kluong@kserver:~/test-webserver$ ls
kluong@kserver:~/test-webserver$ echo "hello world" > index.html
kluong@kserver:~/test-webserver$ python -m SimpleHTTPServer 7000
Serving HTTP on 0.0.0.0 port 7000 ...

In a browser, if I end up going to localhost:7000, I'll see “hello, world”. You can also use the `curl` command in a separate terminal window:

kluong@kserver:~/test-webserver$ curl localhost:7000
hello world

Now this is great, but if you close the terminal window or reboot the machine, the website will be unavailable since the program isn't running anymore. You need something to manage this program, and make sure that it gets run every time the machine is started, and without having someone having to open up a terminal window.

This is where systemd comes in.

A systemd config file could look something like this:

[Unit]
Description=My Website

[Service]
Type=simple
User=kluong
WorkingDirectory=/home/kluong/webserver
ExecStart=/usr/bin/python -m SimpleHTTPServer 7000
Restart=on-failure

[Install]
WantedBy=multi-user.target

TODO: finish this section

Details from the production gateway

You can inspect the current systemd file by going to scelserver-1:

You can run `systemctl status` to check where the unit file is defined:

kluong@scelserver-1:~$ systemctl status xbee-gateway
● xbee-gateway.service - Scel XBee Gateway
   Loaded: loaded (/etc/systemd/system/xbee-gateway.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2021-04-20 14:26:43 HST; 5 months 17 days ago
 Main PID: 661 (run_production.)
      CPU: 16h 58min 11.722s
   CGroup: /system.slice/xbee-gateway.service
           ├─661 /bin/bash /home/scel/control-tower/gateway/run_production.sh auto
           └─725 python gateway_server.py auto

Looks like it's defined at /etc/systemd/system/xbee-gateway.service. Let's take a look at it:

[Unit]
Description=Scel XBee Gateway
After=network.target
After=xbee-pty-bridge.service

[Service]
Type=simple
# Another Type option: forking
User=scel
WorkingDirectory=/home/scel/control-tower/gateway
ExecStart=/home/scel/control-tower/gateway/run_production.sh auto
Restart=on-failure
# Other Restart options: or always, on-abort, etc

[Install]
WantedBy=multi-user.target

It's possible to re-use most of this existing systemd configuration to create the staging staging configuration, the directories and names just have to be changed properly.

We'll be sharing the same postgres instance as our production system - that way we don't have to have a completely new instance of postgres. What we'll want, however is a separate database from what the production data is using, so we can isolate things a bit.

Let's take a look at what the production instance is using.

Turns out, our gateway is hardcoded to use certain values: https://github.com/scel-hawaii/control-tower/blob/master/gateway/src/decoder.py#L127-L128

con = psycopg2.connect("dbname='control_tower' user='control_tower' password='' host='localhost'")

Looks like it's configured to connect to localhost and use the 'control_tower' database through the 'control_tower' user. We'll have to change this so it's not hardcoded if we want to use a different database for the staging instance. But we can revisit that in another change later on.

So we'll need a new database and a new user - how can we do that?

Here are some docs from the postgres website:

• https://www.postgresql.org/docs/9.0/sql-createuser.html
• https://www.postgresql.org/docs/9.0/sql-createdatabase.html

It looks like we can create a new user within postgres by using:

CREATE USER control_tower_staging;

And it looks like we can create a new database using:

CREATE DATABASE control_tower_staging OWNER control_tower_staging;

To be able to do this, you'll need to be be logged in as a user with the `superuser` permission, which the `postgres` user typically has. To do this on the server, you'll just need to switch to the postgres user and run the `psql` command to get a terminal where you can run these commands.

sudo su postgres
psql

Note - be sure to take care when using sudo! This gives you access to do a lot of things to the existing system, including removing system files that you normally wouldn't be able to.

To check the users, use the \du command in postgres:

postgres=# \du
                                   List of roles
       Role name       |                   Attributes                   | Member of
-----------------------+------------------------------------------------+-----------
 control_tower         |                                                | {}
 control_tower_ro      |                                                | {}
 control_tower_staging |                                                | {}
 kluong                |                                                | {}
 postgres              | Superuser, Create role, Create DB, Replication | {}

To check the databases, use the \l command in postgres:

postgres-#
postgres-# \l
                                         List of databases
     Name      |     Owner     | Encoding |   Collate   |    Ctype    |      Access privileges
---------------+---------------+----------+-------------+-------------+-----------------------------
 bears         | control_tower | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 control_tower | postgres      | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =Tc/postgres               +
               |               |          |             |             | postgres=CTc/postgres      +
               |               |          |             |             | control_tower=CTc/postgres +
               |               |          |             |             | control_tower_ro=c/postgres
 kluong        | postgres      | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =Tc/postgres               +
               |               |          |             |             | postgres=CTc/postgres      +
               |               |          |             |             | kluong=CTc/postgres
 postgres      | postgres      | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 template0     | postgres      | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres                +
               |               |          |             |             | postgres=CTc/postgres
 template1     | postgres      | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres                +
               |               |          |             |             | postgres=CTc/postgres

Okay, now the database and user are setup, but we'll also need to setup the tables for the database. There's a schema file in the control_tower repo under the db/ folder that will let you do this.

Make sure the current directory you're in is the control_tower repo. Here's the command you can run as the postgres user:

psql -U control_tower_staging -d control_tower_staging -f db/multi-table.sql

After this is done, this is all the database configuration you'll need to do. There are some additional gateway changes described below.

You can check that the tables were created properly by running the following command:

psql -U control_tower_staging -d control_tower_staging -c "SELECT * FROM pg_catalog.pg_tables WHERE schemaname='public'"

I mentioned earlier we'll have to change the way the gateway works. This is the way the current code connects to the database:

con = psycopg2.connect("dbname='control_tower' user='control_tower' password='' host='localhost'")

https://github.com/scel-hawaii/control-tower/blob/master/gateway/src/decoder.py#L127-L128

What we want to be able to do is specify the database configuration in a separate manner - either using a configuration file, flag or an environment variable. The environment variable is probably the easiest here, so let's go with that. Python programs can read in variables that are set by the environment: https://able.bio/rhett/how-to-set-and-get-environment-variables-in-python--274rgt5

So we can modify the code to do something like this:

db_uri = os.environ["GATEWAY_DB_URI"]
con = psycopg2.connect(db_uri)

We can also use a URI instead of passing in the arguments in the way we did previously. A shell script that calls the gateway would look something like this for staging:

#!/bin/bash

# setup the python env
source ./env/bin/activate 

export GATEWAY_DB_URI="postgresql://control_tower_staging@localhost/control_tower_staging"
python gateway.py /dev/serial/by-id/usb-FTDI_FT231X_USB_UART_DN01DS3L-if00-port0

You'd need to see some RX frames on the tty device to test this properly though, since the gateway doesn't reach out to the database unless it sees a message to decode.

To get this to work in production, you would have to modify the run_production.sh script, and also update the repository production.

Options for testing

• Deploy to production - see what happens
• Deploy to staging - see what happens
• Setup an personal network (on your laptop, or maybe on the server)
• Connect the gateway to a “fake” xbee virtually, using a pty and another script
• Mock out the serial device completely in python

It's not necessarily practical to have hardware to test with all of the time and often hardware might not be available anyways. Luckily, with software we can work around this.

Note: development for the gateway should generally be done in a linux-based environment. The gateway code was primarily designed only for linux-based environments and may not run locally.

You can make a “fake” xbee network using the following python script:

import os
import errno
import time

def symlink_force(target, link_name):
    try:
        os.symlink(target, link_name)
    except OSError as e:
        if e.errno == errno.EEXIST:
            os.remove(link_name)
            os.symlink(target, link_name)
        else:
            raise e

def valid_packets():
    packets = {}
    packets['heartbeat'] = "\x7e\x00\x16\x90\x00\x7d\x33\xa2\x00\x40\xe6\x4b\x5e\x03\xfd\x01\x00\x00\xff\xff\xf0\xfa\x23\x00\x2b\x02\xb2"
    packets['apple'] = "\x7e\x00\x22\x90\x00\x7d\x33\xa2\x00\x40\x9f\x27\xa7\x29\x6c\x01\x01\x00\xff\xff\x80\x6f\x69\x3d\x06\x0f\x71\x7d\x33\x5a\x8a\x01\x00\x76\x01\x22\x00\x6e\x09\x55"
    packets['cranberry'] = "\x7e\x00\x22\x90\x00\x7d\x33\xa2\x00\x41\x25\xe5\x88\x0c\x83\x01\x02\x00\xff\xff\x7c\xf3\x05\x00\xba\x0f\x5c\x08\x05\x00\x20\x73\x3b\x00\xdd\x8b\x01\x00\x7a"
    packets['dragonfruit'] = "\x7e\x00\x24\x90\x00\x7d\x33\xa2\x00\x40\xe6\x72\x7d\x5e\x30\x18\x01\x03\x00\xff\xff\x30\xc8\x07\x00\x6b\x0d\xf4\x00\x06\x00\x00\x00\xb6\x72\x37\x00\xfe\x8b\x01\x00\x00"
    packets['snapdragon'] = "\x7e\x00\x22\x90\x00\x7d\x33\xa2\x00\x40\xa3\x53\x7d\x5e\x20\x9c\x01\x04\x00\xff\xff\x12\xe4\x49\x00\xca\x0d\x44\x0c\x2c\x31\x01\x00\x2f\x01\x34\x00\x64\x00\xbb"

    return packets

master_fd, slave_fd = os.openpty()
symlink_force(os.ttyname(slave_fd), '/tmp/fakexbee')


packets = valid_packets()
while True:
    for key in packets:
        os.write(master_fd, bytes(packets[key]))
        time.sleep(1)

Contributing authors:

Created by kluong on 2021/09/20 23:58.