weatherbox:gateway:fake_and_bridged_mode [Smart Campus Energy Lab Wiki]

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Fake and Bridged Mode ======

 --- //[[luong97@hawaii.edu|Kenny Luong]] 2020/04/19 23:34//

**Page is currently in progress, change is not yet in master.**

====== Summary ======

These two features will allow developers to test changes to the gateway without needing a full network setup with devices and XBees. 

After talking to the current firmware team (who is responsible for the gateway at the moment), I've finally managed to get around to some improvements for testing - two different modes that don't require a full weatherbox to be operational for gateway development!

<code bash>
(env) kluong@kserver:~/control_tower/gateway/src$ python gateway_server.py fake
Setup
Setup XBee Device
Starting main Gateway Loop
</code>


<code bash>
(env) kluong@kserver:~/control_tower/gateway/src$ python gateway_server.py bridged /dev/ttyACM0
Setup
Setup XBee Device
Starting main Gateway Loop
</code>

**Fake mode will work on scelserver** without any additional modifications, so feel free to give it a try when you get a chance. 

These two modes allow for testing the gateway without having a working weatherbox. When used, these are how the modes work:

^ Mode          ^ Description                                                                                                                                                                                             ^
| Fake Mode     | This mode does not require any physical devices at all; it simulates a weatherbox network \\ by sending packets from multiple types of weatherboxes in an infinite loop.                                |
| Bridged Mode  | This mode allows you to hook up a physical device that emits TX frames onto uart lines. This could mean a full weatherbox, or just a microcontroller running a stubbed version of the normal firmware.  |



====== How it works ======

===== Normal Setup =====

This is what the normal pipeline looks like:

{{drawio>weatherbox:gateway:normal_mode}}

The microcontroller on the weatherbox side sends a [[https://www.digi.com/resources/documentation/Digidocs/90002002/Reference/r_frame_0x10.htm?tocpath=Frame%20descriptions%7C_____3 | transmit request]] to the remote xbee: 

The coordinator xbee (attached too the gateway), then sends a [[https://www.digi.com/resources/documentation/Digidocs/90002002/Reference/r_frame_0x10.htm?tocpath=Frame%20descriptions%7C_____3 | receive packet frame]] through uart: 

So, if this is how things normally work, we'd need some hardware to test the gateway to see that it's decoding packets correctly. We'd need:

  * A weatherbox
  * 2 XBees
  * A usb serial adapter to connect to the XBee on the hardware side.

This is problematic for a couple of reasons:

    * Setup is complicated and cumbersome 
    * Setup is expensive






===== Bridged Mode =====

I realized that if I could have some script running that could translate transmit request frames to receive frames on the other side we could connect a weatherbox (or microcontroller) directly to a gateway to test it:

{{drawio>weatherbox:gateway:bridged_mode}}

Any firmware that emits transmit frames onto a UART line connected to a computer will work; in my case, I used a modified version of the ga_stub_hb firmware build loaded onto a stock arduino to accomplish this. The only modification I made was to have the xbee driver write to the hardware serial lines instead of the software serial ports. This allows you to just use the arduino, instead of having to have a separate serial usb converter. 
===== Fake Mode =====

Fake mode takes bridged mode one step further and takes all hardware away completely:

{{drawio>weatherbox:gateway:fake_mode}}

The packets that are sent are hardcoded into the FakeNetwork class:

<code>
     self.packets = {}
     self.packets['heartbeat'] = "\x7e\x00\x16\x90\x00\x7d\x33\xa2\x00\x40\xe6\x4b\x5e\x03\xfd\x01\x00\x00\xff\xff\xf0\xfa\x23\x00\x2b\x02\xb2"
     self.packets['apple'] = "\x7e\x00\x22\x90\x00\x7d\x33\xa2\x00\x40\x9f\x27\xa7\x29\x6c\x01\x01\x00\xff\xff\x80\x6f\x69\x3d\x06\x0f\x71\x7d\x33\x5a\x8a\x01\x00\x76\x01\x22\x00\x6e\x09\x55"
     self.packets['cranberry'] = "\x7e\x00\x22\x90\x00\x7d\x33\xa2\x00\x41\x25\xe5\x88\x0c\x83\x01\x02\x00\xff\xff\x7c\xf3\x05\x00\xba\x0f\x5c\x08\x05\x00\x20\x73\x3b\x00\xdd\x8b\x01\x00\x7a"
     self.packets['dragonfruit'] = "\x7e\x00\x24\x90\x00\x7d\x33\xa2\x00\x40\xe6\x72\x7d\x5e\x30\x18\x01\x03\x00\xff\xff\x30\xc8\x07\x00\x6b\x0d\xf4\x00\x06\x00\x00\x00\xb6\x72\x37\x00\xfe\x8b\x01\x00\x00"
     self.packets['snapdragon'] = "\x7e\x00\x22\x90\x00\x7d\x33\xa2\x00\x40\xa3\x53\x7d\x5e\x20\x9c\x01\x04\x00\xff\xff\x12\xe4\x49\x00\xca\x0d\x44\x0c\x2c\x31\x01\x00\x2f\x01\x34\x00\x64\x00\xbb"
     self.packets['badPacket'] = "\x7e\x00\x16\x90\x00\x7d\x33\xa2\x00\x40\xe6\x4b\x5e\x03\xfd\x01\x00\x59\x90\x95\x95\x94\x88\x48\x48\x49\xe2"
</code>

These packets were taken a while back by grabbing receive frames from an xbee.
====== Why? ======

It's important to have a testing process that easy and can be done without the production environment, otherwise you'll always be trying to test in production!


====== Further Improvements ======


^ Item                                                                                              ^ Description                                                                                                                                       ^
| Automated testing: t                                                                              | This is important, because as the software gets bigger it gets harder and harder to test all of the cases manually by running the whole program.  |
| Unit testing                                                                                      |                                                                                                                                                   |
| End-to-end testing                                                                                |                                                                                                                                                   |
| Packaging story needs to be better; hard to run application with virtualenv                       |                                                                                                                                                   |
| Fake mode with C code - this might be useful for creating new schemas                             |                                                                                                                                                   |
| Deployment needs to be better\\     * systemd\\     * staging environment?                        |                                                                                                                                                   |
| Command line args could use some cleanup                                                          |                                                                                                                                                   |
| Is there a way to de-couple the python xbee library from the underlying transport mechanism       |                                                                                                                                                   |
| We could possibly use a pty to "subscribe" via some mechanism for debugging and testing purposes  |                                                                                                                                                   |
| REST api to get status?                                                                           |                                                                                                                                                   |
| Adding a new schema to the gateway                                                                |                                                                                                                                                   |
| Logs for the gateway could be more useful                                                         |                                                                                                                                                   |
| Migrate from python 2 to python 3                                                                 | Python 2 is going away                                                                                                                            |







====== Further Reading ======

The [[ https://www.digi.com/resources/documentation/Digidocs/90002002/Default.htm#Containers/cont_frame_descriptions.htm%3FTocPath%3DFrame%2520descriptions%7C_____0 | frame descriptions ]] listed on the digi website are actually really helpful.