user:kluong:improvement_-_simplified_data_pipeline [Smart Campus Energy Lab Wiki]

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Improvement - simplified data pipeline ======

**Note: this page is in draft.**


===== Background/Motivation =====

There's a bit of complexity around how handle sensor data. Here are two things:

  * Every generation has it's own packet format
  * There's a different table for each generation

At the time, this was a pretty arbitrary decision that I personally made, thinking that this would make it easier to manage multiple generations of sensor boxes. In reality, I think it makes things way more complicated than they need to. The different hardware generations are all pretty much collecting the same data, so there's no real reason to have different schemas for all the different hardware versions. Some of the hardware generations have sensors have that others do not (like RTC for example). But it might be enough to just leave those things empty. 

One limitation with XBees + ZigBee is that packets can only carry a payload of 72 bytes.

There is the added benefit that 296 teams should also be able to easily construct these packets and send to the gateway as well.

Here's what I think should change:

  * Firmware packets are standard across all hardware types
  * The database is consolidated to one table that stores all sensor data

More thoughts here

  * https://wiki.scel-hawaii.org/doku.php?id=user:kluong:thinking_about_a_generic_schema


===== Implementation =====

  * Go through tasks, put it onto a google doc for collaboration / documentation
    * make sure the initial background and motivation are documented as well
  * Come up with a new simplified schema
  * Create an SQL file (or migration script) to add the new table
  * Test this new table locally
  * Migrate the gateway to store all packets into this unified table
  * Deploy the gateway to staging to test it
  * Deploy the gateway to production
  * Change the firmware to use the new format
  * Deploy firmware on boxes