Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
user:kluong:improvement_-_simplified_data_pipeline [2021/03/26 23:08] kluong |
user:kluong:improvement_-_simplified_data_pipeline [2021/10/07 14:04] (current) kluong |
||
|---|---|---|---|
| Line 2: | Line 2: | ||
| **Note: this page is in draft.** | **Note: this page is in draft.** | ||
| + | |||
| + | |||
| + | ===== Background/Motivation ===== | ||
| There's a bit of complexity around how handle sensor data. Here are two things: | There's a bit of complexity around how handle sensor data. Here are two things: | ||
| Line 8: | Line 11: | ||
| * There's a different table for each generation | * There's a different table for each generation | ||
| - | At the time, this was an slightly arbitrary decision that I personally made, thinking that this would make it easier to manage multiple generations of sensor boxes. In reality, I think it makes things way more complicated than they need to. Here's what I think should happen: | + | At the time, this was a pretty arbitrary decision that I personally made, thinking that this would make it easier to manage multiple generations of sensor boxes. In reality, I think it makes things way more complicated than they need to. The different hardware generations are all pretty much collecting the same data, so there's no real reason to have different schemas for all the different hardware versions. Some of the hardware generations have sensors have that others do not (like RTC for example). But it might be enough to just leave those things empty. |
| + | |||
| + | One limitation with XBees + ZigBee is that packets can only carry a payload of 72 bytes. | ||
| + | |||
| + | There is the added benefit that 296 teams should also be able to easily construct these packets and send to the gateway as well. | ||
| + | |||
| + | Here's what I think should change: | ||
| * Firmware packets are standard across all hardware types | * Firmware packets are standard across all hardware types | ||
| Line 15: | Line 24: | ||
| More thoughts here | More thoughts here | ||
| - | * https://wiki.scel-hawaii.org/doku.php?id=user:kluong:thinking_about_a_generic_schema | + | * https://wiki.scel-hawaii.org/doku.php?id=user:kluong:thinking_about_a_generic_schema |
| + | |||
| + | |||
| + | ===== Implementation ===== | ||
| + | |||
| + | * Go through tasks, put it onto a google doc for collaboration / documentation | ||
| + | * make sure the initial background and motivation are documented as well | ||
| + | * Come up with a new simplified schema | ||
| + | * Create an SQL file (or migration script) to add the new table | ||
| + | * Test this new table locally | ||
| + | * Migrate the gateway to store all packets into this unified table | ||
| + | * Deploy the gateway to staging to test it | ||
| + | * Deploy the gateway to production | ||
| + | * Change the firmware to use the new format | ||
| + | * Deploy firmware on boxes | ||
| + | |||
| + | |||