Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
data_infrastructure:iniital_proposal [2016/06/21 02:50] kluong [Compute/Data Backend] |
data_infrastructure:iniital_proposal [2021/09/19 21:59] (current) |
||
|---|---|---|---|
| Line 6: | Line 6: | ||
| The purpose of the data infrastructure is to create a platform that can be used to easily collect and store sensor data from any source. This platform will eventually fully support the efforts of the forecasting teach, which given good access to data will be able to do their own independent research. | The purpose of the data infrastructure is to create a platform that can be used to easily collect and store sensor data from any source. This platform will eventually fully support the efforts of the forecasting teach, which given good access to data will be able to do their own independent research. | ||
| + | |||
| + | https://www.draw.io/#G0Bxowpw1NF2d3YmRaRjBHWTJTckE | ||
| ====== Outline ====== | ====== Outline ====== | ||
| Line 13: | Line 15: | ||
| IV. Technical Modules \\ | IV. Technical Modules \\ | ||
| - | ====== Motivation ====== | + | ====== Previous Work ====== |
| - | + | ||
| - | Currently, the core project in the smart campus energy lab involves collecting data. | + | |
| - | Issues with the previous project: | + | **Issues with the previous project** |
| * Documentation was poor | * Documentation was poor | ||
| Line 25: | Line 25: | ||
| * Limited to one data type | * Limited to one data type | ||
| + | ====== Motivation and Summary ====== | ||
| - | ====== Goals ====== | + | It is currently very difficult to reliably gather time series data from embedded sensor devices. |
| + | This project aims to provide the software infrastructure to reliably collect data, add new | ||
| + | sensors, extend new sensor types and analyze such data. | ||
| - | * Highly available | ||
| - | * Easy to interface with | ||
| - | * Easy to contribute to | ||
| - | * Communicate development well | ||
| + | ====== Specifications ====== | ||
| + | High Level Categories: | ||
| + | |||
| + | * Availability | ||
| + | * Interfaces | ||
| + | * Libraries | ||
| + | * Graphing | ||
| + | * Contributions | ||
| + | * Documentation | ||
| + | * Extend-ability | ||
| + | * Logging | ||
| + | * Verification | ||
| + | * Validation | ||
| + | |||
| + | |||
| + | Misc: | ||
| + | |||
| + | * Outside Users should be able to easily view nodes publicly | ||
| + | * Each node deployment should be able to be tracked | ||
| + | * Lab users should be able to download datasets using any scripting language | ||
| + | * We should be able to validate the data that is collected | ||
| + | * We should be able to scan if a sensor is down or not | ||
| + | |||
| + | |||
| + | Okay this is really hard. | ||
| ====== Technical Modules ====== | ====== Technical Modules ====== | ||
| Here is a block diagram: | Here is a block diagram: | ||
| - | {{ :data_infrastructure:sensor_infrastructure_proposal.png?direct&300 |}} | + | {{ :data_infrastructure:sensor_infrastructure_proposal_2.png?direct&300 |}} |
| ===== High Level Blocks ===== | ===== High Level Blocks ===== | ||
| Line 45: | Line 69: | ||
| * **Client** - Primary interface into the data infrastructure - sensors with transport layers such as ZigBee will dump their data to these clients. | * **Client** - Primary interface into the data infrastructure - sensors with transport layers such as ZigBee will dump their data to these clients. | ||
| * **Messaging Bus/Gateway** - Monitors all of the clients and makes sure that they are authorized to send data. Rejects invalid clients. | * **Messaging Bus/Gateway** - Monitors all of the clients and makes sure that they are authorized to send data. Rejects invalid clients. | ||
| - | * **Compute / Data Backend** - Responsible for processing packets as they come in and putting them into the database. | + | * ** Data Backend** - Contains all of the logic necessary to store and process data. Contains a publicly accessible API that can be used to build client applications. |
| + | * ** Compute Backend** - Able to run large compute jobs such as graphing or analysis scripts. Serves dataset results and graphs through a filesystem or the API. | ||
| ===== Client ===== | ===== Client ===== | ||
| Line 98: | Line 122: | ||
| * Sensor Platform | * Sensor Platform | ||
| * Data Sensor Platform | * Data Sensor Platform | ||
| - | |||