Differences

This shows you the differences between two versions of the page.

--- data_infrastructure:iniital_proposal [2016/06/21 02:44]
kluong [Technical Modules]
+++ data_infrastructure:iniital_proposal [2016/06/24 03:59]
kluong [Past]
@@ Line 13: / Line 13: @@
 IV. Technical Modules  \\
-====== Motivation ======
+====== Previous Work ======
-Currently, the core project in the smart campus energy lab involves collecting data.
 Issues with the previous project:
@@ Line 25: / Line 23: @@
   * Limited to one data type
+====== Motivation and Summary ======
-====== Goals ======
+It is currently very difficult to reliably gather time series data from embedded sensor devices.
+This project aims to provide the software infrastructure to reliably collect data, add new
+sensors, extend new sensor types and analyze such data.
-  * Highly available
-  * Easy to interface with
-  * Easy to contribute to
-  * Communicate development well
+====== Specifications ======
+High Level Categories:
+  * Availability
+  * Interfaces
+  * Libraries
+  * Graphing
+  * Contributions
+  * Documentation
+  * Extend-ability
+  * Logging
+  * Verification
+  * Validation
+Misc:
+  * Outside Users should be able to easily view nodes publicly
+  * Each node deployment should be able to be tracked
+  * Lab users should be able to download datasets using any scripting language
+Okay this is really hard.
 ====== Technical Modules ======
 Here is a block diagram:
-{{ :data_infrastructure:sensor_infrastructure_proposal.png?direct&300 |}}
+{{ :data_infrastructure:sensor_infrastructure_proposal_2.png?direct&300 |}}
+===== High Level Blocks =====
-  * **Client** - Used by the end user to collect data and
+  * **Client** - Primary interface into the data infrastructure - sensors with transport layers such as ZigBee will dump their data to these clients.
   * **Messaging Bus/Gateway** - Monitors all of the clients and makes sure that they are authorized to send data. Rejects invalid clients.
-  * **Compute / Data Backend** - Responsible for processing packets as they come in and putting them into the database.
+  * ** Data Backend** - Contains all of the logic necessary to store and process data. Contains a publicly accessible API that can be used to build client applications.
+  * ** Compute Backend** - Able to run large compute jobs such as graphing or analysis scripts. Serves dataset results and graphs through a filesystem or the API.
+===== Client =====
+  * **Client Gateway**
+  * **Queue**
+  * **Client Connector**
+===== Messaging Bus =====
+  * **Reverse Proxy/Balancer**
+  * **Gateway Database**
+  * **Gateway**
+  * **Gateway Queue**
+===== Compute/Data Backend =====
   * **Worker** - Processes data and makes sure that they
@@ Line 51: / Line 90: @@
   * **Core database** - Database that stores all of our data. Currently postgresql.
   * **Mirror database** - Public database that is RO for public users. Mirrored from the core database.
-  * **Master Queue** - Main queue that exists between the gateway and the worker scripts. This makes it possible to upgrade the gateway without losing any data in the network.
+  * **Gateway Queue** - Main queue that exists between the gateway and the worker scripts. This makes it possible to upgrade the gateway without losing any data in the network.
-  * **Client Queues** - Queues that exist on the client systems to buffer against possible disconnects from the main server
 ====== System Validation ======
@@ Line 80: / Line 118: @@
   * Sensor Platform
   * Data Sensor Platform