Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
data_infrastructure:iniital_proposal [2016/06/21 02:42] kluong [Technical Modules] |
data_infrastructure:iniital_proposal [2016/06/24 03:54] kluong [Specifications] |
||
---|---|---|---|
Line 26: | Line 26: | ||
- | ====== Goals ====== | ||
- | * Highly available | ||
- | * Easy to interface with | ||
- | * Easy to contribute to | ||
- | * Communicate development well | ||
+ | ====== Specifications ====== | ||
+ | High Level Categories: | ||
+ | |||
+ | * Availability | ||
+ | * Interfaces | ||
+ | * Libraries | ||
+ | * Graphing | ||
+ | * Contributions | ||
+ | * Documentation | ||
+ | * Extend-ability | ||
+ | * Logging | ||
+ | * Verification | ||
+ | * Validation | ||
+ | |||
+ | |||
+ | Misc: | ||
+ | |||
+ | * Outside Users should be able to easily view nodes publicly | ||
+ | * Each node deployment should be able to be tracked | ||
+ | * Lab users should be able to download datasets using any scripting language | ||
+ | |||
+ | |||
+ | Okay this is really hard. | ||
====== Technical Modules ====== | ====== Technical Modules ====== | ||
Here is a block diagram: | Here is a block diagram: | ||
- | {{:data_infrastructure:sensor_infrastructure_proposal.png?direct&300|}} | + | {{ :data_infrastructure:sensor_infrastructure_proposal_2.png?direct&300 |}} |
+ | |||
+ | ===== High Level Blocks ===== | ||
+ | |||
+ | |||
+ | * **Client** - Primary interface into the data infrastructure - sensors with transport layers such as ZigBee will dump their data to these clients. | ||
+ | * **Messaging Bus/Gateway** - Monitors all of the clients and makes sure that they are authorized to send data. Rejects invalid clients. | ||
+ | * ** Data Backend** - Contains all of the logic necessary to store and process data. Contains a publicly accessible API that can be used to build client applications. | ||
+ | * ** Compute Backend** - Able to run large compute jobs such as graphing or analysis scripts. Serves dataset results and graphs through a filesystem or the API. | ||
+ | ===== Client ===== | ||
+ | |||
+ | |||
+ | * **Client Gateway** | ||
+ | * **Queue** | ||
+ | * **Client Connector** | ||
+ | |||
+ | ===== Messaging Bus ===== | ||
+ | |||
+ | |||
+ | * **Reverse Proxy/Balancer** | ||
+ | * **Gateway Database** | ||
+ | * **Gateway** | ||
+ | * **Gateway Queue** | ||
+ | |||
+ | ===== Compute/Data Backend ===== | ||
- | * **Client** - Used by the end user to collect data and | ||
- | * **Gateway** - Monitors all of the clients and makes sure that they are authorized to send data. Rejects invalid clients. | ||
* **Worker** - Processes data and makes sure that they | * **Worker** - Processes data and makes sure that they | ||
* **Node Manager** - Manages the registration and display of data nodes | * **Node Manager** - Manages the registration and display of data nodes | ||
Line 48: | Line 88: | ||
* **Core database** - Database that stores all of our data. Currently postgresql. | * **Core database** - Database that stores all of our data. Currently postgresql. | ||
* **Mirror database** - Public database that is RO for public users. Mirrored from the core database. | * **Mirror database** - Public database that is RO for public users. Mirrored from the core database. | ||
- | * **Master Queue** - Main queue that exists between the gateway and the worker scripts. This makes it possible to upgrade the gateway without losing any data in the network. | + | * **Gateway Queue** - Main queue that exists between the gateway and the worker scripts. This makes it possible to upgrade the gateway without losing any data in the network. |
- | * **Client Queues** - Queues that exist on the client systems to buffer against possible disconnects from the main server | + | |
====== System Validation ====== | ====== System Validation ====== | ||
Line 77: | Line 116: | ||
* Sensor Platform | * Sensor Platform | ||
* Data Sensor Platform | * Data Sensor Platform | ||
- |