Rust Data Collector PoC — Design Prompt

# Rust Data Collector PoC — Prompt for AI Code Generation

## Objective

Build a Rust PoC to evaluate whether a Raspberry Pi can handle real-time ingestion, storage, analysis, and cloud sync of data from ~20 sensor channels at ~100Hz. The system acts as a **data historian** for a fatigue lab.

---

## Architecture

### Data Ingestion (Producer)
- Simulate 20 channels with numerical IDs (u16)
- **Fast channels** (~10): Forces/vibration — high variance, changes every sample
- **Slow channels** (~10): Temperature — low variance, slow drift
- Sampling rate: ~100Hz (10ms interval)
- Background thread pushes samples into a channel/queue

### Tiered Storage

| Tier | Medium | Capacity | Purpose |
|---|---|---|---|
| Hot | In-Memory (VecDeque circular buffer) | 1 minute | Real-time analysis |
| Cold | SQLite (rusqlite, WAL mode) | 24 hours | Persistence, survives power failure |
| Cloud | Remote endpoint | Unlimited | Off-site backup |

**Flush interval**: Write from memory to SQLite every **1 second** (batch insert in single transaction)

**Schema** (Home Assistant style — tall/long format):
```
Record { timestamp: u64, channel_id: u16, value: f32 }
samples table: timestamp, channel_id, value, is_uploaded (bool), retention_mode (raw/avg)
```

### Real-Time Analysis
- Calculate Simple Moving Average (SMA) per channel (window: 50 samples)
- Maintain alarm flag per channel — raised if raw value or SMA exceeds threshold
- Print console alert on threshold breach

### Data Lifecycle (Hourly Compression Job)
- **If alarm occurred** in the hour → retain all raw data
- **If no alarm** → replace raw samples with per-second averages (downsampling), delete raw data
- Reduction: ~3600x for quiet periods

### Cloud Synchronization
- Upload every **1 minute**
- Query DB for un-uploaded data → simulate HTTP POST
- **Store-and-forward**: If connection fails, keep data locally, retry next cycle
- Do not delete local data until upload confirmed

---

## Technical Constraints

- Use `std::sync::mpsc` or `crossbeam` for thread communication
- SQLite with **WAL mode** for crash safety on Pi
- Batch inserts (1 transaction per flush, not per row) — critical for SD card performance
- Memory-bounded: only 1 minute of raw data in RAM
- Must compile on `armv7` / `aarch64` (Raspberry Pi toolchain)

## Concurrency Model

Separate threads/tasks for:
1. **Acquisition** — generates samples, pushes to queue
2. **Analysis/Flushing** — computes SMA, checks alarms, flushes to SQLite
3. **Maintenance/Compression** — hourly rollup job
4. **Upload** — cloud sync with retry logic

## Deliverables

- Full Rust project (`main.rs`, `Cargo.toml`)
- Minimal dependencies: `rand`, `chrono`, `rusqlite`, optionally `crossbeam`
- Code comments on performance bottlenecks specific to Raspberry Pi

## Success Criteria

- Runs for simulated 1 minute without crash
- Prints simulated alarms when thresholds breached
- Outputs final summary: memory usage, total records stored
- Demonstrates compression (raw → averaged) for alarm-free windows
- Data survives simulated "power failure" (SQLite WAL)

---

## Feasibility Assessment

**Highly feasible on Raspberry Pi.** A Pi 4 (or even 3B+) can easily handle:
- 20 channels × 100Hz = 2,000 inserts/sec in memory
- SQLite batch writes of 2,000 rows/sec are well within SD card capabilities
- Bottleneck is disk I/O (SD cards), mitigated by 1-second batch flush
- Cloud upload is trivial at this data rate (~14 bytes × 2000/sec = ~28 KB/sec)

## Key Things to Verify in Generated Code

- `VecDeque` for circular buffer (not unbounded `Vec`)
- `fsync` / transaction commit in SQLite writer
- Batch inserts in single transaction (not individual inserts)
- Separate in-memory buffer from database connection
- No `String` allocations inside hot loop


id: d77ba108702f41bb988a9c3b38ff1275
parent_id: 6babd84074504358ba664996756055a4
created_time: 2026-06-08T08:14:53.163Z
updated_time: 2026-06-08T08:14:53.163Z
is_conflict: 0
latitude: 0.00000000
longitude: 0.00000000
altitude: 0.0000
author: 
source_url: 
is_todo: 0
todo_due: 0
todo_completed: 0
source: joplin-desktop
source_application: net.cozic.joplin-desktop
application_data: 
order: 1780906493163
user_created_time: 2026-06-08T08:14:53.163Z
user_updated_time: 2026-06-08T08:14:53.163Z
encryption_cipher_text: 
encryption_applied: 0
markup_language: 1
is_shared: 0
share_id: 
conflict_original_id: 
master_key_id: 
user_data: 
deleted_time: 0
type_: 1