Overview

# Overview

# A New Choice for Stream Computing

In traditional stream data processing scenarios, developers usually face two extreme choices:

Time-series Database Solutions: Such as InfluxDB, TimescaleDB, etc., while having strong storage capabilities, their real-time computing capabilities are limited and complex aggregation operations have poor performance.
Distributed Framework Solutions: Such as Apache Flink, Apache Storm, etc., while powerful, they have complex deployment, huge resource consumption, and are not suitable for edge environments.

StreamSQL provides a third choice: a lightweight stream processing engine designed specifically for edge computing, capable of real-time aggregation operations on massive data in resource-constrained environments, filling the gap between lightweight and high performance.

# 🌐 Typical Application Scenarios

Edge-Cloud Collaborative Architecture: In IoT and Industry 4.0 scenarios, a large number of sensors and devices generate massive amounts of raw data. Direct upload to the cloud would cause:

Huge network bandwidth pressure
High cloud computing costs
Real-time requirements cannot be met

StreamSQL's Solution:

Massive raw data → Edge StreamSQL processing → Aggregated results → Cloud storage/analysis
   (TB level)        (real-time filtering/aggregation)      (MB level)      (long-term storage)

1
2

Specific Scenario Examples:

Smart Factory: 1000 devices generate 100,000 data points per second, edge real-time aggregation to 100 key indicators per minute
Smart City: Environmental data from thousands of sensors, edge preprocessing only uploads abnormal events and statistical summaries
Connected Vehicles: Fleet real-time data aggregated at edge gateway, only sending driving behavior analysis and abnormal alerts to the cloud

# What is StreamSQL

StreamSQL is a lightweight stream processing engine designed specifically for IoT edge scenarios. It uses familiar SQL syntax to process unbounded data streams, allowing developers to implement complex real-time data analysis with simple SQL statements. It can also be integrated as a RuleGo (opens new window) component into the RuleGo ecosystem, leveraging its rich input/output and data processing component library and rule engine capabilities.

# Core Architecture

# Core Components

SQL Parser (rsql): Parses standard SQL statements, generates abstract syntax tree (AST), configures stream processing pipeline
Stream Processor (stream): Real-time data processing, filtering and transformation, aggregation calculations
Window Manager (window): Time windows: tumbling, sliding, session, count windows, window lifecycle management
Function System (functions): 60+ built-in functions, pluggable custom functions, 8 function type supports
Expression Engine (expr): Arithmetic expression calculation, logical expression processing, field reference resolution

# Design Philosophy

🪶 Lightweight First: Pure Go implementation, zero external dependencies, memory efficient, second-level startup. Designed specifically for edge devices and resource-constrained environments.

📝 SQL First: Uses standard SQL syntax, declarative programming, intuitive business logic expression, extremely low learning cost.

🔧 Extension First: Plugin architecture, supports hot-pluggable functions, deep integration with RuleGo, leveraging its rich input/output component ecosystem, and supports hot updates.

# Technical Features

🚀 High Performance: Stream processing, in-memory computing, concurrent design, fully utilizing multi-core performance.
🔧 Easy to Use: Uses standard SQL syntax, no need to learn complex stream processing frameworks.
🛡️ Pure In-Memory Computing: No disk storage, all calculations are performed in memory, extremely fast processing speed.
🔍 Analytic Functions & Change Detection: Cross-event state functions like lag / had_changed / changed_col(s) / latest / acc_*, with OVER (PARTITION BY/WHEN) support — purpose-built for CDC change detection, context backtracking, and lifecycle accumulation. See Analytic Functions.
🧩 Complex Event Processing (CEP): MATCH_RECOGNIZE (SQL:2016) recognizes event-sequence patterns — consecutive threshold crossings, rise-then-drop, start→run→stop workflows, out-of-order events, time-windowed sequences. Flink-SQL-aligned (PATTERN/DEFINE/MEASURES/navigation/aggregates/SUBSET/FINAL/WITHIN all supported), with four guards bounding edge memory. See Pattern Recognition.
📦 Basic Library Design: Can be integrated as a basic library into other projects, only depends on Go standard library and expr expression engine, no third-party dependencies.
🔄 RuleGo Integration: Can be deeply integrated with RuleGo, leveraging its rich component ecosystem and rule engine capabilities.
- Data Input: MQTT, HTTP, WebSocket, TCP/UDP, message queues, files, etc.
- Data Output: databases, message queues, REST API, email, SMS, etc.
- Data Processing: filtering, transformation, routing, aggregation, etc. 60+ components
- Extension Capability: custom components, hot-pluggable, dynamic configuration
- Low Code: No need to write code, just configure rule chains to implement complex data processing logic.

# Comparison with Other Solutions

# 📊 Comprehensive Comparison Analysis

Feature	StreamSQL	Apache Flink	Apache Storm	ksqlDB	eKuiper	Time-series Database
Deployment Complexity	Extremely Simple	Complex	Medium	Medium	Simple	Medium
Resource Usage	Extremely Low (10MB)	High (GB level)	Medium	High (GB level)	Extremely Low (10MB)	Medium
Embeddability	✅ Embeddable/Standalone	❌ Heavy Dependencies/Standalone	⚠️ Medium Dependencies/Standalone	❌ Heavy Dependencies/Standalone	⚠️ Few Dependencies/Standalone	⚠️ Medium Dependencies/Standalone
SQL Support	✅ Complete	✅ Complete	❌ None	✅ Complete	✅ Complete	⚠️ Limited
Complex Event Processing (CEP)	✅ MATCH_RECOGNIZE	✅ Support	❌ None	❌ None	❌ None	❌ None
Edge Deployment	✅ Excellent	❌ Not Suitable	⚠️ Barely	❌ Not Suitable	✅ Excellent	⚠️ Barely
Real-time Aggregation	✅ Support	✅ Powerful	⚠️ Limited	✅ Support	✅ Support	❌ Poor Performance
Learning Cost	Extremely Low	High	High	Medium	Low	Low
Nested Fields	✅ Native Support	⚠️ Needs Configuration	❌ Not Supported	⚠️ Limited	✅ Support	⚠️ Limited
Cluster Scaling	❌ Single Machine	✅ Horizontal Scaling	✅ Horizontal Scaling	✅ Horizontal Scaling	❌ Single Machine	✅ Support
Development Language	Go	Java/Scala	Clojure	Java	Go	Multiple

# Performance Benchmark

StreamSQL is built for the edge. Single-instance throughput on a 128MB gateway (one rule, single core, x86 measured, v1.0.3):

Rule	ns/op	allocs/op	Throughput
Filter	522	6	~1.92M msg/s
Transform	1359	12	~740K msg/s
Analytic + partition (lag PARTITION BY)	2095	18	~480K msg/s

Memory is not the bottleneck: 128MB holds 100k+ devices of partition state (measured: 50k partitions use only 4.5MB heap).
CPU throughput is: one rule saturating one core is the optimal usage; multi-core scales by running parallel independent instances (raise GOGC for near-linear scaling).
Bounded & stable: no goroutine leaks, heap does not grow with load or partition count.
Typical capacity (A53 low-end 128MB gateway): ~250k–320k devices at 1 msg/s per device; ~2500–3200 devices at 100 msg/s per device (filter rule).

ARM gateway figures are estimates derived from x86 (A53 low-end filter ~240k–320k msg/s); measure on your target SoC before production. Benchmark code: BenchmarkGateway_* in test/e2e/stress_test.go.

# Applicable Scenarios

# ✅ Suitable Scenarios

Edge Computing: IoT devices, industrial gateways, edge servers
Real-time Analysis: device monitoring, anomaly detection, real-time alerts
Quick Deployment: single machine deployment, containerization, embedded applications
Prototype Validation: quickly validate stream processing logic and business rules
RuleGo Enhancement: add SQL stream processing capabilities to RuleGo rule chains

# ❌ Unsuitable Scenarios

Large-scale Clustering: distributed systems requiring horizontal scaling
Complex State: applications requiring persistent state management
High Concurrent Writes: data volume exceeding single machine processing capability
Transaction Processing: complex transaction scenarios requiring ACID properties

# Core Application Scenarios

# 🏭 Industrial IoT - Massive Device Data Real-time Processing

Device Monitoring and Predictive Maintenance: Real-time monitoring of device status, anomaly detection, predictive maintenance

Scenario Description: A manufacturing factory has 1000+ devices, generating 10,000+ data points per second, requiring real-time monitoring and anomaly detection

-- Device anomaly detection example - processing million-level TPS data streams
SELECT device_id,
       COUNT(*) as data_points,
       AVG(temperature) as avg_temp,
       MAX(temperature) as max_temp,
       STDDEV(vibration) as vibration_variance,
       CASE WHEN AVG(temperature) > 80 OR STDDEV(vibration) > 5 
            THEN 'ALERT' ELSE 'NORMAL' END as status
FROM stream
GROUP BY device_id, TumblingWindow('5m')
HAVING status = 'ALERT'

1
2
3
4
5
6
7
8
9
10
11

# 🏙️ Smart City

Traffic Flow Optimization: Real-time analysis of traffic flow, congestion detection, traffic light optimization Environmental Monitoring: Air quality monitoring, pollution warnings, environmental data analysis

# 🚗 Connected Vehicles - Large-scale Fleet Real-time Monitoring

Vehicle Status Monitoring: Real-time monitoring of vehicle operation status, fault diagnosis, driving behavior analysis

Scenario Description: A logistics company has 5000+ vehicles, each vehicle reports 20+ data points per second, totaling 100,000+ events/sec

-- Vehicle anomaly detection - processing million-level data streams
SELECT vehicle_id,
       COUNT(*) as total_events,
       AVG(engine_temp) as avg_engine_temp,
       MAX(engine_temp) as max_engine_temp,
       AVG(fuel_consumption) as avg_fuel,
       COUNT(CASE WHEN brake_pressure > 80 THEN 1 END) as hard_brake_count,
       COUNT(CASE WHEN speed > 120 THEN 1 END) as speeding_count
FROM stream
GROUP BY vehicle_id, TumblingWindow('10m')
HAVING avg_engine_temp > 90 OR hard_brake_count > 5 OR speeding_count > 3

1
2
3
4
5
6
7
8
9
10
11

Edge Deployment Advantages:

Multiple edge nodes distributed deployment, each node handles 1000+ vehicles
Local real-time processing, reduces network transmission pressure
Fault isolation, single point failure does not affect overall system

# 💰 Financial Risk Control

Real-time Risk Control: Transaction behavior analysis, fraud detection, risk scoring

-- Abnormal transaction detection
SELECT user_id,
       COUNT(*) as transaction_count,
       SUM(amount) as total_amount
FROM stream
WHERE amount > 0
GROUP BY user_id, TumblingWindow('1h')
HAVING transaction_count > 20 OR total_amount > 10000

1
2
3
4
5
6
7
8

# 🧩 Pattern Recognition & Change Detection — Unique Capabilities

The three scenarios above are all "time-windowed statistics" that windowed aggregation handles. But two kinds of requirements aggregation cannot do are StreamSQL's signature edge over other lightweight engines:

① Consecutive-threshold debounced alerting (CEP) — a single crossing may be sensor jitter or a transient spike; N consecutive crossings signal a real anomaly:

-- Alert only after temperature exceeds 80°C three times in a row (debounce), within 10 minutes
SELECT * FROM stream
MATCH_RECOGNIZE (
    PARTITION BY device_id
    ORDER BY ts
    MEASURES MATCH_NUMBER() AS mn, LAST(A.temp) AS peak
    ONE ROW PER MATCH
    PATTERN (A{3}) WITHIN '10m'
    DEFINE A AS temp > 80
)

1
2
3
4
5
6
7
8
9
10

② Start→Run→Stop workflow recognition (CEP) — recognize an ordered event sequence; it cares about event order, not statistics:

-- Recognize a complete workflow, emit each run's start/stop time and peak temperature
SELECT * FROM stream
MATCH_RECOGNIZE (
    PARTITION BY device_id ORDER BY ts
    MEASURES FIRST(A.ts) AS start_ts, LAST(C.ts) AS stop_ts, MAX(B.temp) AS run_peak
    ONE ROW PER MATCH
    PATTERN (A B+ C) WITHIN '2h'
    DEFINE A AS status='START', B AS status='RUNNING', C AS status='STOP'
)

1
2
3
4
5
6
7
8
9

③ State change detection (analytic / CDC) — emit only when a metric truly changes, with the previous value, filtering redundant repeats:

-- Report only on device state change (CDC)
SELECT device_id, status, lag(status) AS prev_status
FROM stream
WHERE had_changed(true, status)

1
2
3
4

Which to use: compare adjacent events (did it change?) → analytic; ordered/sequence patterns (B after A, N times in a row) → CEP; time-windowed stats (per-minute average) → windowed aggregation + HAVING. See Pattern Recognition and Analytic Functions.

# 🔥 Core Advantages

Say Goodbye to Heavy Frameworks: No need to deploy complex Flink clusters, single machine can handle million-level TPS
Beyond Time-series Databases: Not only storage, but also real-time computing, millisecond-level response
Edge Computing First Choice: 10MB memory usage, perfect for resource-constrained environments
Extremely High Development Efficiency: Standard SQL syntax, zero learning cost, minute-level onboarding
High Performance Processing: Single machine can reach million-level TPS, in-memory computing, millisecond-level response

# 📈 Applicable Scale

Small Projects: Quick prototype validation, MVP development
Medium Applications: Single machine million-level TPS, meets most business needs
Edge Scenarios: IoT gateways, industrial controllers, in-vehicle systems
Hybrid Architecture: Combined with RuleGo, build complete edge computing solutions

StreamSQL = Real-time capability of time-series databases + Computing power of Flink + Minimal deployment and integration experience

# 🎈 Code Hosting

🐙 Gitee (opens new window) | 🐙 Github (opens new window) | 🐙 GitCode (opens new window)

# 🧸 Contributing

Contributions to improve StreamSQL are welcome — we look forward to your PRs!

Contribute code: Source at StreamSQL (opens new window), feel free to open Issues or Pull Requests.
Maintain docs: Docs at StreamSQL-Doc (opens new window), help with translation and revision is welcome.

# Next Steps

📚 Quick Start - 5-minute onboarding guide
🧩 Core Concepts - Understand the basics
🪟 Windows - Deep dive into window processing
🔍 Analytic Functions - Change detection/CDC, lag, cumulative statistics
🧩 Pattern Recognition (CEP) - MATCH_RECOGNIZE event-sequence pattern recognition
💻 Case Studies - See real-world examples
📖 API Reference - Complete API documentation

Edit this page on GitHub

Last Updated: 2026/07/14, 06:12:38

← Support this project Quick Start→