What Is Real-Time Data?
Real-time data refers to information that becomes available almost instantaneously after the underlying event occurs. In practice, “real-time” exists on a spectrum:
True real-time: Sub-second latency (milliseconds)
- Stock trading systems
- Fraud detection
- Industrial process control
Near real-time: Seconds to minutes
- Live dashboards
- Operational monitoring
- Customer-facing applications
Frequent batch: Minutes to hours
- Business reporting
- Financial analytics
- Most business intelligence
Real-Time vs. Batch Processing
| Aspect | Real-Time | Batch |
|---|---|---|
| Latency | Seconds | Hours to days |
| Complexity | Higher | Lower |
| Cost | Higher | Lower |
| Use case | Operational | Analytical |
| Data volume | Event-by-event | Bulk processing |
When Do You Actually Need Real-Time?
Real-Time Is Essential
- Fraud detection (stop bad transactions)
- Operational alerts (system down)
- Customer-facing features (order status)
- Trading and pricing decisions
- Safety-critical systems
Near Real-Time Is Sufficient
- Executive dashboards (hourly refresh fine)
- Sales performance monitoring
- Inventory tracking
- Customer support metrics
Batch Is Appropriate
- Financial reporting
- Month-end close
- Historical analysis
- Compliance reporting
- Most management reports
The Real-Time Trade-Off
Real-time capability comes with costs:
Infrastructure complexity: Streaming architectures are more complex than batch
Higher costs: Processing event-by-event is more expensive
Data quality challenges: Less time to validate and clean
Skills requirements: Specialized engineering expertise needed
Maintenance burden: More components to monitor and maintain
Real-Time Architecture Components
Event Sources
Systems generating real-time events:
- Transaction systems
- IoT sensors
- User activity
- Application logs
Message Queues
Buffering and routing events:
- Apache Kafka
- Amazon Kinesis
- Google Pub/Sub
- Azure Event Hubs
Stream Processing
Analyzing events in flight:
- Apache Flink
- Apache Spark Streaming
- Amazon Kinesis Analytics
Real-Time Storage
Databases optimized for current state:
- Redis
- Apache Cassandra
- Time-series databases
Delivery
Getting data to consumers:
- WebSockets
- Server-sent events
- Push notifications
- Real-time dashboards
Real-Time Data Quality
Real-time data presents quality challenges:
Late-arriving data: Events may arrive out of order
Incomplete data: Not all information available immediately
Duplicates: Same event may be delivered multiple times
Corrections: Initial data may need adjustment
Best practice: Combine real-time operational view with batch-validated analytical view.
How Go Fig Handles Data Freshness
Go Fig provides appropriate freshness for finance use cases:
Configurable refresh: Choose hourly, daily, or on-demand updates
Near real-time dashboards: Key metrics update throughout the day
Scheduled workflows: Run pipelines on your required schedule
Event triggers: Start processing when new data arrives
Historical accuracy: Batch processing ensures data quality
For most financial analytics, near real-time (hourly or more frequent) provides the right balance of freshness and data quality.
Questions to Ask About Real-Time
Before investing in real-time capabilities:
-
What decision requires this speed? Can you act on data in seconds?
-
What’s the cost of delay? Is hourly data really a problem?
-
Is the data quality sufficient? Can you trust unvalidated data?
-
Do you have the skills? Can your team maintain streaming systems?
-
Is the ROI there? Do benefits justify the complexity?
Often, improving data accessibility with hourly updates delivers 90% of the value at 10% of the real-time cost.