Modeling Temporal and Historical Data
1. Understanding Temporal Data Types
| Term | Meaning |
|---|---|
| Valid Time | When the fact is true in reality |
| Transaction Time | When stored in DB |
| Bitemporal | Both valid and transaction time tracked |
| SCD | Slowly changing dimension (type 1/2/3/4/6) |
| Application time | SQL:2011 PERIOD FOR app_time |
| System versioning | SQL:2011 SYSTEM_TIME (auto history) |
2. Implementing Bi-Temporal Tables
Example: Bi-Temporal Row
CREATE TABLE policy_price (
id BIGSERIAL,
policy_id BIGINT,
price NUMERIC(10,2),
valid_from DATE NOT NULL,
valid_to DATE NOT NULL,
recorded_at TIMESTAMPTZ DEFAULT NOW(),
superseded_at TIMESTAMPTZ,
PRIMARY KEY (id)
);
-- Query: what did we know on 2026-01-15 about price for valid 2026-03?
SELECT price FROM policy_price
WHERE policy_id = ? AND '2026-03-01' BETWEEN valid_from AND valid_to
AND recorded_at <= '2026-01-15'
AND (superseded_at IS NULL OR superseded_at > '2026-01-15');
3. Modeling Effective Dating
| Column | Use |
|---|---|
| effective_from / effective_to | Validity window |
| Open-ended | NULL or '9999-12-31' for current |
| Constraint | EXCLUSION on overlapping ranges per entity |
| Update | Close current row, insert new with new dates |
4. Designing Historical Tracking Tables
| Pattern | Detail |
|---|---|
| Shadow table | Mirror schema + op + changed_at |
| SQL:2011 SYSTEM_TIME | Auto-history (SQL Server, MariaDB, DB2) |
| CDC stream | Debezium → log/store |
| JSON diff | Store only changed fields |
5. Implementing Event Sourcing Schemas
| Concept | Detail |
|---|---|
| Event log | Append-only sequence of events |
| Schema | event_id, aggregate_id, version, type, payload, occurred_at |
| Projection | Derive current state by replaying events |
| Snapshot | Periodic checkpoint for faster replay |
| CQRS pairing | Write events; read projections |
6. Modeling Point-in-Time Queries
Example: AS OF Query (SQL:2011)
SELECT * FROM employees
FOR SYSTEM_TIME AS OF TIMESTAMP '2025-12-31 23:59:59';
| Approach | Detail |
|---|---|
| SQL:2011 AS OF | Supported in MariaDB, SQL Server, DB2, Oracle Flashback |
| Manual | WHERE effective_from <= t AND effective_to > t |
| Event replay | Apply events up to timestamp |
7. Designing Temporal Foreign Keys
| Approach | Detail |
|---|---|
| PERIOD FK (SQL:2011) | FOREIGN KEY ... PERIOD app_time REFERENCES ... |
| Validity overlap | Child period must be within parent period |
| Manual | Trigger or check constraint enforcing temporal containment |
8. Handling Temporal Data Updates
| Operation | Pattern |
|---|---|
| Update field | Close current row (set effective_to), insert new row |
| Correct past | Update existing row in place (mark recorded_at) |
| Delete | Set effective_to = NOW(); never physical delete |
| Bulk reload | Insert new generation, switch pointer |
9. Implementing Slowly Changing Dimensions
| SCD Type | Behavior |
|---|---|
| Type 0 | No change tracking |
| Type 1 | Overwrite (no history) |
| Type 2 | New row per change with effective dates |
| Type 3 | Previous value column (last N changes) |
| Type 4 | Current table + history table |
| Type 6 (hybrid) | Type 1 + 2 + 3 combined |
10. Optimizing Temporal Query Performance
| Tip | Detail |
|---|---|
| Index temporal columns | (entity_id, effective_to DESC) |
| Range types + GiST (PG) | Fast overlap queries |
| Partial index for current | WHERE effective_to IS NULL |
| Partition history by period | Drop old data fast |
| Separate current vs history | Hot table stays small |