Technical deep-dives on architecture patterns, event sourcing, and building reliable systems.
Materialized View that can be Incrementally Updated every x hours can replace the corresponding ELT pipelines. With Transparent Query Rewrite, such MVs can play an important role in AI/BI.
AI/BI-powered Analytics have 3 assumptions: [1] dimensional data model [2] transformation pipeline [3] data integrity & quality. However, the realistic gaps still exist in all these areas which lead to lower-than-expected accuracy of AI-for-analytics. This article explains the WHY and HOW to `shift left` and mitigate this from the ground truth level by investing into the true data engineering paradiams.
Do we need to build k8s4ai to replace k8s or build stateful patterns on top of k8s? What should happen in the next 2~3 years.
Understand the pitfall of I/O saturation symptom caused by insanely simple queries in ClickHouse and StarRocks. Learn the best practice of physical data modeling and query shape for decent QPS analytical workloads.
Understand the 'heatmap' of Snowflake costs and how to focus on the most impactful areas (compute/warehouse allocation, highly-skewed workload, SQL craftsmanship, migration to other engines, operational intelligence) with effective mitigations.
A practical guide to Command Query Responsibility Segregation (CQRS) and how it complements event sourcing for scalable systems.
Practical patterns for building distributed systems that gracefully handle failures, from retries to circuit breakers to saga patterns.
Learn the fundamentals of event sourcing architecture and understand when and why you should consider it for your applications.