Technical deep-dives on architecture patterns, event sourcing, and building reliable systems.
Why data type conversion is a real problem the moment data crosses a system boundary — a cross-engine type matrix covering ClickHouse, Arrow, Parquet, Vortex, Spark, Flink, Iceberg, Delta, Hudi, Paimon, Protobuf, Avro, Thrift, Postgres, Snowflake, StarRocks, DuckDB, and Trino.
The Clock Is Not a Dependency: Orchestration for the Agentic Data Era. Airflow 3.x has finally added more senors to incorporate the pattern of triggers. Let's evaluate the practical trend which the data-intensive workflows and jobs need to orchestracted with (some level of) knowledge: data content (esp. watermark, completeness/uniquness, quality, MV status), compute resource, SLA defined by downstreams, failure clues/metadata. How to fuse dependencies across Airflow, Temporal, Lakeflow, BPM, N8N? Or pushing all kinds of workflow into Airflow?
Materialized View that can be Incrementally Updated every x hours can replace the corresponding ELT pipelines. With Transparent Query Rewrite, such MVs can play an important role in AI/BI.
AI/BI-powered Analytics have 3 assumptions: [1] dimensional data model [2] transformation pipeline [3] data integrity & quality. However, the realistic gaps still exist in all these areas which lead to lower-than-expected accuracy of AI-for-analytics. This article explains the WHY and HOW to `shift left` and mitigate this from the ground truth level by investing into the true data engineering paradiams.
Do we need to build k8s4ai to replace k8s or build stateful patterns on top of k8s? What should happen in the next 2~3 years.
Understand the pitfall of I/O saturation symptom caused by insanely simple queries in ClickHouse and StarRocks. Learn the best practice of physical data modeling and query shape for decent QPS analytical workloads.
Understand the 'heatmap' of Snowflake costs and how to focus on the most impactful areas (compute/warehouse allocation, highly-skewed workload, SQL craftsmanship, migration to other engines, operational intelligence) with effective mitigations.
A practical guide to Command Query Responsibility Segregation (CQRS) and how it complements event sourcing for scalable systems.
Practical patterns for building distributed systems that gracefully handle failures, from retries to circuit breakers to saga patterns.
Learn the fundamentals of event sourcing architecture and understand when and why you should consider it for your applications.