Book Notes By Chris

The Apache Flink project is well known for its low-latency stream processing capabilities. But what do we know beyond the documentation? There is much documentation dedicated to deploying and operating clusters and building applications. But today there are services like AWS Kinesis Data Analytics that hide significant complexity allowing you to quickly reach the limits of the framework. I am reading "Stream Processing with Apache Flink" to give myself a better idea of what it means to really own a Flink application in production.

Introduction

The opening chapters provide great context for why one might use Flink. While they do not explicitly enumerate and compare alternate streaming frameworks, the authors do a great job highlighting some key concerns of stream processing, namely dealing with state, scale, and performance. To the uninitiated, these might seem unnecessary but for anyone who's build a data pipeline with just AWS Lambda and owned it for a few months or years, you know that these concerns should not be afterthoughts.

Use Cases

The authors identify three main use cases: event-driven applications, data pipelines, and streaming analytics. Regarding event-driven applications, the authors compare using Flink to an alternate architecture using microservices. In a microservices architecture, you may have several representations of state which can become inconsistent or multiple methods of inter-process communication (messaging, API's, gRPC vs. REST, etc.) that can get in the way of correctness and expressiveness. There are also enormous performance gains to having a framework that safely manages and protects your application state while keeping massive amounts of event data in-memory rather than transferring over multiple web API's and database connections on every record.

Alternatives

I want to talk about alternatives more explicitly. The book calls out a specific Lambda architecture, batch processing, and prior generations of stream processing as alternatives to using Flink. Documentation on Apache Flink has similar callouts. However, I think many data engineers today will find that these comparisons are not very helpful.

Personally, I have had great success pairing AWS Kinesis with AWS Lambda functions for near-realtime event processing and there are equivalents in lots of places. So let me identify a few major differences between that architecture and stream processing with Flink that will hopefully go beyond the obvious.

What Flink Provides Compared to AWS Kinesis + Lambda

1. Windowing

2. Using multiple streams (which necessitates windowing)

3. SQL queries on stream tables

4. Different scaling mechanisms

5. Automatic Optimization - Flink can dedicate all available compute to working operations while I/O-bound tasks wait. Lambda will be DIY and most people will just let them block or introduce multiple lambdas with additional streams, costs, and complexity.

6. Packaging and deployment

7. Open source flexibility

8. Options for message ordering guarantees (Lambda is DIY)

9. Choice of event-time versus processing-time semantics (Lambda is just processing time)

10. Knowledge is centered around stream processing while Kinesis/Lambda community, practices, books, and resources are tend to center around each service separately

What about checkpointing?

Actually, as a colleague of mine pointed out, Kinesis manages read positions for its consumers. This actually solves use cases around failures or updates within the Lambda consumers.