Real Time Performance Analysis of Data Processing Pipelines with Spring Cloud Data Flow, Micrometer

Realtime monitoring is a critical capability for building and operating distributed, dataintensive applications at scale. While Spring Cloud Data Flow (SCDF) helps you orchestrate streaming and batch data pipelines on multiple platforms, the Micrometer integration, on the other hand, provides the mechanism to collect multidimensional performance metrics. SCDF leverages Micrometer to provide a turnkey solution to collect, curate, and visualize key statistics of streaming and batch data pipelines. With practical code examples, we ll demonstrate how to consistently produce structured metrics (both for streaming and batch data pipelines) to Prometheus, and how to visualize them with dashboards such as Grafana. From an observability standpoint, we ll share the best practices to monitor the dataintensive applications running in Cloud Foundry and Kubernetes, which could be the basis for throughputbased elastic autoscaling.