Microservices + Event Driven Architecture in tech. company | Sponsored talk from KB

Original name

Microservices + Event Driven Architecture v tech. firmě │ Partnerská přednáška KB

Author(s)

Jiří Pinkas

Length

55:57

Date

14-11-2023

Language

Czech 🇨🇿

Rating

⭐⭐⭐⭐☆

  • ✅ Nicely managed talk, the speaker can naturally talk in the informative form.

  • ⛔ Overly confident guy telling presenting a golden standard that everyone must follow.

  • ⛔ I cannot agree with the statement that "Microservices and event-driven architecture should be everywhere".

  • ⛔ The latter part of the session was rather focused on generic description of patterns instead of talking about the way KB was designed.

""The company must be mature enough "to enjoy" the microservices.""

""We are blind and deaf without monitoring and alerting.""

New Digital Bank

Greenfield digital bank from scratch. 1000 developers and 200 teams.

Technological stack:

  • Spring Boot 2 (migrating to Spring Boot 3)

  • Java 17 / Kotlin

  • Event-driven architecture based on Apache Kafka (along with JMS in some parts).

  • OpenAPI defines contracts of REST communication between client and server.

  • Git-ops approach, Kubernetes, Helm

  • CI/CD via Jenkins pipelines

  • Centralized logging, monitoring and tracing

Microservices prerequisites

The company must be mature enough "to enjoy" the microservices.

  • CI/CD pipeline

  • Containerization (ideally with Kubernetes)

  • Observability (central logging, monitoring, tracing)

  • Robust testing strategy

  • Fault-tolerant architecture

  • Mass automation

Argo CD and GitOps is a standard that can be fully automated: Code is pushed to GIT, CI/CD pipeline takes the repository, builds an image and pushes to the Docker repository. Argo synchronizes configuration into Kubernetes that pulls the Docker images and runs in orchestrated containers.

CI/CD

Basic process: Checkout → Build application → Run tests → Run Sonar (quality gate checks: code coverage, security rating) → Run OWASP dependency check → Push to Docker registry

The CI/CD should focus on code quality and vulnerabilities as they can be everywhere (code, dependencies…​) which is nowadays a huge threat:

  • Sberbank had Docker images exposed online.

  • Russian Raiffeisenbank was hacked every day over many months until fix.

The threats come from the east and the bank regulator is about to make the banking application security stricter. In practice, simple Sonar and Dependency checker can find out most of the problems including security rating → the quality gateway should have solid security rating threshold and the rules should be centralized across the company. Spring Boot 2.7 is about to end the security patches support, so it is recommended to migrate to Spring Boot 3 as soon as possible.

A huge problem with dependency vulnerabilities is that they are discovered and updated every day, so the checking process should be continual. Most of the vulnerabilities come from the 3rd party or legacy dependencies:

  • We need to upgrade as often as possible.

  • The proprietary and legacy in-house applications must be checked and maintained.

All above is valid not only for microservice architecture.

Observability

All logs lead to Elastic

Centralized logging is the must in the microservice architecture (and all the rest) as we don’t want to go through the logs of the particular replica in case of exception.

Basic set-up (there can be middleware inserted): Filebeat/Fluentbeat → Logstash (optionally) → Elastic → Kibana → 👤

Monitoring and alerting

Without it, we are blind and deaf. We can see particular errors in the logs in the methods, but it is needed to see a "whole picture" first to understand what and why it happened.

Basic set-up: Prometheus (scrapping endpoints and metrics, alert manager) → Time-series DB → Grafana → 👤

Tracing

A mistake can be anywhere between microservices or combined across the microservices: Distributed tracing a must.

Basic set-up: Low-level tracing too (Jaeger tracing is a perfect tool, or Zipkin) + Business monitoring (Kiali can construct a diagram of microservices communication) → 👤

All above are minimum prerequisites for a mature architecture.

Event-driven architecture

Same as REST, it is not suitable for every use-case and has cons/pros.

REST comparison

Problems:

  • One of endpoints don’t work → Timeouts, retry policy, circuit breaker

  • In case of wrongly configured timeouts, the thread pool for connections (Tomcat has 200 by default) gets depleted as all the threads would wait.

Solution: Event-driven architecture as the messages is either processed or not: It can wait until the service is available as long as it needs, though the overhead is managing another service like Kafka or MQ.

Patterns

CQRS (Command Query Responsibility Segregation)

Problem: The application is responsible for managing insert/update/delete/select of an entity which makes the application complex that increases and eventually turns into a monolith → The unit of scalability becomes the application itself which is not ideal.

Solution: CQRS: Separates 2 responsibilities, commands (insert/update/delete) from queries (selects) into two applications → The unit of scalability is smaller.

CQRS + Read Model

Various clients can use various read methods, for example PostgreSQL serves as the master datasource and for the sake of optimized full-text search which PostgreSQL is not suitable for, the data are replicated to Elastic that serves perfectly for such a purpose.

ab-partial-all-1

Event Sourcing

If we need a log of changes over an entity, event sourcing persists the entity state as a sequence of change events. In order to work efficiently with such an entity, we need a current snapshot of its state received with the CQRS pattern, otherwise we need to run through all the events and apply them to get the wanted state.

ab-partial-all-1

Transactional Outbox

Problem: If we need to persist a record to the relation database, and send an event into Kafka, the operation is not atomic.

Wrong approach
ab-partial-all-1

In the past, there were popular 2-phase commits or distributed transactions, but they are a big nope as they are not scalable and cumbersome to implement, debug and maintain.

Solution: Transactional Outbox pattern.

  • Solution via polling (not ideal due to polling overhead):

    ab-partial-all-1

  • Solution using CDC (Change Data Capture) for example Debezium that can stream database changes:

    ab-partial-all-1

SAGA: Choreography-based SAGA

This pattern is a replacement for 2-phase commit or distributed transactions. For example: An order is in the PENDING state and the "Order Created" event is sent. The Customer Service receives the event ands the result (paid / not paid). Finally, the Order Service changes the order state to COMPLETED.

Eventual consistency does not assure immediate consistency but consistency once the cycle is completed (the items might wait in the queue for a while).

ab-partial-all-1

SAGA: Orchestration-based SAGA

In case of simple and low number of microservices, choreography-based SAGA solution is good, however in more complex system, orchestration-based SAGA is needed.

The implementation is based on message broker orchestrator that distributes messages.

ab-partial-all-1

API Gateway

  • Granularity of backend API is a bit different from what client needs: Different clients need different data.

  • We want to split backend APIs for the sake of flexibility.

ab-partial-all-1

BFF: Backend for frontend

It is a variation of API gateway pattern defining separate API gateways for each client.

ab-partial-all-1

Recommended resources