Skip to content
Tutorials Center
My learning

Topics

Lesson 58 of 77 ยท DevOps, Tools and System Design

Microservices, Kafka and Kubernetes: Resilience and Messaging Questions

Answer microservices, Kafka and Kubernetes interview questions: service boundaries, circuit breakers, sagas, partitions and consumer groups, probes and rollouts.

"Would you split this into microservices?" is a favorite senior-level question because there is no single right answer; the interviewer is listening for trade-offs. The follow-ups are predictable: how do the services talk, what happens when one is down, how do you keep data consistent without a shared database, what does Kafka guarantee, and what does Kubernetes actually do when you deploy a new version.

The earlier system design lessons cover the basics (load balancers, caches, queues) and classic problems (rate limiters, consistent hashing, idempotency), and Docker, CI/CD and Kubernetes objects cover containers and the core Kubernetes resources. This lesson goes deeper on service boundaries, resilience, messaging with Kafka, and the Kubernetes control loop. A real cluster or broker cannot run on the site, so configuration files are read-only, and the runnable examples are small single-process Node stand-ins that model the logic. See also the Microservices, Kafka and Kubernetes tutorials.

Questions behind the questions

Boundaries Where do you cut the system?
Resilience What happens when a call fails?
Messaging How do events flow and replay?
Orchestration Who keeps it running?

Every answer should mention a cost.

1 / 2

What are microservices, and when should you not use them?

Short answer: A microservice architecture splits an application into small services that are independently deployable, each owning one business capability and its own data, and talking over the network. The benefits are independent releases and scaling, team autonomy and fault isolation. The costs are real: network failures and latency, distributed data consistency, harder debugging and testing, and much more operational work (deployment, monitoring, tracing).

How to draw boundaries: follow business capabilities (orders, payments, catalog), not technical layers; aim for high cohesion inside a service and loose coupling between services. A good boundary is one where a change rarely needs two teams to release together. Each service owns its database; others get the data through its API or events, never by reading its tables. Services talk synchronously (HTTP or gRPC: simple, but couples availability) or asynchronously (events through a broker: decoupled, but eventually consistent). An API gateway gives clients one entry point and handles authentication and routing.

A mature answer starts with: "A modular monolith is often the right first step. I would split out a service when a part has different scaling needs, a separate team, or a different release cadence." Starting with ten services for a small team is a common mistake.

What happens when a service is down? Timeouts, retries and circuit breakers

Short answer: Remote calls fail, so every call needs a timeout (otherwise threads pile up waiting), a bounded retry with exponential backoff and jitter (only for idempotent operations, to avoid a retry storm), and a circuit breaker. The breaker watches failures: after a threshold it opens and calls fail fast without touching the sick service, which gives it time to recover and protects the caller's resources. After a cooldown it goes half-open and lets a trial request through: success closes it, failure re-opens it. Other tools are bulkheads (separate thread pools per dependency so one slow service cannot exhaust all threads), fallbacks (cached or default data) and rate limiting.

Node.js A circuit breaker that fails fast

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Read the log: the first three calls hit the failing service and trip the breaker; calls four and five fail immediately (only 3 real calls were made); after the cooldown, one trial call goes through, succeeds, and the circuit closes again.

How do you keep data consistent without distributed transactions?

Short answer: Each service has its own database, so you cannot use one ACID transaction across them. Two-phase commit across services is slow and fragile, so the standard answer is a saga: a sequence of local transactions, each publishing an event or command for the next step, with a compensating action to undo each completed step if a later one fails. Sagas come in two styles: choreography (services react to each other's events) and orchestration (a coordinator tells each service what to do). The result is eventual consistency.

A related trap is the dual write: saving to the database and publishing an event are two separate operations, and a crash between them loses one. The transactional outbox pattern writes the event to an outbox table in the same local transaction as the data, and a relay publishes it afterwards (at least once, so consumers must be idempotent).

Node.js A saga with compensation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The payment step fails, so the saga runs the compensation for the one step that had completed (reserve stock) and the state is restored. A real saga also has to handle a failing compensation and persistence of the saga's own progress.

What is Kafka and what does it guarantee?

Short answer: Kafka is a distributed, replicated, append-only log. Producers write records to a topic, which is split into partitions; each record gets an offset (its position in the partition). Records are kept for a retention period (they are not deleted when read), so consumers can replay.

The facts to know:

  • Ordering is guaranteed only within a partition. Records with the same key go to the same partition, so all events for one customer stay ordered.
  • A consumer group shares the work: each partition is read by exactly one consumer in the group, so the number of useful consumers is capped by the number of partitions. Different groups each get every record independently.
  • Offsets and delivery: a consumer commits its offset after processing. If it crashes before committing, it re-reads from the last committed offset: at-least-once delivery. Committing before processing would give at-most-once. "Exactly-once" is achieved with idempotent producers, transactions (in Kafka-to-Kafka flows) or an idempotent consumer.
  • Replication: each partition has a leader and followers; acks=all with a minimum in-sync replica count trades latency for durability.
  • Rebalancing: when consumers join or leave, partitions are redistributed, which pauses consumption briefly.
  • Kafka versus a queue: a classic queue removes a message when it is consumed; Kafka keeps the log and consumers track their own position.
Node.js A partitioned log with a consumer group

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Notice the key idea in the output: all of Alice's events land in one partition, in order. Each partition is owned by one worker in the group, and after the offsets are committed a second poll delivers nothing. When the consumer "crashes" before committing, the same six records are delivered again, which is exactly why consumers must be idempotent. (This model has three partitions, and one of them happens to be empty, because only three distinct keys were used.)

What does Kubernetes do when you deploy a new version?

Short answer: Kubernetes is a declarative control system. You describe the desired state ("3 replicas of image shop:v2") and controllers continuously compare it with the observed state and take actions to close the gap: the reconciliation loop. A Deployment manages ReplicaSets, which manage Pods. Changing the image creates a new ReplicaSet and performs a rolling update: it starts new pods and removes old ones gradually, controlled by maxSurge and maxUnavailable, and only counts a new pod as available when its readiness probe passes.

YAML
# Read-only: a Deployment with probes and resource settings
apiVersion: apps/v1
kind: Deployment
metadata:
  name: shop
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate: { maxSurge: 1, maxUnavailable: 0 }
  selector:
    matchLabels: { app: shop }
  template:
    metadata:
      labels: { app: shop }
    spec:
      containers:
        - name: shop
          image: shop:v2
          resources:
            requests: { cpu: 250m, memory: 256Mi }
            limits: { memory: 512Mi }
          readinessProbe:
            httpGet: { path: /ready, port: 8080 }
          livenessProbe:
            httpGet: { path: /health, port: 8080 }

The distinctions interviewers check:

  • Liveness versus readiness: a failing liveness probe means the container is stuck, so Kubernetes restarts it. A failing readiness probe means it is not ready for traffic, so it is removed from the Service's endpoints but not restarted.
  • Requests versus limits: a request is what the scheduler reserves when placing the pod; a limit is the ceiling. Exceeding a memory limit gets the container killed (OOMKilled); exceeding a CPU limit only throttles it.
  • Service types: ClusterIP (internal), NodePort, LoadBalancer (cloud load balancer); an Ingress routes HTTP traffic by host and path.
  • Scaling: the Horizontal Pod Autoscaler changes the replica count from metrics such as CPU; the cluster autoscaler adds nodes.
  • Config: ConfigMap for settings, Secret for sensitive values (base64-encoded, not encrypted by default, so enable encryption at rest and restrict access).
  • Rollback: kubectl rollout undo deployment/shop returns to the previous ReplicaSet.

The stand-in below models the loop itself: desired state is three pods on v2, observed state is two pods on v1, and the controller makes one safe change per tick.

Node.js A reconciliation loop and a rolling update

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The shape matches a real rolling update: add a new pod, wait until it is ready, remove one old pod, and repeat, so capacity never drops below what is needed.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A circuit breaker has opened after repeated failures. What does it do to subsequent calls?

    An open breaker rejects calls immediately to protect resources and give the dependency time to recover; after a cooldown it goes half-open and tests with one call.
  2. 2. Which statement about Kafka ordering is correct?

    A topic is split into partitions; each partition is an ordered log. Using the same key keeps related events in order.
  3. 3. A consumer crashes after processing a record but before committing its offset. What happens?

    The group resumes from the last committed offset, so the record is processed twice unless processing is idempotent.
  4. 4. Why do microservices use sagas instead of one distributed transaction?

    A saga gives eventual consistency by undoing completed steps with compensations when a later step fails.
  5. 5. What does a failing readiness probe cause in Kubernetes?

    Readiness controls whether a pod receives traffic. Liveness controls restarts.

Summary

  • Microservices give independent deployment and scaling at the cost of network failures, distributed data and operational load; a modular monolith is often the better start.
  • Protect every remote call with timeouts, bounded retries with backoff and jitter (idempotent operations only), bulkheads and a circuit breaker.
  • Replace distributed transactions with sagas (local transactions plus compensations) and the transactional outbox; accept eventual consistency and make consumers idempotent.
  • Kafka is a partitioned, replicated log: ordering per partition, one consumer per partition within a group, offsets committed after processing for at-least-once delivery.
  • Kubernetes reconciles desired and observed state; Deployments roll out via ReplicaSets, readiness probes gate traffic, liveness probes restart, and requests and limits differ.

Next: Low-Level Design Questions.