LALIT's TECH BLOG

Circuit Breaker pattern in Scala

Posted on Mar 28, 2020

image

Circuit breaker pattern is a common microservice resiliency pattern to make system responsive after series of failures and have a fallback mechanism.

[NOTE] Spoiler — This would be a simple implementation and the Scala code would be stateful and have side-effects.

The Need

Doing a remote call or executing a task which is outside the domain boundary of the system is very common in modern applications. Especially in microservice world, we are bound to make calls to other microservices.

In such scenarios, we need to consider eventual failures. If some failures are too continuous and consistent in nature, they can hog too much of the system resources resulting into failures. We can avoid such cases using the circuit-breaker pattern.

The Concept

image courtesy: https://martinfowler.com/bliki/CircuitBreaker.html

The concept is simple, the circuit represents the connection between the caller and the callee. It can only have 3 states: closed, open and half-open. The system starts with the “closed” circuit initially.

We monitor the consecutive (or within a time window) failures from the callee and when the failures exceed a threshold we trip the circuit to “open” and all the further calls from the caller won’t reach the callee and rather caller is returned with a default fallback result.

— can be implemented using a failure result counter.

We also keep monitoring the time since opening the circuit and when a certain timeout is reached, we put the circuit to “half-open” meaning, only one ‘probe’ call will pass through and its result will determine if the circuit will be closed or opened based on its success or failure.

— can be implemented using a background timer.

Implementation

CircuitState — a trait extended by possible states

CircuitResult[T] — a trait extended by Success and Failure results of the Circuit with success return type T.

Circuit[R] — a class denoting a circuit where the call returns the result of type Future[R], future denoting an async call.

States of the Circuit

Transition of states

Main block execution handler — It acts as a router and invokes other handlers based on the Circuit’s state. After state transition, the thread request may come back here to be routed again based on the updated state.

Handling failure response from the Callee —

Handling closed circuit— Executing the provided block of code and yield a success or failure as the result asynchronously, while maintaining/updating the count of consecutive failures.

Handling Half-Open Circuit —

Handling open circuit— return the successful future with CircuitSuccess wrapping the default value.

Half-Opener Background Timer — Will trip the circuit to half-open after timeout reached since opening the circuit.

CircuitImplicits — Implicits making the use of our Circuit class a breeze!

We can now do something like:

implicit val sampleCircuit: Circuit[Int] = new Circuit[Int](
    "sample-circuit",
    5,                // max allowed consecutive closed failures
    5.seconds,        // half-opener timeout
    1,                // max allowed consecutive half-open failures
    -1,               // invalid success result
    println
)

//this will use the above created circuit to execute
Future.successful(2 + 2).executeAsync

Testing it out — You can find the test cases here.

image