VINAY BHAJANTRI
← SELECTED WORK SYSTEM / CASE STUDY ONLINE
CASE STUDY / Cloud Reliability / AWS

HealOps

A cloud-native reliability platform that captures real ECS failures, records incidents in DynamoDB, measures recovery time, and visualizes the incident lifecycle through FastAPI.

ENGINEERING SIGNAL / 01

Observing real AWS failures end to end — from unhealthy task to replacement, incident record, and measured recovery.

SOURCE ↗
~/portfolio/healops/architecture LIVE SYSTEM MAP
01Runtime + healthECS Fargate / ALB
02Failure signalsECS events / EventBridge
03Incident processorAWS Lambda / Event processing
04Incident system of recordDynamoDB / NoSQL incident model
05Dashboard + deliveryFastAPI / Terraform
REQUEST→LOGIC→STATE→DELIVERY
STACK /AWSECS FargateALBEventBridgeLambdaDynamoDBFastAPITerraformJenkinsECR
01 / Product Context

Why this system exists.

HealOps is a cloud reliability and failure-observation platform built around actual AWS-managed events. ECS Fargate runs the workload behind an ALB; failures produce ECS events, EventBridge routes them to Lambda, DynamoDB stores the incident lifecycle, and a FastAPI dashboard exposes what happened and how quickly the platform recovered.

A self-healing platform can replace a failed task and still leave engineers with little understanding of the incident. The goal was to make recovery observable: capture the failure signal, preserve its reason and infrastructure context, record recovery, calculate MTTR from AWS timestamps, and present a useful incident history without inventing synthetic data.

02 / Engineering Role

End-to-end ownership.

I built the reliability workflow across application, infrastructure, and delivery: FastAPI endpoints and controlled failure injection, ECS/ALB runtime behavior, EventBridge rules, Lambda incident processing, DynamoDB records, Terraform infrastructure, and a Jenkins pipeline that builds, pushes to ECR, and deploys updated ECS task revisions.

03 / Architecture

The product as connected layers.

01

Runtime + health

The application runs as an ECS Fargate service behind an ALB. Health checks determine whether a task remains eligible for traffic and ECS maintains the service desired count.

ECS Fargate · ALB · Target Groups
02

Failure signals

Task stops, unhealthy behavior, deployment failures, and essential container exits become AWS events instead of being inferred from a fake dashboard state.

ECS events · EventBridge
03

Incident processor

Lambda parses incoming ECS events, classifies failures, writes structured incident metadata, and participates in recording the incident lifecycle used for MTTR.

AWS Lambda · Event processing
04

Incident system of record

DynamoDB stores failure reason, detection and recovery timestamps, MTTR, healing action, cluster/task metadata, and incident status.

DynamoDB · NoSQL incident model
05

Dashboard + delivery

FastAPI exposes dashboard and incident APIs. Terraform provisions the AWS foundation and Jenkins drives Docker build, ECR push, and ECS deployment.

FastAPI · Terraform · Jenkins · ECR
04 / Core Workflows

Where product logic becomes behavior.

01

Inject → fail → replace

  1. Controlled endpoint crashes, hangs, or becomes unhealthy
  2. ALB/ECS observes unhealthy runtime behavior
  3. ECS stops or replaces the affected task
  4. The service returns toward its desired healthy state
02

Event → incident record

  1. AWS emits the ECS failure event
  2. EventBridge matches the event
  3. Lambda parses and classifies the failure
  4. Structured incident metadata is written to DynamoDB
03

Detect → recover → MTTR

  1. Detection time comes from the AWS event
  2. Recovery is captured from the platform lifecycle
  3. MTTR is calculated from detection and healed timestamps
  4. Dashboard exposes incident state and recovery evidence
04

Code → ECS delivery

  1. Code change enters Jenkins pipeline
  2. Docker image is built and pushed to ECR
  3. ECS service receives a new task revision
  4. ALB validates health before the workload serves normally
05 / Engineering Decisions

Not just what was built — why.

DECISION / 01

Use real AWS signals, not mocked incidents

The project treats ECS/EventBridge timestamps and failure metadata as the source of truth. The dashboard reflects platform behavior instead of generating a separate simulated incident story.

DECISION / 02

Observe self-healing instead of reimplementing it

ECS already maintains desired task count. HealOps focuses on capturing and explaining that recovery rather than adding another replacement mechanism on top of the managed scheduler.

DECISION / 03

Persist incidents in DynamoDB

Incident events are append-friendly operational records with structured metadata and variable failure details, making DynamoDB a natural system of record for the event-driven path.

DECISION / 04

Provision the reliability path as code

Terraform defines networking, ECS, ALB, EventBridge, IAM, Lambda, and DynamoDB so the demonstration is reproducible as infrastructure, not a sequence of manual console clicks.

06 / Challenges

Complexity designed around.

CHALLENGE / 01

Turning asynchronous cloud events into one incident lifecycle

Failure detection and recovery do not arrive as one synchronous request. The system has to preserve enough event context and timestamps to present a coherent failure-to-healing story.

CHALLENGE / 02

Separating application failure from platform recovery

The app deliberately exposes controlled failure modes, while ECS and ALB determine health and replacement behavior. That separation keeps the experiment close to real managed-service behavior.

CHALLENGE / 03

Measuring recovery without fake clocks

MTTR is derived from AWS-originated detection and recovery timestamps rather than browser timers or manually entered values.

CHALLENGE / 04

Connecting runtime, events, storage, and delivery

The useful result only appears when ECS/ALB health, EventBridge, Lambda, DynamoDB, FastAPI, Terraform, and Jenkins work as one operational system.

07 / Outcome

What the system supports now.

  • Real ECS task and health-related failures are captured as incidents instead of being represented by mock data.
  • Incident records include failure reason, detection/recovery timestamps, MTTR, healing action, and infrastructure metadata.
  • Controlled crash/unhealthy/hang endpoints provide repeatable ways to validate AWS recovery behavior.
  • Terraform provisions the AWS reliability architecture and Jenkins automates build-to-ECS delivery through ECR.
  • Observed recoveries in the project demonstrate fast managed-service replacement while preserving the evidence needed to inspect what happened.

HealOps is backed by a public GitHub repository. The case study is derived from the implemented AWS architecture, documented failure-injection paths, infrastructure code, and deployment workflow.

← BACK TO SELECTED WORK