Zero‑Touch AWS Elastic Disaster Recovery: Full‑Lifecycle Automation of DRS Agents with SSM and EventBridge

Introduction

Enterprises that run large‑scale EC2 fleets (think tens of thousands of instances) face a silent but critical challenge: keeping every machine continuously recoverable with AWS Elastic Disaster Recovery (DRS). The DRS agent is the linchpin – it continuously streams block‑level snapshots to the recovery site. Miss a single install, or let an agent drift out of sync, and you expose a recovery blind spot.

Traditional approaches rely on ad‑hoc scripts or AWS Lambda functions triggered by CloudWatch events. While functional, they introduce cold‑start latency, runtime limits, and a maintenance overhead that scales poorly. In this editorial we dissect a pure‑SSM + EventBridge solution that uses a single EC2 tag to drive the entire agent lifecycle – install → heal → offboard – without any Lambda functions.

“If you can describe the state of a machine with a tag, let the control plane do the heavy lifting.” – Mubashir Arham, Founder & Chief Architect, FutureStack

Deep Architectural Breakdown

1. The DRS Agent Lifecycle

The agent itself is a systemd service (drs-agent.service) that talks over TLS‑mutual authentication to the DRS replication endpoint. All state (registration ID, replication token) lives in the instance’s /etc/drs directory and is persisted across reboots.

2. Systems Manager Automation Engine

AWS Systems Manager (SSM) provides two key primitives:

1. Automation Documents (SSM Documents) – YAML/JSON definitions that describe a sequence of steps (run‑command, AWS API calls, branching). They run in the SSM Execution Engine, a fully managed, horizontally‑scaled service that can execute up to 10,000 concurrent automations per region.
2. Targeting via Resource Groups & Tags – SSM can resolve a dynamic target list at execution time based on EC2 tags, allowing us to treat a tag as a single source of truth for the desired DRS state.

The automation documents we use are idempotent:

• The install document checks for the presence of /opt/aws/drs/agent before downloading.
• The heal document queries systemctl status drs-agent and validates the binary checksum.
• The offboard document confirms that the instance is not part of any active recovery plan before deregistering.

Because the automation runs in the control plane, there is no dependency on the instance’s compute resources – the heavy lifting (downloading the agent, invoking the DRS registration API) is performed by the SSM Run Command worker nodes, which are provisioned on demand.

3. EventBridge Event Routing

EventBridge acts as the event bus that glues tag changes to automation triggers. The flow is:

1. Tag Change Detection – EC2 emits a TagChange event whenever a tag is added, modified, or removed. This event includes the resourceId, key, and value.
2. Rule Filtering – A single EventBridge rule matches on:
   JSON

   8 linesCopy

   {
     "source": ["aws.ec2"],
     "detail-type": ["EC2 Tag Change"],
     "detail": {
       "tagKey": ["DRSState"],
       "tagValue": ["Install", "Heal", "Offboard"]
     }
   }

3. Target – The rule invokes the SSM Automation Execution API (StartAutomationExecution) with the appropriate document ARN and passes the instance ID as a parameter.

Because EventBridge guarantees at‑least‑once delivery and sub‑second latency (typically 200‑400 ms), the system reacts to a tag change almost instantly, satisfying the “near‑real‑time” requirement for DR readiness.

4. State Management via EC2 Tags

The single tag DRSState encodes the desired lifecycle state:

• Install → triggers the install automation.
• Heal → triggers the health‑check automation (run on a schedule and on demand).
• Offboard → triggers the deregistration automation.

Using a tag as the source of truth eliminates the need for an external state store (DynamoDB, SSM Parameter Store) and ensures eventual consistency: if the tag drifts, the next EventBridge event will re‑apply the correct automation.

5. Latency & Compute Dynamics

Because the heavy network I/O (binary download, TLS handshake) occurs on SSM worker nodes that reside in the same VPC (via VPC endpoints), we avoid Internet egress costs and keep latency predictable.

FutureStack Perspective (First‑Person Commentary)

Mubashir Arham – Chief Architect, FutureStack

When I first saw teams cobble together a Lambda‑based choreography for DRS agent management, I recognized two systemic flaws:

1. Cold‑start latency – Even a 50 ms Lambda cold start adds up when you are orchestrating thousands of instances. Multiply that by a rolling deployment and you have minutes of unnecessary downtime.
2. Operational debt – Maintaining separate Lambda functions, IAM roles, and CloudWatch log groups creates a sprawl that quickly outpaces the original intent of “automation”.

The SSM‑first pattern solves both problems. SSM Automation is stateful, idempotent, and native to the EC2 lifecycle. By anchoring the desired state to a single tag, we enforce a declarative model that mirrors the modern “GitOps” philosophy – the tag is the manifest, the automation is the reconciler.

Where most engineering teams stumble is over‑engineering the observability layer. It’s tempting to push every metric into CloudWatch Logs or a custom dashboard. In practice, the SSM Execution History already provides per‑step status, timestamps, and error payloads. By leveraging SSM Execution Triggers (another EventBridge source), you can surface failures directly to an Opsgenie alert without an extra monitoring stack.

Finally, don’t forget security hygiene. The automation documents run with an IAM role that has the minimal ssm:StartAutomationExecution, drs:Register*, and drs:Deregister* permissions. Because the role is instance‑agnostic, you can rotate it centrally without touching each EC2 profile.

In short, the tag‑driven SSM‑EventBridge pipeline is the canonical way to achieve zero‑touch DRS agent management at scale. It aligns with FutureStack’s guiding principle: Let the cloud’s control plane do the heavy lifting, and keep your data plane lean.

Code / Implementation Blueprint (TypeScript CDK)

Below is a complete CDK construct that provisions:

1. The EventBridge rule listening for DRSState tag changes.
2. The three SSM Automation documents (install, heal, offboard).
3. The IAM role with least‑privilege permissions.

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as iam from 'aws-cdk-lib/aws-iam';
import * as ssm from 'aws-cdk-lib/aws-ssm';
import * as events from 'aws-cdk-lib/aws-events';
import * as targets from 'aws-cdk-lib/aws-events-targets';

export class DrsAgentAutomationStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // 1️⃣ IAM role for SSM Automation execution
    const automationRole = new iam.Role(this, 'DrsAutomationRole', {
      assumedBy: new iam.ServicePrincipal('ssm.amazonaws.com'),
    });
    automationRole.addToPolicy(new iam.PolicyStatement({
      actions: [
        'ssm:StartAutomationExecution',
        'drs:RegisterAgent',
        'drs:DeregisterAgent',
        'drs:CreateReplicationConfiguration',
        'drs:DeleteReplicationConfiguration',
        'ec2:DescribeInstances',
      ],
      resources: ['*'], // tighten with ARN conditions in prod
    }));

    // 2️⃣ Automation Documents (simplified for brevity)
    const installDoc = new ssm.CfnDocument(this, 'InstallDrsAgent', {
      documentType: 'Automation',
      content: {
        schemaVersion: '0.3',
        description: 'Install and register DRS agent',
        mainSteps: [
          {
            action: 'aws:runShellScript',
            name: 'downloadAgent',
            inputs: {
              runCommand: [
                'if [ ! -d /opt/aws/drs ]; then',
                '  yum install -y https://drs-agent.s3.amazonaws.com/latest.rpm',
                'fi',
              ],
            },
          },
          {
            action: 'aws:executeAwsApi',
            name: 'registerAgent',
            inputs: {
              Service: 'drs',
              Api: 'RegisterAgent',
              InstanceId: '{{ InstanceId }}',
            },
          },
        ],
      },
    });

    const healDoc = new ssm.CfnDocument(this, 'HealDrsAgent', {
      documentType: 'Automation',
      content: {
        schemaVersion: '0.3',
        description: 'Validate and heal DRS agent',
        mainSteps: [
          {
            action: 'aws:runShellScript',
            name: 'checkService',
            inputs: {
              runCommand: [
                'systemctl is-active --quiet drs-agent || systemctl restart drs-agent',
              ],
            },
          },
        ],
      },
    });

    const offboardDoc = new ssm.CfnDocument(this, 'OffboardDrsAgent', {
      documentType: 'Automation',
      content: {
        schemaVersion: '0.3',
        description: 'Deregister and clean up DRS agent',
        mainSteps: [
          {
            action: 'aws:executeAwsApi',
            name: 'deregister',
            inputs: {
              Service: 'drs',
              Api: 'DeregisterAgent',
              InstanceId: '{{ InstanceId }}',
            },
          },
          {
            action: 'aws:runShellScript',
            name: 'removeAgent',
            inputs: {
              runCommand: [
                'yum remove -y drs-agent',
                'rm -rf /opt/aws/drs',
              ],
            },
          },
        ],
      },
    });

    // 3️⃣ EventBridge rule for tag changes
    const drsTagRule = new events.Rule(this, 'DrsTagChangeRule', {
      eventPattern: {
        source: ['aws.ec2'],
        'detail-type': ['EC2 Tag Change'],
        detail: {
          tagKey: ['DRSState'],
          tagValue: ['Install', 'Heal', 'Offboard'],
        },
      },
    });

    // Map tag values to the appropriate Automation document ARN
    const documentMap: { [key: string]: string } = {
      Install: installDoc.attrArn,
      Heal: healDoc.attrArn,
      Offboard: offboardDoc.attrArn,
    };

    // Target uses a custom InputTransformer to inject the correct doc ARN
    drsTagRule.addTarget(
      new targets.SsmAutomationTarget({
        documentName: events.RuleTargetInput.fromObject({
          // Placeholder – actual selection happens in a Lambda‑free way via
          // EventBridge InputTransformer (requires CloudFormation custom resource).
          // For brevity, we assume a static mapping per tag value.
        }),
        role: automationRole,
        parameters: {
          InstanceId: events.RuleTargetInput.fromEventPath('$.detail.resourceId'),
        },
      })
    );
  }
}

Note: The CDK snippet abstracts the InputTransformer logic for brevity. In production you would use events.RuleTargetInput.fromObject with a Map expression or a Step Functions state machine that selects the document ARN based on the tag value.

Empirical Trade‑Off Table

*Latency measured from the moment the EC2 tag is applied to the moment the DRS agent reports healthy in the DRS console.

Future Industry Trajectory (18‑36 months)

1. Agent‑less Replication – AWS is already experimenting with EBS‑direct streaming that bypasses a host‑resident agent. When that becomes GA, the install/heal/offboard problem will evaporate, but the declarative tag model will still be valuable for expressing intent (e.g., DRSState=Enabled).
2. Policy‑as‑Code for DR – Expect tighter integration between AWS Config, Open Policy Agent (OPA), and SSM Automation. Teams will codify DR compliance rules (must have DRS agent v≥2.5) and automatically remediate via the same automation pipeline.
3. Cross‑Account Orchestration – Large enterprises operate multi‑account landing zones. EventBridge Event Buses can now fan‑out tag events across accounts, enabling a central DR‑control tower that drives agent lifecycle in every workload account.
4. AI‑augmented Health Scoring – FutureStack is prototyping a model that ingests SSM execution logs, CloudWatch metrics, and DRS replication lag to produce a DR health score. The model can trigger a Heal automation pre‑emptively before a failure manifests.
5. Zero‑Trust Networking for Replication – With the rise of VPC‑Lattice and AWS PrivateLink, replication traffic will be confined to private fabrics, reducing attack surface and eliminating the need for outbound Internet egress when downloading agents.

Bottom line: The tag‑driven SSM‑EventBridge pattern is a foundational building block. It not only solves today’s operational pain points but also provides a plug‑in point for the next wave of serverless DR innovations.

Closing Thoughts

Automating the DRS agent lifecycle without Lambda is more than a cost‑saving trick; it is a architectural shift toward control‑plane‑centric orchestration. By leveraging SSM Automation’s idempotency, EventBridge’s low‑latency event routing, and EC2 tags as the single source of truth, you achieve:

• Sub‑second reaction to state changes.
• Unlimited horizontal scalability.
• A dramatically reduced operational burden.

If your organization still relies on ad‑hoc scripts or Lambda‑driven agents, you are paying for complexity that the AWS control plane already offers for free. Adopt the pattern, bake it into your IaC pipelines, and let FutureStack help you extend the same philosophy to other lifecycle‑intensive services (e.g., AWS Backup, EFS replication, RDS cross‑region clones).

Stay resilient, stay automated.