27 November 2018

Airbnb's Journey from Self-Managed Redis to ElastiCache for Redis (DAT319)

by mo


Julie Trias

At Airbnb, we use Redis extensively as an in-memory data store to reduce latency and provide sub-millisecond response for our website, search, images, payments, and more. We migrated our self-managed Redis environment from EC2 classic to fully-managed Amazon ElastiCache for Redis to reduce operational overhead and improve availability. Now, all our Redis is in an AWS managed service that provides multi-Availability Zone support, automatic failover, and maintenance. Attend this session to learn how we migrated our Redis environment while ensuring data integrity and zero downtime.

  • Redis -> ElastiCache

Lots of data. S3 (104 PB) Aurora, Redis

82 ec2 redis clusters -> 113 elasticache redis clusters -> 146 clusters

redis is not just memcache.

  • authentication -> sessions
  • feeds for homes
  • messaging

SmartStack

redis (primary) -> (replica) (SmartStack Redis)

cons:

  • operation overhead
  • manual failover
  • no support for sharding
  • no encryption support

HARedis

pros

  • auto failover
  • multi AZ

cons

  • more ops overhaed
  • no support for sharding
  • no enc support
  • many restricted commands

why elasticache?

  • difficult to manage
  • poor experience
  • doesn’t scale
  • expensive ec2 instace types (memory intensive)

elasticache redis

  • key value store
  • fully managed and hardened
  • secure compliant
    • enc at rest
  • ha and reliable
    • multple primaries
    • multi AZ
    • auto failover
  • easily scalable
    • up to 6.1 TiB of in memory dataq.
  • zero downtime
  • no data loss
  • no cold starts

chef config change pushed out to point to new redis cluster.

master in ec2, replica in ec2, replica in elasti-cache replica in elasticache would become the new primary.

nerve is an open source tool that helps with service discover. the nerves in your body report to your brain when a signal is received.

direct traffic to elasticache replica. promote elasticache node to primary. elasticache is multi AZ.

Resources:

devops