slides

z, ?	toggle help (this)
space, →	next slide
shift-space, ←	previous slide
d	toggle debug mode
## <ret>	go to slide #
c, t	table of contents (vi)
f	toggle footer
r	reload slides
n	toggle notes
p	run preshow

loading presentation...

Building a cloud service on a cloud infrastructure at

Also, cloud.

Mikhail Panchenko, Surge 2011

Who Am I?

- Pancakes
- Infrastructure Engineer at SimpleGeo
- Backend Engineer at Flickr before that
- Backend and Frontend Engineer at Yahoo! Ops/Tools before that
- Philosophy, Economics, and French major before that
- @mihasya
- pancakes@simplegeo.com

- Tools for mobile/geo developers
- Primarily focused on services, some data-oriented APIs
- PaaS, I guess? I've lost track a bit
- Availability, redundancy part of brand
  - Our outage = your outage
- No pressure

Agenda

Goals
A little bit of theory
Challenges in The Cloud
General Architecture
Implementation Details

Note on elasticity - this isn't just about outages

Architectural Goals

High availability
Linear scalability
Elasticity/Flexibility
Redundancy/Fault Tolerance

Read: don't wake me up, please

Sound Familiar?

this talk isn't solely about the cloud, cloud adds challenges to universal problems

Some Theory, Food for Thought

reference Allspaw, Ben Fried for context

The Internets as Complex Systems

http://www.amazon.com/Normal-Accidents-Living-High-Risk-Technologies/dp/0691004129

"Complex interactions are those of unfamiliar sequences, or unplanned and unexpected sequences, and either not visible or not immediately comprehensible."

Charles Perrow. Normal Accidents: Living with High-Risk Technologies (p. 78). Kindle Edition.

"The notion of baffling interactions is increasingly familiar to all of us. [...] As systems grow in size and in the number of diverse functions they serve, and are built to function in ever more hostile environments, increasing their ties to other systems, they experience more and more incomprehensible or unexpected interactions. They become more vulnerable to unavoidable system accidents."

Charles Perrow. Normal Accidents: Living with High-Risk Technologies (p. 72). Kindle Edition.

Fortunately,
This Is Only The Internet

"The beauty of this is its simplicity. Once a plan gets too complex, everything can go wrong."

Walter Sobchak, The Big Lebowski

Interactions

Linear = assembly line, Complex = nuclear powerplant, web arch

Linear vs Complex

Coupling

tight = heart surgery, loose = dinner (dishes, groceries, fallback to pizza)

Tight vs Loose

Three Mile Island

"... they found that radioactive water was not traveling to the tank they intended, but because of complex flow and pressure interactions, was going to a different, wrong tank, which also overflowed, this time in the auxiliary building."

Charles Perrow. Normal Accidents: Living with High-Risk Technologies (pp. 22-23). Kindle Edition.

Amazon Web Services

"The traffic shift was executed incorrectly and rather than routing the traffic to the other router on the primary network, the traffic was routed onto the lower capacity redundant EBS network."

"Summary of the Amazon EC2 and Amazon RDS Service Disruption in the US East Region" http://aws.amazon.com/message/65648/

Common Theme

Previously independent systems become coupled as a result of unanticipated interactions, leading to fundamentally surprising results

When pumping radioactive water into the wrong tank, the behavior of the program is undefined

But where does The Cloud come in??

hat tip to Cal Henderson

The Trifle Analogy

Photo by mathematically_impossible

The Trifle Analogy

Photo by mathematically_impossible

A complex system consisting of complex subsystems

Photo by wwarby

The Trifle Analogy

Original photos by mathematically_impossible and miheco

Tightly coupled to a complex system over which you have no control and into which you have no insight

Photo by 20after4

Recall
"Baffling Interactions"

"The notion of baffling interactions is increasingly familiar to all of us. [...] As systems grow in size and in the number of diverse functions they serve, and are built to function in ever more hostile environments, increasing their ties to other systems, they experience more and more incomprehensible or unexpected interactions. They become more vulnerable to unavoidable system accidents."

Charles Perrow. Normal Accidents: Living with High-Risk Technologies (p. 72). Kindle Edition.

DECOUPLE DECOUPLE DECOUPLE
( also, simplify )

tuned mass damper from Taipei

Photo by erikcharlton

shared resources - flickr infinite recursion story

Decouple Your Subsystems

Shared resources are the most common source of unexpected interaction
Resist temptation to double up on roles
Use queues, caches as buffers
- NOTE: those are complex subsystems of their own

shared resources - flickr infinite recursion story

Decouple Your Subsystems

Explicit Decoupling

CPU Affinity
- Webserver on 1-7; SSH etc on 8
- Crude, but gets the job done
More robust solutions - containers

Decouple Your Functionality

Service architecture
Each service does one thing well
Easier to measure, understand, and accommodate resource demands
Reduce potential for interactions, cross-functional failure

Decouple from Your Environment with Configuration Management

- Decouple from your platform (OS/kernel)
  - Easy to test/bench potential candidates
  - Easy to migrate if you find a winner
  - This is especially important when dealing with cloud
- Automate as much of deploy/bootstrap process as possible
  - Probably won't help much during a provider outage due to stampede
  - BUT: DirectConnect
  - You might not always be in the cloud..

Decouple Your Datacenters

"temporarily disable a datacenter" instead of coming up with elaborate failure strategies

Most robust redundancy mechanism
Hot-hot keeps you on your toes
Simplifies, not just for the cloud
- Yahoo! now foregoing datacenter features like HVAC
- "If it gets too hot in Washington, turn that DC off for a while"
- I'm sure they're not the only ones

Decouple Your Datacenters

"AZ" - Basic building block for EC2
This is the level they (theoretically) decouple at
They are probably thinking along the same lines we are - must be able to turn off one AZ without impact in the other

stampede to the other datacenter if one goes down entirely; also, potential for AWS to fuck up (see EBS/RDS outage) - they are susceptible to all the same problems

( there's a hidden interaction there )

Every datacenter as an independent microcosm of your overall architecture

The Birds 'n' the Bees

Bird's Eye View

Photo by reschroederimages

Bird's Eye View

( note the absence of specifics )

Bird's Eye View

Maintenance - Divide & Conquer

Local Degradation - Divide & Conquer

Incompatible Upgrade - Guess!

Incompatible Upgrade - Yay!

Baffling Single Node Failure

`202 Accepted`

Spike in Write Traffic

Really simple operational steps for stressful tasks & situations

Temporally decouple the problem from the resolution

Go back to sleep

Photo by joshme17

Now, how about those specifics?

Write Path

ELB

Dynamic Load Balancing
Flexible virtual IP
Easy to add/remove AZs
Uses healthchecks to automatically evict nodes

Gate - "Layer 8 Proxy"

Lightweight Node.js daemon
OAuth
Rate Limiting
Basic routing to actual services

Recall
"Decouple Your Functionality"

Services - Pick Your Own Adventure

Node.js and Python
- Some people just hate Node.js
Can be anything, as long as Gate can talk to it
- ( another reason to decouple )
Highly specialized

Building a cloud service on a cloud infrastructure at

Also, cloud.

Who Am I?

Agenda

Architectural Goals

Read: don't wake me up, please

Sound Familiar?

Some Theory, Food for Thought

The Internets as Complex Systems

Fortunately, This Is Only The Internet

Interactions

Linear vs Complex

Coupling

Tight vs Loose

Three Mile Island

Amazon Web Services

Common Theme

When pumping radioactive water into the wrong tank, the behavior of the program is undefined

But where does The Cloud come in??

The Trifle Analogy

The Trifle Analogy

A complex system consisting of complex subsystems

The Trifle Analogy

Tightly coupled to a complex system over which you have no control and into which you have no insight

Recall "Baffling Interactions"

DECOUPLE DECOUPLE DECOUPLE( also, simplify )

Decouple Your Subsystems

Decouple Your Subsystems

Explicit Decoupling

Decouple Your Functionality

Decouple from Your Environment with Configuration Management

Decouple Your Datacenters

Decouple Your Datacenters

( there's a hidden interaction there )

Every datacenter as an independent microcosm of your overall architecture

The Birds 'n' the Bees

Bird's Eye View

Bird's Eye View

( note the absence of specifics )

Bird's Eye View

Maintenance - Divide & Conquer

Local Degradation - Divide & Conquer

Incompatible Upgrade - Guess!

Incompatible Upgrade - Guess!

Incompatible Upgrade - Yay!

Baffling Single Node Failure

202 Accepted

Spike in Write Traffic

Really simple operational steps for stressful tasks & situations

Temporally decouple the problem from the resolution

Go back to sleep

Now, how about those specifics?

Write Path

ELB

Gate - "Layer 8 Proxy"

Recall"Decouple Your Functionality"

Services - Pick Your Own Adventure

RabbitMQ

Cassandra

Partition Tolerance

It's not just for outages

Recall "Divide & Conquer"

This too is a partition

Thank You!

Fortunately,
This Is Only The Internet

Recall
"Baffling Interactions"

DECOUPLE DECOUPLE DECOUPLE
( also, simplify )

`202 Accepted`

Recall
"Decouple Your Functionality"

Recall
"Divide & Conquer"