There are several different deployment models and tools to implement those models…
One important way to classify deployments is in-place, where instances are upgraded making for a faster upgrade, or disposable, where instances are treated as immutable and new instances are created and old instances are destroyed which makes it safer, slower, and a bit more expensive. AZ & regional capacity issues also make immutable infrastructure difficult at scale.
Another way to classify deployments is bootstrapping, where code is installed to a running instance at launch, versus pre-baking where the application is loaded on the image.
There are numerous different types of deployments including all instances at one time, rolling, immutable environment, and blue/green.
In this scenario, all instances at one time regardless of how many instances there are..
- Downtime: yes; because all instance get the new deployment at the same time
- Failed deployment: downtime; redeploy the last known good configuration
- DNS: no change
- Speed: fast
In a Rolling Deployment deployment only some instances at a time are upgraded in place. In many scenarios, a minimum number of instances stays in service.
- Downtime: none
- Failed Deployment: Rollback affected instances
- DNS: none
- Speed: longer
An Immutable Environment deployment is similar to a Rolling deployment except with new resources. For instance, two autoscaling groups would be used.
- Downtime: None
- Failed Deployment: Rollback new resources; slow
- DNS: none
- Speed: longer; costs more
In a Blue-Green deployment a new identical environment is created then tested and something, perhaps DNS or other setting is changed to send the new environment live. The Blue environment is the live environment; the green environment is new, not-yet-live version. Works best when implemented on loosely coupled applications; doesn’t work well when schema is tightly couple to application, deployment aware application using feature flags, or is COTS app that isn’t blue/green friendly.
- Downtime: None
- Failed Deployment: The fastest; simply change networking configuration back
- DNS: Yes, sometimes; this can be a disadvantage because of DNS caching
- Speed: longer; costs more
A/B testing or canary testing is a variation where a portion of traffic is sent to the green environment to test it out. A/B might not be for deployment; it might be just for testing.
- Update DNS Routing - works well for single instance, or an ELB in conjunction with ASG, ECS or EB; DNS changes are problematic; ELB prewarming may be required; the higher costs or AWS service limits might be an issues because of two concurrent online systems during deployment
- Swap ASG behind ELB - startup new ASG with new configuration; canary test green systems; put some blue instances in
StandByDon’t wanna mess with DNS; very fast rollback with no DNS change; swapping ECS Service behind ELB is a similar idea
- Swap Launch Config in ASG - Update launch config; double ASG Size; put old config in standby; OK? reduce ASG size terminating old configuration first (Barely blue/green); difficult canary analysis -> hard to detect errors with two launch configs in play and schema changes must be independent of app changes (ideally); switching ECS Task definitions is a similar idea
- Swap EB Environments - Create new environment with a new version; Do a “Swap Environment URL”
- Clone OpsWorks Stack - Clone stack; deploy version using
Deploycommand; change DNS
Schema change is a giant problem with blue/green deployment; decoupling the schema change from the code change which can be be implemented as schema change first or schema change last. Schema change first is preferred because rolling back a schema change in production is a high risk manuever.
Schema change can also be handled by creating a centralized write process either through a queue used for writes during the deployment OR the green environment writes to both databases OR each environment writes to there own DB and uses an async process to synchronize the data.
In-place deployments are traditional method of application deployment. Both deployment and rollback cause downtime so it is hard to have time to debug a problem in the production environment with the new version. In-place deployments are generally faster, good or maybe better for session-less apps and ideal for OpsWorks or Puppet.
Disposable deployment types are generally slower because new resources must be created and provisioned. Tools that have great support for Green/Blue deployments like Elastic Beanstalk, CodeDeploy/ALB, APIGateway, and CloudFormation are better DevOps tool choices in these scenarios.
Product Strengths and Weaknesses
EB - fastest and simplest way to deploy code; Beanstalk doesn’t allow as much flexibility for some configurations and deployments and is more suitable for shorter application lifecycles where an environment can be thrown away with each deploy; not good for lots of dependencies with more than apt-get functionality or heavy configuration; can provision RDS but don’t want to; only changes to git repos are moved to pro; source generally from S3
Opsworks - more focused on application management; can’t do rolling batch with Opsworks…
Cloudformation - control; can combine with EB or OpsWorks
Hybrid Deployment Models
CloudFormation with Puppet - puppet master holds instruction and definitions; clients connect and follow instructions
CloudFormation with Chef - better for longer deployments
CloudFormation with Elastic BeanStalk - less flexible than OpsWorks; better with immutable deployments
Scenario -> Product
- maintain capacity; Speed not issue = rolling with batch ; Opsworks? Cloudformation
- in-place? Opsworks & CodeDeploy
- Super fast? in-place; all-at-once faster than rolling
- Consistent or lots of deployments? in-place
- No DNS change? all-at-once, rolling, immutable, blue/green with launch config change OR ASG swap
- No downtime? Blue/Green or rolling
- Pre-baked AMI? Cloudformation & OpsWorks
- Canary = Route53 or an ELB; not well supported with Elastic Beanstalk