12

Is it possible to do something like the Github zero downtime deploy on Heroku using Unicorn on the Cedar stack?

I'm not entirely sure how the restart works on Heroku and what control we have over restarting processes, but I like the possibility of zero downtime deploys and up until now, from what I've read, it's not possible

There are a few things that would be required for this to work.

  1. First off, we'd need backwards compatible migrations. I leave that up to our team to figure out.
  2. Secondly, we'd want to migrate the db right after a push, but before the restart (assuming our migrations are fully backwards compatible, this should not affect anything)
  3. Thirdly, we'd want to instruct Unicorn to launch a new master process and fork some workers, then swap the PIDs and gracefully shut down the old process/workers

I've scoured the docs but I can't find anything that would indicate this is possible on Heroku. Any thoughts?

Community
  • 1
  • 1
brad
  • 30,001
  • 27
  • 98
  • 151

4 Answers4

26

I can't address migrations, but the part about restarting processes and avoiding wait time:

There is an beta feature for heroku called preboot. After a deploy, it boots your new dynos first and waits a while before switching traffic and killing the old ones:

https://devcenter.heroku.com/articles/labs-preboot/

I also wrote a blog post that has some measurements on my app's performance improvements using this feature:

http://ylan.segal-family.com/blog/2012/08/27/deploy-to-heroku-with-near-zero-downtime/

Ylan S
  • 1,601
  • 15
  • 9
8

You might be interested in their feature called preboot.

Taken from their documentation:

This feature provides seamless deploys by booting web dynos with new code before killing existing web dynos.

Some apps take a long time to boot up, and this can cause unacceptable delays in serving HTTP requests during deployment.

There are a few caveats:

  • You must have at least two web dynos to use this feature. If you have your web process type scaled to 1 or 0, preboot will be disabled.
  • Whoever is doing the deployment will have to wait a few minutes before the new code starts serving user requests; this happens later than it would without preboot (but in the meanwhile, user requests are still served promptly by old dynos).
  • There will be a short period (a minute or two) where heroku ps shows the status of the new code, but user requests are still being served by old code.

There is much more information about it, so refer to their documentation.

Community
  • 1
  • 1
Michael van Rooijen
  • 6,512
  • 4
  • 34
  • 32
2

No - this is currently not possible using Unicorn on Heroku cedar. I've been bugging Heroku about this for weeks.

Here was Heroku Support's reply to my email on March 8, 2012:

Hi, you could enable maintenance mode when doing a deploy, at least your users would see a maintenance page instead of an error, and also request queue wouldn't build up.

We're definitely aware this is a pain and we're working to offer rolling / zero-downtime deploys in the future. We have no ETA to announce, though.

Jeremy Haile
  • 136
  • 6
2

It is possible, but requires a fair amount of forward planning. As of Rails 3.1 there's three tasks that need carrying out

  • Upload the new code
  • Run any database migrations
  • Sync the assets

Uploading code and restarting is fairly straightforward, the main problem lies with the other two, but the way round them is the pretty much the same.

Essentially you need to:

  • Make the code compatible with the migration you need to run
  • Run the migration, and remove any code written specifically for it

For instance, if you want to remove a column, you’ll need to deploy a patch telling ActiveRecord to ignore it first. Only then you can deploy the migration, and clean up that patch.

In short, you need to consider your database and the code compatability an work around them so that the two can overlap in terms of versioning.

An alternative to this method might be to have two versions of the application running on Heroku at the same time. When you deploy, switch the domain to the other version, do the deploy, and switch it back again. This will help in most instances, but again, database compat is an issue.

Personally, I would say that if your deployments are significant to require this sort of consideration, taking parts of the application offline are probably the safest answer. By breaking up an application into several smaller applications can help mitigate this and is a mechanism that I use regularly.

Neil Middleton
  • 21,690
  • 17
  • 76
  • 129
  • I'm aware of the database migration issue, I'm more interested in the details of sending the proper signals to Unicorn on Heroku to restart its process (ie as the article describes) rather than doing a straight up restart of all of the dynos, as well as controlling when the migrations take place (ie before or after restart) – brad Dec 06 '11 at 15:13
  • Restart always needs to be post migration otherwise AR will not load the correct schema. As for restarts, you're pretty much all or nothing. The only zero-downtime answer would be two applications sharing one database and restarting independently. – Neil Middleton Dec 06 '11 at 17:54
  • Not true actually. As far as I know, Heroku restarts automatically after a push. It's up to the user to manually migrate (which happens after). I can only assume the schema.rb file gets loaded on first request or something? Anyway, this is part of my question, can one inject tasks in a Heroku deploy. I'll re-word it. – brad Dec 06 '11 at 18:15
  • Yes, restarts do occur post push - but you need to restart the application post migration if the app has already loaded the old one (from users hitting the application) – Neil Middleton Dec 07 '11 at 09:39
  • right, so part of my question involves that, can i restart *after* migrating rather than before. I know I can manually do it, I guess I'm looking for more of an automated way (ie some config that allows me to control when restarts occur) – brad Dec 07 '11 at 21:20
  • Using Heroku_san (the rubygem) will allow you to set a post_deploy hook for executing additional tasks. I, for instance, use a pre-deploy for flipping maintenance on, and a post for syncing assets and restarting... – Neil Middleton Dec 07 '11 at 23:08