Post-Deployment script on Elastic Beanstalk: Restart Delayed Job

Jan 8, 2015

UPDATE 2021!

There is a newer post for Elastic Beanstalk using the Amazon Linux 2 platform.

Elastic Beanstalk on Amazon Linux 2: Run command or script after deployment

Original Post

I recently moved one of the Rails applications that I manage, from Engineyard to Elastic Beanstalk on Amazon AWS. All in all, it ended up taking a little bit longer than I expected. To get the webapp up and running was not the big issue, but all the things around it, like third party dependencies and background workers, etc.

I wont get in to all the details in this post, but I do want to mention the background worker, which is DelayedJob in our case. It was quite easy to ssh in to the instance and manually start up an instance. But as everyone can guess, that is not a good solution. Image doing that after every deploy...

So after some research online, I found a ton of articles and posts about how other people have managed to do this. Some mentioned the use of container_commands in elasticbeanstalk config files, but if they ever worked, it was a very long time ago. Instead, most people were using the config files to place shell scripts into a post deployment folder. This seemed like the way to go.

I want to quickly mention a few of the posts that I found as I was looking around.

To summarise what seemed to be working for most people, it was to add the following config file in the .ebextensions folder in your app repository

files:
  "/opt/elasticbeanstalk/hooks/appdeploy/post/99_restart_delayed_job.sh":
    mode: "000755"
    owner: root
    group: root
    content: |
      #!/usr/bin/env bash
      . /opt/elasticbeanstalk/support/envvars
      cd $EB_CONFIG_APP_CURRENT
      su -c "RAILS_ENV=production script/delayed_job --pid-dir=$EB_CONFIG_APP_SUPPORT/pids restart" $EB_CONFIG_APP_USER

However, this approach was not working for me with the latest builds of elastic beanstalk.

The first thing I noticed was that the EB_* environment variables where not set when my scripts were executed. After some additional research, I found other posts of people experiencing the same problem, and all of them where only a few months old. So it seemed like something had changed within elastic beanstalk. And after I read the following thread on the aws support forum, I got it verified that these variables had been removed.

AWS Developer Forums: AWS EB – Ruby stacks - containerfiles

But in that thread, a solution was posted explaining how to set these variables manually by calling a get-config script. It took me some time to actually try it in my script though. Initially I tried calling it in the console from a ssh session, and that did not work. It said that a particular gem was not present. So I though that this method had been removed as well. But once I actually tried it in my script, it actually worked fine. Here is an example syntax:

EB_APP_CURRENT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_deploy_dir)

So now I had the variables available for use, but the script was still failing. When it was trying to execute the bundle command, it returned the following error message:

Could not find rake-10.1.1 in any of the sources (Bundler::GemNotFound)

And here is where I spent most of my time trying to get it to work. I realised that it was not using the correct ruby version when executing the script, and hence the gems that bundler tried to load were not installed. But I didn't know what to do about it. I tried changing the PATH to point to other locations but it just created a bunch of other errors.

But in the end, I found the solution by looking at the pre deployment scripts that is part of the elastic beanstalk defaults. In particular the script 

/opt/elasticbeanstalk/hooks/appdeploy/pre/10_bundle_install.sh

In that file, they call a script called use-app-ruby.sh. That sounded very promising to me. And once I implemented it in my script, the entire thing ran successfully.

Here is the result of it all:

files:
  "/opt/elasticbeanstalk/hooks/appdeploy/post/99_restart_delayed_job.sh":
    mode: "000755"
    owner: root
    group: root
    content: |
      #!/usr/bin/env bash
      # Using similar syntax as the appdeploy pre hooks that is managed by AWS
      # Loading environment data
      EB_SCRIPT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k script_dir)
      EB_SUPPORT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k support_dir)
      EB_APP_USER=$(/opt/elasticbeanstalk/bin/get-config container -k app_user)
      EB_APP_CURRENT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_deploy_dir)
      EB_APP_PIDS_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_pid_dir)

      # Setting up correct environment and ruby version so that bundle can load all gems
      . $EB_SUPPORT_DIR/envvars
      . $EB_SCRIPT_DIR/use-app-ruby.sh

      # Now we can do the actual restart of the worker. Make sure to have double quotes when using env vars in the command.
      # For Rails 4, replace script/delayed_job with bin/delayed_job
      cd $EB_APP_CURRENT_DIR
      su -s /bin/bash -c "bundle exec script/delayed_job --pid-dir=$EB_APP_PIDS_DIR restart" $EB_APP_USER
Tags: , , ,