Your first A/B test with Rails

If you are not yet familiarized with the A/B tests, let me introduce you to this topic in few sentences. While unit, integration, or acceptance tests ensure that your application behaves as you expect it, A/B tests help you to choose which version of the element in your application works better.

For example, if you have a mailing list and wondered what text you should put on the subscribe button to encourage more people to click on it, A/B tests are your friend. They will help you to test different options and choose the one that your users liked the most. How does it work in practice? It’s simple:

  • You prepare two versions of buttons, let’s say “Click me” and “Find out more”
  • You use one of the Ruby gems to run A/B tests (I will show you which ones are the best) for a given time; a week or a month
  • After the time, you check which button was clicked more often
  • The Ruby gem for A/B tests takes care of showing different buttons for different users
  • You select the winner and update the button text; you can now test another element to improve the user experience

A/B tests are robust. They help you check what works better for your app to increase the good things you care about. If it sounds interesting, let me show you how you can run A/B tests in your Rails application.

Small Rails test app

Before showing you how you can implement A/B tests in a Rails application, I have to create one. It will be a straightforward app with two actions: index and perform_action. The index view will contain a button that will lead to the perform_action view after the user clicks it. Let’s begin the development:

# generate new rails application
rails new abtest
cd abtest/

# create database
rake db:create

# create files for controller and views
touch app/controllers/home_controller.rb
mkdir app/views/home/
touch app/views/home/index.html.erb
touch app/views/home/perform_action.html.erb

The next step is to create a simple definition of HomeController:

class HomeController < ApplicationController
  def index; end
 
  def perform_action;end
end

Update config/routes.rb file to enable routes:

Rails.application.routes.draw do
  root to: 'home#index'
  get '/perform_action' => 'home#perform_action', as: :perform_action
end

Implement the body for app/views/home/index.html.erb view:

<div style='text-align:center;'>
 <button>
   <%= link_to('Perform action', :perform_action) %>
 </button>
</div>

And the thank you message in app/views/home/perform_action.html.erb:

Action performed, thank you!

That’s it! You can now run the rails s command to start the server, and we are ready to install the first gem and see A/B tests in action.

The Split gem

I really like this gem because it’s simple, and it does its work well. As usually in such cases, let’s start by adding the gem to our application:

bundle add split

Database adapter configuration

By default, you can use Redis or Cookies adapter to store the information about experiments. You can also implement your adapter; to make it work, just follow the same API design as Split::Persistence::CookieAdapter.

The common step of every adapter is the initial configuration that requires from us the initializer file creation:

touch config/initializers/split.rb

Redis adapter configuration

Split.configure do |config|
  config.allow_multiple_experiments = true
  config.redis = Redis.new
end

Cookies adapter configuration

Split.configure do |config|
  config.persistence = :cookie
  config.persistence_cookie_length = 2592000 # 30 days
end

In this article, I’m going to use the Redis adapter.

First A/B test

We finished the configuration part so we can add our first test. Our goal is to check which of the following button titles would encourage the user to click more: “Click me” or “See something awesome”.

Starting the experiment

The first step is to start the experiment in the controller by setting a name of the experiment and available options:

class HomeController < ApplicationController
  def index
    @ab_test = ab_test(:button_test, 'control', 'variant')
  end
 
  def perform_action;end
end

You may see other approaches, but I usually test two things and name them control and variant. In our case, the control is the old approach, the button with the “Click me” name. The variant is the alternative approach that might be better than the old one; we want to find this out.

In the view, we have to simply check which experiment type we are running and show the proper option:

<div style='text-align:center;'>
  <button>
    <% link_text = @ab_test == 'control' ? 'Click me' : 'See something awesome' %>
    <%= link_to(link_text, :perform_action) %>
  </button>
</div>

As you probably noticed, when you start the experiment with the ab_test method, you get the experiment option in return. The first step is completed: we started the investigation.

Finishing the experiment

The second step is to finish the experiment. If the user would click on the link, we have to complete the experiment, so the Split gem knows if the experiment with the given option was successful or not:

class HomeController < ApplicationController
  def index
    @ab_test = ab_test(:button_test, 'control', 'variant')
  end
 
  def perform_action
    ab_finished(:button_test, reset: false)
  end
end

To finish the experiment, you have to call the ab_finished method. The reset option indicates if the same user can start the experiment multiple times.

If you would call ab_finished without starting the experiment, nothing will happen. You don’t have to worry that someone would access the action where you finish the experiment without visiting the page where you are starting it.

Viewing the experiment results

We started and finished the experiment; now it’s time to view the result. The split gem provides a simple but yet meaningful dashboard to view the experiment results.

To activate it, update the config/routes.rb file and mount the dashboard’s engine:

require 'split/dashboard'
 
mount Split::Dashboard, at: 'split'

You can visit http://localhost:3000/split address. You would see a friendly dashboard where you can check the results for a given experiment, delete the whole experiment, or even force the current user to see a specific experiment variant.

Securing the dashboard in the production environment

We want to run experiments in the production environment, but we don’t want to show the results to anyone who is not authorized to see them. Hopefully, we can quickly secure the dashboard with a simple password and username.

Open the config/initializers/split.rb file and add the following code:

Split::Dashboard.use Rack::Auth::Basic do |username, password|
  ActiveSupport::SecurityUtils.secure_compare(
    ::Digest::SHA256.hexdigest(username),
    ::Digest::SHA256.hexdigest(ENV.fetch("SPLIT_USERNAME"))
  ) & ActiveSupport::SecurityUtils.secure_compare(
    ::Digest::SHA256.hexdigest(password),
    ::Digest::SHA256.hexdigest(ENV.fetch("SPLIT_PASSWORD"))
  )
end

Now, simply set the SPLIT_USERNAME and SPLIT_PASSWORD environment variables on the server to secure the dashboard.

The Field test gem

The split gem is the most popular Ruby gem for performing A/B tests, but we also have one alternative that it’s worth considering. The field test gem is slightly different because it offers the database storage and experiment configuration in YAML format. Let’s add it to our gemfile:

bundle add field_test

The gem provides only one adapter so let’s install it and migrate the database:

rails generate field_test:install
rails db:migrate

First A/B test

I will repeat the same test we did with the Split gem. We will check which button text is more attractive to the user: “Click me” or “See something awesome”.

Starting the experiment

The first step is to update the config/field_test.yml and add the experiment name and variants:

experiments:
  button_text:
    variants:
      - control
      - variant

I followed the same schema as in the Split’s gem case; control is for the old button name and variant for the alternative text.

The next step is to update the controller so the experiment can be started as soon as the controller’s action is triggered:

class HomeController < ApplicationController
  def index
    @ab_test = field_test(:button_text)
  end
 
  def perform_action; end
end

The view will be the same as for the version with the Split gem:

<div style='text-align:center;'>
  <button>
    <% link_text = @ab_test == 'control' ? 'Click me' : 'See something awesome' %>
    <%= link_to(link_text, :perform_action) %>
  </button>
</div>

Finishing the experiment

We can finish the test with the field_test_converted method:

class HomeController < ApplicationController
  def index
    @ab_test = field_test(:button_text)
  end
 
  def perform_action
    field_test_converted(:button_text)
  end
end

The experiment is now finished, and we can check the results.

Viewing the experiment results

The field test gem also provides a simple dashboard where you can check the experiment statistics. It also allows you to edit the current user’s variant. To add the dashboard, open the config/routes.rb file and add the following line:

mount FieldTest::Engine, at: "field_test"

Now you can access http://localhost:3000/field_test and check the results of our experiment.

Securing the dashboard in the production environment

If you want to secure the dashboard, you can do this by setting the FIELD_TEST_USERNAME and FIELD_TEST_PASSWORD environment variables. Once they would be present, The gem would automatically secure the dashboard.

Next steps

The next steps are simple: test, test, and test! Then select the winning variant, apply the change and repeat the process for the next element unless you would be happy with the conversion or actions in your application.