A/B Testing Features by Segmenting Users

"A/B test everything."

Those of us in the startup world hear it routinely. But it is a useless statement. Ask "how" and you will be met with the shifting eyes of the ignorant.

Indeed, A/B testing is powerful but tricky. Whole companies have been built around making it easier for others to do and track. But those generally help with marketing pages. How do we A/B test a feature in our app? When we release code, how do we allow access to a subset of our users? Turns out, there is no magic here, just some run of the mill problem solving.

Before we get into the code, let's define an ideal. Our solution should satisfy a number of requirements:

1. Allow us to call user.in_test?( 'test_name' ) anywhere in the code
2. Allow segments to be created on arbitrary requirements
3. Centralize all logic that includes/excludes users from tests
4. Provide a source-of-truth list of actively running tests
5. Be performance-focused

These are all readily achievable with a little upfront infrastructure. Let's start with the database and build up from there.

The tables

Our solution will use two tables to achieve the goals. We will need a table and a join table.

CREATE TABLE segments (
    id          integer NOT NULL,
    name        character varying NOT NULL,
    deleted     boolean DEFAULT false NOT NULL,
    created_at  timestamp without time zone NOT NULL,
    updated_at  timestamp without time zone NOT NULL
);

CREATE TABLE user_segments (
    id          integer NOT NULL,
    user_id     integer NOT NULL,
    name        character varying NOT NULL
);

For performance purposes, we will use segments.name as the foreign key instead of segments.id. This means we can find the names of a user's segments just by looking at the join table.

Segmenting logic

Now that we have tables, we will need the ability to include them in the tests they belong in. This file, let's call it Segmenter, will wrap all of our include/exclude logic for segments.

class Segmenter
  ACTIVE_TESTS = [
    { name: 'green_buy_buttons', function: :segment_green_buy_buttons },
    { name: 'fancy_new_feature', function: :segment_fancy_new_feature },
  ].freeze

  attr_reader :segments, :user

  def initialize( user )
    @user = user

    @segments = {}
    unordered_segments = Segment.all
    unordered_segments.each do |segment|
      @segments[ segment.name ] = segment
    end
  end

  def segment
    include_in_tests = []
    tests_currently_in = UserSegment.where( user_id: user.id ).pluck( :name ).to_set

    ACTIVE_TESTS.each do |test|
      # Create any missing Segment records
      if !segments[ test[:name] ]
        new_segment = Segment.create( name: segment[:name] )
        segments[ new_segment.name ] = new_segment
      end

      # Include this user in tests they qualify for
      if self.send( test[:function] )
        include_in_tests << test[ :name ]
      end
    end

    # Some basic set math to find current, old and new tests
    segments_to_add = include_in_tests.to_set - tests_currently_in
    segments_to_delete = tests_currently_in - include_in_tests.to_set

    # Remove the segments this user no longer belongs in
    UserSegment.where( name: segments_to_delete.to_a, user_id: user.id ).delete_all()

    # Add the user to segments they now are included in
    segments_to_add.each do |new_segment_name|
      UserSegment.create( user_id: user.id, name: new_segment_name )
    end
  end

  private

  def segment_green_buy_buttons
    user.id.even?
  end

  def segment_fancy_new_feature
    user.state == 'NY'
  end
end

This is a bit of code, so let's touch on everything this will achieve for us. First we have the ACTIVE_TESTS array. This will be a list of all tests actively running in our app. Next we have the segment() function. This will be responsible for including a user in any new tests they qualify for as well as removing them from any test they were in but longer exists.

Finally, the private functions are the barriers of entry for each test included in the ACTIVE_TESTS constant. Since we are dynamically calling them with if self.send( test[:function] ), if one exists in the array but not the file, we are going to see explosions in a hurry.

Including users in segments

Now that we have our segmenter, we need to identify when we actually plan to run it. One obvious time is to run it on our users every release. That will be time consuming and unnecessary in most cases, though. We could run it after a user's login, forcing their segments to be updated each time they return to the site.

Your specific app will determine when is the best time to run the segmenter on a user. Remember that you are balancing correctness vs performance. Running it on every request is likely to be too slow, albeit being the most "correct".

Checking test inclusion

Finally we need the ability to trivially determine test inclusion in our code. This will be achieved by a call directly on user.

class User
  has_many :user_segments

  def in_test?( test_name )
    user_segments.pluck( :name ).include?( test_name.to_s )
  end

  # ...
end

With this, we can very easily implement tests in our code.

button_color = 'red'
if current_user.in_test? 'green_buy_buttons'
  button_color = 'green'
end

You get what you emphasize

Remember that we are doing all of this to improve the quality of our product. If we make testing difficult, it just won't happen. We need tests to be so trivial to add and use that we don't ever have an excuse not to use them.

When you begin to segment users into tests, it opens a world of possibilities for seeing the positive and negative effects of our changes on user behavior. This is important because rarely are features strictly positive or negative. Often they will improve some user behavior while degrading it elsewhere in our app. Using tests, we gain visibility into these shifts.

The last reason i would bang the table for tests is it allows you to test major features with minimal risk. Release an enormous feature to the masses is a great way to have 100% of your customers complain about things you didn't think about or catch during testing. Segmenting your users and releasing it to 10% is a far safer proposition to everyone. This let's you adjust your features to be directly in line with customer need, without reducing the overall trust in a feature before it ever had a fighting chance.

tl;dr: A/B testing features is important but difficult. This code helps demystify the process and makes it simple to do.