Building a Rails Notification Queue 3: Queue Processing

This is the third and final piece on creating a notification queue in Rails. If you are just finding this post, i suggest starting with the first post on the database, as it is quite short.

A recap

In the previous two posts we have looked

• the notification database table
• the jsonb datatype and its benefits
• how to schedule notifications
• how to send notifications

The obvious omission here is... what processes the queue itself.

Processing script

In our case, we do all of our scheduled scripts as rake tasks kicked off from cron. I suppose that topic warrants a post as well, so look out for it in the future. Onto the code.

namespace :notifications do
	desc "Process all of the notifications scheduled to be sent by now"
	task :process_queue => :environment do
		# Check for an existing lock. Exit if found.
		# HEROKU USERS: do NOT use this (more below)
		break if File.exist?( "#{Rails.root}/tmp/queue.lock" )

		# No lock exists. Create one.
		File.open("./tmp/queue.lock", "w") {}

		begin
			# Load notifications for this sending batch
			notifications = Notification.unsent.where( 'scheduled_for < ?', Time.now )

			notifications.each do |notification|
				begin
					notification.send_notification()
				rescue StandardError => e
					Rails.logger.error( "Notification Error: #{e.inspect}" )
					Rails.logger.error( e.backtrace )
				end
			end
		ensure
			File.delete("./tmp/queue.lock")
		end
	end
end

There are some interesting bits to this script that we should touch on.

The first of them is the semaphore. A semaphore is a way to ensure only a single copy of a script is running at one time. Because our script loads all unsent messages before it begins to process them, a second instance of this processing script, launched before the first one completes, will load some notifications the first one will process. This semaphore prevents us from sending a notification twice.

To accomplish this, we use the existence of a lock file, which is nothing but an empty file, to control the script's execution. Notice how we put the File.delete in an ensure block to assure it is never skipped.

The next notable portion of code is the query, Notification.unsent.where( 'scheduled_for < ?', Time.now ). The only reason i call attention to this query is the <. This allows us to effectively send notifications in batches. Each batch is every notification scheduled to be sent between the last time the processing script started and the next time it loads unsent notifications.

Process scheduling

Which brings us into the topic of how, and when, the schedule the processing queue. This will vary between implementations based on the number of users on your site and the importance of a notification going out as close as possible to its intended send time.

In our case, we run the script every minute. At this pace, regular app traffic takes the script between 5-30 seconds to process all unsent notifications. 1 minute intervals means our server isn't spending a lot of time querying unnecessarily for unset messages. Also, because none of our notifications are critical with their send minute, having 1 minute batches is not a customer issue.

A note to heroku users

Heroku users need to modify this script. Heroku's scheduler has a lot of oddities, not the least of which is silent script termination. Using the heroku scheduler, scripts that run for long periods of time, 10-15 minutes, are silently terminated. Using a lock file can cause obvious problems in this case. Because of this issue, if you are using heroku as your scheduler, a semaphore is more dangerous than valuable.

Done?!

Yes! Those are all the main pieces of the notification system. As i have mentioned in the previous posts, i urge people to customize this to this specific codebase. Many of the things in the architecture have been wins for my codebases, but that doesn't mean it is perfect for yours as is. Play around with it and find your special sauce.

tl;dr: The scheduling details of a notification script can be easily modified if the processing script itself is robust.