How to set up short polling for computation heavy api requests

In a React/Rails ecosystem

At , our backend is heavily integrated with third party services like Airbnb. Thus, we have many computationally heavy api endpoints that serve our front end React app. In many cases, an user hits a button (to create a listing on Airbnb for instance) and we show a loading spinner until the desired action (creating a listing on Airbnb) has succeeded.

Not what our UI looks like

There are several ways of implementing client-server request response cycle.

  1. Short polling
  2. Long polling
  3. Websockets

In long polling, the client sends a request to the server and waits for a response. The server holds the request and does not return a response until the work is done. In this case, until the listing has been created on Airbnb.

In websockets, a persistent connection between a client and server is established and both parties can use the connection to send data at any time.

In both of these cases, a connection is open until a response has been returned.

Short polling, on the other hand, does not keep the connection open. The client makes a request to the server to start a task (i.e. create Airbnb listing), and a response is returned immediately. Then, the client makes another request to the server to see if the task is done. Based on the status of the task, the client will either wait a few seconds to check status again, or continue execution.

Depending on the web server itself, one method might be better than another. We chose short polling for our Ruby on Rails app hosted with Puma on Heroku. I will get into more details on why at the end of this article.

To make short polling work, we spin up a parallel background job to do the heavy lifting once the request is received for creating a listing on Airbnb. We then immediately free up the server thread to handle other web traffic.


To implement the desired UI behavior with regards to listing creation with short polling, the client needs to:

  1. Dispatch an action to change a variable in the state so that loading spinner is displayed.
  2. Call an api endpoint to queue a background job.
  3. Every few seconds, call another api endpoint to see if that background job is done.
  4. If it’s done, dispatch action to change a variable in the state so that success message is shown.

Thus, in the backend, we need:

  1. Two api endpoints (one for enqueuing job and one for getting job status).
  2. A background job itself.
  3. A way of storing the status of the job.

Front end and back end implementation

Rails comes with Active Job. Active Job is a framework for declaring jobs and making them run on a variety of queuing backends such as Sidekiq and Resque.

is a ruby gem that uses ActiveSupport::Cache to track job status in memory.

redis_uri = URI.parse(ENV["SOME_REDISCLOUD_URL"])
ActiveJobStatus.store = ActiveSupport::Cache::RedisStore.new({
host: redis_uri.host,
port: redis_uri.port,
db: 0,
password: redis_uri.password,
namespace: "cache"
})

To get status of a job, we can do

ActiveJobStatus.fetch(job_id).status

However, this info is only stored in memory for 72 hours. At Pillow, we want a record of all background jobs for tracking and diagnosis purposes. To create a permanent record of background jobs, we set up a table for it in the database.

I. Backend Setup

schema:

create_table "background_jobs", force: :cascade do |t|
t.string "job_name", limit: 255
t.string "job_uid", limit: 255
t.integer "status"
t.datetime "created_at"
t.datetime "updated_at"
t.text "data", default: "{}"
end

route:

resources :background_jobs, only: [:create, :show]

controller:

def create
data = JSON.parse(params[:data])
job_name = params[:job_name]
  queued_job = job_name.constantize.perform_later(data)
  job = BackgroundJob.create(
job_name: job_name,
data: data,
job_uid: queued_job.job_id,
status: ActiveJobStatus.fetch(queued_job.job_id).status
)
  render json: { job_id: job.id }, status: :ok
end
def show
background_job = BackgroundJob.find(params[:id])
  in_memory_status = ActiveJobStatus.fetch(background_job.job_uid).status
  background_job.update(status: in_memory_status)
  render json: { status: in_memory_status }, status: :ok
end

The create method is called to create a record of the background job in the database and queue a background job.

The front end will need to pass whatever data the background job will need to do stuff ALONG WITH A JOB NAME.

Notice how in the create method we had job_name.constantize.perform_later(data). By using constantize, we remove the need to have a bunch of elsif. I.e., we don’t do

job_name = params[:job_name]
if job_name == 'CreateAirbnbListing'
CreateAirbnbListing.perform_later(data)
elsif job_name == 'SomeOtherJob'
SomeOtherJob.perform_later(data)
...

If we don’t pass in a job name at all, we would have to create a different endpoint for each job.

This POST endpoint returns the id of the job that was created so that this id can be used to call the show endpoint to check the status of that background job.

If you don’t care about storing a permanent copy of your background jobs for the record, then you can just render the queued job’s job_id in the create method and use that job_id as params for status lookup in the show method.

The show endpoint returns the status of the job to the front end.

II. Front end set up (short polling implementation)

Briefly, in semi-complete code, this is what the saga for handling background job would look like in our powerpoint example.

import enqueueBackgroundJob from '../background_job'
export default function *createAirbnbListing() {
yield *takeEvery( "CREATE_AIRBNB_LISTING", createListing )
}
function *createListing( action ) {
const { bedroom_count, bathroom_count, beds_count } = action
  // 1. dispatch action to change state and show loading
yield put( showLoading( ) )
  // 2. queue background job to handle text message
const listingIsCreated = yield call(enqueueBackgroundJob, 'CreateAirbnbListing', { bedroom_count, bathroom_count, beds_count })
  // 3. dispatch action to show success (or fail) message
if ( listingIsCreated ) {
yield put( showSuccessMessage( ) )
} else {
yield put( showFailureMessage( ) )
}
}

This below, is the javascript code for implementing short polling.

background_job.jsx

import { call } from 'redux-saga/effects'
import { delay } from './async'
const BACKGROUND_JOBS_BASE_URL = '/api/background_jobs'
const myHeaders = new Headers()
myHeaders.append('Content-Type', 'application/json')
export default function* enqueue( job_name, options ) {
const data = { job_name, data: JSON.stringify(options) }
let backgroundJobId = null
  // 1. create background job and queue job
  fetch( BACKGROUND_JOBS_BASE_URL, {
method: 'POST',
headers: myHeaders,
cache: 'default',
body: JSON.stringify(data)
}).then( response => {
return response.json()
}).then( result => {
backgroundJobId = result['job_id']
})
  // 2. poll status of background job every 3 sec
  // wait for fetch to return backgroundJobId
yield call( delay, 1000 )
  const url = `${BACKGROUND_JOBS_BASE_URL}/${backgroundJobId}`
let i = 0
while ( i < 20) { // arbitrary limit to avoid infinite loop
const finished = yield call( hasJobFinished, url)
if ( finished ) {
return true
}
i++
    // wait 3 seconds before next GET request to pull status
yield call( delay, 3000 )
}
  return false
}
function *hasJobFinished( url ) {
return fetch(url).then( response => {
return response.json()
}).then( result => {
if ( result['status'] === 'completed' ) {
return true
} else {
return false
}
})
}

The delay method returns a Promise that won’t be resolved until the time has passed by putting the resolve method in a setTimeout function.

function* delay( ms, result = ms ) {
return new Promise(resolve => {
setTimeout( () => resolve( result ), ms )
})
}

I did not put error handling in the example above, but that’s something you’d want to do in real life.


Brief note about :

The code above used put and call.

put takes in an action as the parameter. It effectively instructs the middleware to dispatch an action to the Store. It is non-blocking.

call is blocking and will block the saga until it returns. It can take in a generator function or a normal function. If the function returns a Promise, as in the case with delay and hasJobFinished, the generator passed into call is suspended until the Promise resolves.


In summary, what we have set up here is a generic way of enqueuing any background job. One advantage of this set up is maintainability. There is no need to create or change controller actions for new background jobs added.

Short polling vs. Long polling vs. Websockets

Let me now dig more into why we chose short polling.

We are a Ruby application running on Puma server. In Heroku, we set max thread that each process can run to be 1 (i.e. single threaded process). This eliminates the need for our entire application codebase to be . Additionally, we set each server to have 3 separate/concurrent processes. This means that each server can process a max of 3 simultaneous connections. We have 2 separate servers, so if we had chosen long polling, if 6 users are simultaneously trying to create a listing, no one else would be able to access our site. This is not a problem with short polling as the main web server thread is quickly freed up because the listing creation is handled with a queued job.

Rails 4 (unlike Rails 5), does not come with built-in support for websockets. Implementing websockets will introduce more moving pieces. Additionally, setting up websockets with Heroku introduces more uncertainty to the number of websockets connections allowed. Short polling is easier to implement and debug. For our system, real time data isn’t crucial either. Considering these, we chose short polling.

However, there are a few disadvantages too. First, a response won’t be returned to the client immediately once the work is done. Second, and perhaps the bigger problem, is higher traffic to the server itself. This is especially bad if client has slow connection, as the backend thread will be blocked by establishing connection. However, because we have Nginx on top of the Puma app server, this isn’t too much of an issue. With this set up, establishing connections (Nginx) and handling requests (Puma) are separated.

And for completeness sake, there is server push. Server push is a performance feature in HTTP/2. It allows the web server to notify the client when the job is done without client polling. It might be better than short polling, but since we’re not a HTTP/2 app, that’s not an option for us yet.


Lastly, I just want to say that I’m still pretty much a noob when it comes to server infrastructure. Please correct me or give me feedback if you know something I don’t with regards to the content of this article.