How ActiveJob actually works under the hood and how it processes arguments and serializes them? This question appeared in one Ruby telegram chat recently. The first question was "How to pass object to job and not making sql request in perform method?". Which led me to dig into ActiveJob code and find out how it actually works.
From the readme:
Active Job is a framework for declaring jobs and making them run on a variety of queuing backends
...
And you'll be able to switch between them without having to rewrite your jobs.
ZMQ is an universal messaging library which gives you socket-based transport with different types: N-N, fan-out, pub-sub, request-reply and etc. It really fast and small but meantime it has very powerful routing mechanisms.
Really, why not to use default choice: Sidekiq?!
Yes, usually Sidekiq is enough, but what if:
* we have two different backends has been written on two different languages: producer on Ruby and consumer on Python?
* high throughput and no waiting time is crucial? (sidekiq has poll_interval by default 15s which means it checking queues every 15s)
* advanced data serialization/deserialization or maybe message has a binary nature (images, sounds, etc)
* microservices, but not so much time/money to maintain kafka/rabbit cluster
We will start from a small ruby gem which will be adapter for the AJ. All we need to do is implement two class methods enqueue and enqueue_at. Then we will do some custom serialization/deserialization and that's probably all for now.
First thing first: let's make a gem and name it like sidediq pretty unique, I think. (https://github.com/fobiasmog/sidediq)
We'll take boilerplate code from the AJ repo rails/activejob/lib/active_job/queue_adapters/abstract_adapter.rb
Here is our adapter
require "sidediq/version"
require "sidediq/railtie"
require "active_job"
require "sidediq/job"
module ActiveJob
module QueueAdapters
class SidediqAdapter
def enqueue(job)
Sidediq::Job.perform(job.serialize)
end
def enqueue_at(job, timestamp)
Sidediq::Job.perform(job.serialize, at: timestamp)
end
end
end
end
Nothing special, just calling our module class methods
And our job "performer" or client module. Let's make it very straightforward for now (and server code as well)
# sidediq/lib/sidediq/job.rb
require 'ffi-rzmq'
module Sidediq
class Job
class << self
def perform(job_data, at: nil)
context = ZMQ::Context.new
socket = context.socket(ZMQ::PUSH)
socket.connect("tcp://127.0.0.1:5555")
socket.send_string(job_data)
res = socket.recv_string(reply, ZMQ::DONTWAIT)
puts "Client received: #{res} bytes"
socket.close
context.terminate
end
end
end
end
#!/usr/bin/env ruby
# sidediq/bin/sidediq
require 'ffi-rzmq'
context = ZMQ::Context.new
socket = context.socket(ZMQ::PULL)
socket.bind("tcp://127.0.0.1:5555")
print "Server is running and waiting for messages...\n"
loop do
msg = ""
socket.recv_string(msg)
puts "Server received: #{msg}"
end
print "Server shutting down...\n"
socket.close
context.terminate
But we need to be able to do some useful work and send some meaningful payload to our job, let's make some TODO list
* [ ] Write our first "real" job
* [ ] Make protobuf file and serealize it (and deserialize on the server side)
* [ ] Make our server to execute job with deserialized job data
First thing first: gem install sidediq
Don't forget to add your adapter in rails configuration
config.active_job.queue_adapter = :sidediq
Take a brief look on rails/activejob/lib/active_job/railtie.rb (initializer "active_job.set_configs" do |app|) - you will see that during startup it takes your value and tries to assign it to queue_adapter= setter (another file rails/activejob/lib/active_job/queue_adapter.rb#queue_adapter=).
So your adapter (ActiveJob::QueueAdapters.lookup(name_or_adapter).new) will be found because of previous monkey patching
And our first job
class PublishJob < ActiveJob::Base
queue_as :default
def perform(model)
puts "Publishing job"
end
end
Pretty simple, right? Let's debug it.
bundle exec sidediqj = PublishJob.perform_later(ModelA.first)Let's see what's inside j.serialize
{
"job_class" => "PublishJob",
"job_id" => "d1fe0782-2296-451c-b33a-e23b100c8507",
"provider_job_id" => nil,
"queue_name" => "default",
"priority" => nil,
"arguments" => [{"_aj_globalid" => "gid://job-tests/ModelA/1"}],
"executions" => 0,
"exception_executions" => {},
"locale" => "en",
"timezone" => "UTC",
"enqueued_at" => "2026-01-13T21:27:20.785045000Z",
"scheduled_at" => nil
}
arguments - is what we actually passing to our job as args (serialized version, by default AJ serializer)
A lot of data will be passed through a network (in my case it 400 bytes, but imagine if we're sending some json structure, it could be 10-100kb! Might be a case when you're sending your model to a different service, with different data structure or even different framework/language/db)
First of all we need to add protobuf gem and create our proto files
gem install google-protobuf
syntax = "proto3";
package schema_pb;
message ModelA {
string name = 1;
}
and generate a proto file
protoc --proto_path=app/protos/ --ruby_out=app/protos/ app/protos/schema.proto
then create end encode our first message
msg = SchemaPb::ModelA.new(name: ModelA.first.name)
=> <SchemaPb::ModelA: name: "Hello">
SchemaPb::ModelA.encode(msg)
=> "\n\x05Hello"
Another ruby (and not only) documentation around the google protobuf is here
I wanna make (de-)protobuf'isation smooth and almost invisible but leave some space for customization - the answer is DSL.
Let's make a draft of what it should look like:
class PublishJob < ActiveJob::Base
queue_as :default
serializer SchemaPb::ModelA
def perform(model:)
# leave it empty but Python's "pass" looks more natural, imao
end
end
And its again the time for monkey patching:
# sidediq/lib/sidediq.rb
module Sidediq
...
module Serializer
extend ActiveSupport::Concern
module ClassMethods
# will be called during your job's class initialization
def serializer(klass)
self.sidediq_serializer = klass
end
end
included do
class_attribute :sidediq_serializer
end
end
end
module ActiveJob
class Base
include Sidediq::Serializer
def initialize(...)
@serializer_class = self.class.sidediq_serializer
@deserializer_class = @serializer_class
super(...)
end
end
...
end
And don't forget to change enqueue method to be able to process and encode data
module Sidediq
class << self
def enqueue(job)
serialized = job.serialize
Sidediq::Job.perform(serialized)
end
...
...
end
serialized = job.serialize - now will serialize everything including arguments with default AJ serializers, which means all metadata will be passed to consumer as stringified json string, which left the room for improvement for later.
Finally, with this approach we'll be able to pass kwargs (that's why job.arguments.first takes a place) to our job's perform_later class method and send protobuf'ed data to our consumer (zmq "server"). But for now we have to explicitly pass all required proto fields and nothing more (protobuf is pretty strict about incoming data)
PublishJob.perform_later(model: { name: ModelA.first.name })
Now let's start server on a new terminal tab: bundle exec sidediq and perform our job
We will see something like this
Server received: {"job_class" => "PublishJob", "job_id" => "fc0b1b51-a8c2-4a1e-88c8-7be473ec25d7", "provider_job_id" => nil, "queue_name" => "default", "priority" => nil, "arguments" => "\n\a\n\x05Hello", "executions" => 0, "exception_executions" => {}, "locale" => "en", "timezone" => "UTC", "enqueued_at" => "2026-01-25T20:51:03.959181000Z", "scheduled_at" => nil}
"arguments" => "\n\a\n\x05Hello" is our protobuf'ed string
We've sent our message, but that's not the final step yet. Let's run actual job code (what we've implemented in our def perform method)
# sidediq/bin/sidediq
require "json"
require "sidediq/job"
...
loop do
msg = ""
socket.recv_string(msg)
# we're sticking the simple way - incoming msg is json'able string
data = json.load(msg)
Sidediq::Job.execute(job_data)
end
...
Parsed job's metadata and payload should be properly passed to job execution class method ActiveJob::Execution::ClassMethods.execute This method will takes care about callbacks, deserialization and calling our actual perform job instance method. The only one thing which we should take care about is de-protobufization of job['arguments']
And now it is right time to talk a bit about ruby2_keywords_hash
In ruby3 some breaking change has been made: now you have to explicitly expand you kwargs in function definition, which means that this construction
def foo(x:, y:); end
h = {x: 1, y: 2}
foo(h)
is not gonna work anymore (use foo(**h) or foo(x: 1, y: 2) instead)
With AJ the problem is perform(*arguments) this approach is not unwrap you hash object to kwargs unless the object is not ruby2_keywords_hash marked. More details is here
So, that what was possible in ruby2
def perform(foo:, bar:)
end
arguments = [{ foo: 1, bar: 2 }]
perform(*arguments) # ruby3: wrong number of arguments (given 1, expected 0; required keywords: foo, bar) (ArgumentError)
To bypass this:
arguments = [ Hash.ruby2_keywords_hash({ foo: 1, bar: 2 }) ]
With that in mind and the fact that protobuf is not allowing you to use array as top-level element we are making a requirement to use only keywords arguments in job's perform method
But also keep in mind that arguments contains ruby2_keywords_hash'ed last argument (if keyword argument) because ActiveJob::Core#initialize function is already ruby2_keyword'ed.
And after that it will be much clearer why and how it works
# lib/sidediq.rb
module ActiveJob
module Core
...
private
def serialize_arguments(args)
raise NotImplementedError unless serializer_block
serializer_class.encode(
serializer_class.new(*args)
)
end
def deserialize_arguments(arguments)
raise NotImplementedError unless deserializer_class
[
Hash.ruby2_keywords_hash(
deserializer_class.decode(arguments).to_hash
)
]
end
end
end
Simply
PublishJob.perform_later(model: ModelA.first)
Of corse this gem is just for fun and there is a lot of corner cases which I just skipped, and server part is really stupid loop, but that's not the point of this article: main goal is "know your instrument". It's just good to understand, how a parts of the framework are working.