Migrating from Resque to Sidekiq

source

From Forks to Threads

The journey from Resque to Sidekiq is fundamentally about exchanging process isolation for memory efficiency—trading Unix forks for Ruby threads, gaining order-of-magnitude throughput improvements while accepting the discipline of thread-safe code.

Migrating from Resque to Sidekiq represents a shift in architectural philosophy for background job processing in Ruby. While both systems use Redis as their backend, they take radically different approaches to Concurrency—Resque forks processes for job isolation, while Sidekiq uses threads for efficiency.

Architecture Comparison

The fundamental difference lies in how each system achieves isolation and concurrency.

Resque: Process-Based Isolation

Resque’s architecture embodies Unix philosophy—fork a child process for each job, execute it in isolation, and let the operating system clean up resources when the process exits.

graph TB
    subgraph "Resque Architecture"
        Parent[Parent Worker Process]
        Redis[(Redis)]

        Parent -->|polls queue| Redis
        Parent -->|forks| Child1[Child Process 1<br/>Job Execution]
        Parent -->|forks| Child2[Child Process 2<br/>Job Execution]
        Parent -->|forks| Child3[Child Process 3<br/>Job Execution]

        Child1 -->|exits| OS[OS Memory Cleanup]
        Child2 -->|exits| OS
        Child3 -->|exits| OS
    end

    style Parent fill:#e1f5ff
    style Child1 fill:#fff4e6
    style Child2 fill:#fff4e6
    style Child3 fill:#fff4e6

Resque's Hidden Tax

Resque pays ~50% CPU overhead on forking and process scheduling—your jobs run fast, but half your processor cycles go to the operating system just managing isolation mechanics.

Key Characteristics:

Process per job: Each job runs in a forked child process
Memory isolation: Jobs cannot interfere with each other’s memory
Automatic cleanup: OS reclaims all memory when process exits
CPU overhead: ~50% CPU spent on forking and process scheduling
Memory cost: 50-100MB per worker process
Crash resilience: One job crashing doesn’t affect the parent worker

Sidekiq: Thread-Based Efficiency

Sidekiq uses threads within a single process, leveraging O-bound workload patterns to achieve high concurrency despite Ruby’s Global Interpreter Lock.

graph TB
    subgraph "Sidekiq Architecture"
        Process[Sidekiq Process<br/>~125MB Memory]
        Redis[(Redis)]

        Process -->|thread 1| Job1[Job 1 Executing]
        Process -->|thread 2| Job2[Job 2 Waiting on I/O]
        Process -->|thread 3| Job3[Job 3 Executing]
        Process -->|thread 4-10| More[7 more threads...]

        Redis -->|BRPOP| Process

        Job2 -->|I/O wait releases GIL| GIL[Global Interpreter Lock]
        Job1 -->|holds| GIL
        Job3 -->|queued for| GIL
    end

    style Process fill:#e1f5ff
    style Job1 fill:#c8e6c9
    style Job2 fill:#fff9c4
    style Job3 fill:#c8e6c9

Key Characteristics:

Threads per process: 10-30 threads (configurable) in one Ruby process
Thread safety required: All job code must be thread-safe
Memory efficiency: ~125MB for 10 concurrent jobs vs 750MB+ for Resque
GIL advantage: Threads multiply throughput for I/O-bound jobs
Lower overhead: Minimal context switching, no fork() overhead
Shared state risk: Bugs can affect multiple jobs in same process

Performance Comparison

The Memory Paradox

A single 125MB Sidekiq process running 10 threads can outperform 10 separate 75MB Resque processes (750MB total)—achieving better throughput with 6x less memory by exploiting I/O wait time rather than fighting it.

The performance differences are dramatic, especially at scale.

Memory Usage

Resque (100 concurrent jobs):

100 worker processes × 75MB = 7.5GB memory

Sidekiq (100 concurrent jobs):

10 processes × 10 threads × 125MB = 1.25GB memory

Throughput Benchmarks

Real-world benchmark processing 150,000 jobs:

System	Configuration	Time	Jobs/Second
Resque	1 worker	396s	379
Sidekiq (MRI)	1 process, 50 threads	312s	481
Sidekiq (MRI)	3 processes, 50 threads each	120s	1,250

Key Insight: Sidekiq achieves 3.3x throughput improvement with proper configuration due to eliminated forking overhead and efficient thread scheduling.

The I/O Advantage

The GIL's Unexpected Gift

Ruby’s Global Interpreter Lock—normally seen as a threading liability—becomes an asset in Sidekiq: threads automatically release it during I/O waits, enabling 10 threads to process 100 concurrent jobs if each job is 90% waiting.

The Sidekiq Concurrency Model excels when jobs spend time waiting on external services:

sequenceDiagram
    participant T1 as Thread 1
    participant GIL as Global Interpreter Lock
    participant T2 as Thread 2
    participant DB as Database

    Note over T1,T2: I/O-Bound Job Pattern

    T1->>GIL: Acquire GIL
    T1->>T1: Execute Ruby code (2ms)
    T1->>DB: Query database
    Note over T1: Releases GIL during I/O

    GIL->>T2: T2 acquires GIL
    T2->>T2: Execute Ruby code (2ms)
    T2->>DB: HTTP API call
    Note over T2: Releases GIL during I/O

    DB-->>T1: Response (50ms later)
    T1->>GIL: Re-acquire GIL
    T1->>T1: Process response (1ms)

    DB-->>T2: Response (100ms later)
    T2->>GIL: Re-acquire GIL
    T2->>T2: Process response (1ms)

During I/O waits (database queries, HTTP requests, file operations), threads release the GIL, allowing other threads to execute Ruby code. This “virtual Parallelism” means 10 threads can process 100 jobs simultaneously if each job spends 90% of its time in I/O.

API and Code Comparison

Sidekiq maintains API compatibility with Resque for most common patterns, making migration straightforward.

Job Definition

Resque:

class ImageProcessor
  @queue = :images
 
  def self.perform(image_id)
    image = Image.find(image_id)
    image.process!
  end
end

Sidekiq:

class ImageProcessor
  include Sidekiq::Job
 
  def perform(image_id)
    image = Image.find(image_id)
    image.process!
  end
end

Key Differences:

Sidekiq uses include Sidekiq::Job instead of class instance variable
No self. prefix on perform method
Queue specified in sidekiq_options instead of @queue

Enqueuing Jobs

Both Systems:

# Immediate execution
ImageProcessor.perform_async(123)
 
# Scheduled execution
ImageProcessor.perform_in(1.hour, 123)
ImageProcessor.perform_at(tomorrow, 123)

This compatibility is intentional—Sidekiq was designed as a drop-in replacement for the most common Resque patterns.

Configuration

Resque Configuration:

# config/initializers/resque.rb
Resque.redis = Redis.new(url: ENV['REDIS_URL'])
Resque.redis.namespace = "resque:myapp"
 
Resque.before_fork do |job|
  ActiveRecord::Base.connection.disconnect!
end
 
Resque.after_fork do |job|
  ActiveRecord::Base.establish_connection
end

Sidekiq Configuration:

# config/initializers/sidekiq.rb
Sidekiq.configure_server do |config|
  config.redis = { url: ENV['REDIS_URL'], namespace: 'sidekiq' }
 
  config.on(:startup) do
    # Runs once when Sidekiq starts
    # No need for connection management - threads share connection pool
  end
end
 
Sidekiq.configure_client do |config|
  config.redis = { url: ENV['REDIS_URL'], namespace: 'sidekiq' }
end

Fork's Invisible Trap

Forked processes inherit their parent’s database connections—but those connections are now invalid, forcing every Resque worker to disconnect and reconnect on every job. Sidekiq’s threads sidestep this entirely with native connection pooling.

Critical Difference: Resque needs database reconnection logic because each fork gets a copy of parent’s database connection (which is invalid). Sidekiq’s threads automatically use connection pooling, eliminating this complexity.

Hooks and Middleware

Resque Hooks:

class MyJob
  @queue = :default
 
  def self.before_enqueue(*args)
    # Return false to cancel job
  end
 
  def self.after_perform(*args)
    # Cleanup after job
  end
 
  def self.on_failure(exception, *args)
    # Handle failures
  end
end

Sidekiq Middleware:

class MyMiddleware
  def call(worker, job, queue)
    puts "Before job: #{job['class']}"
    yield  # Execute the job
    puts "After job: #{job['class']}"
  rescue => e
    puts "Job failed: #{e.message}"
    raise
  end
end
 
Sidekiq.configure_server do |config|
  config.server_middleware do |chain|
    chain.add MyMiddleware
  end
end

Sidekiq’s middleware approach is more flexible—you can inject logic at multiple points, modify job arguments, wrap execution with timing/logging, and handle cross-cutting concerns centrally.

Redis Data Structure Differences

Both systems use Redis, but with different patterns reflecting their architectural differences.

Resque Redis Keys

queue:images              # List - job payloads
queues                    # Set - all queue names
workers                   # Set - active worker IDs
worker:<id>               # String - current job for worker
worker:<id>:started       # String - worker start timestamp
workers:heartbeat         # Hash - worker heartbeat timestamps
stat:processed            # String - total processed counter
stat:failed               # String - total failed counter
failed                    # List - failed job records

Sidekiq Redis Keys

queue:images              # List - job payloads (same as Resque)
queues                    # Set - queue names (same as Resque)
schedule                  # Sorted Set - scheduled jobs (timestamp score)
retry                     # Sorted Set - jobs to retry (timestamp score)
dead                      # Sorted Set - permanently failed jobs
processes                 # Set - active process IDs
processes:<id>            # Hash - process metadata
processes:<id>:work       # Hash - in-flight jobs for process
stat:processed            # String - processed counter
stat:processed:YYYY-MM-DD # String - daily processed (auto-expire)
stat:failed               # String - failed counter
stat:failed:YYYY-MM-DD    # String - daily failed (auto-expire)

Key Innovations in Sidekiq:

Sorted Sets for Scheduling: Jobs become self-scheduling using timestamp scores—no separate poller infrastructure needed beyond the built-in Poller
Process Metadata: Detailed process health monitoring (memory, latency, busy threads)
Dead Set: Capped at 10,000 jobs with 6-month retention, preventing unbounded growth
Time-Series Statistics: Daily granularity with automatic TTL expiration

Migration Strategy

A methodical, incremental approach minimizes risk and allows learning from early jobs.

Phase 1: Preparation and Setup

1. Install Sidekiq Alongside Resque

# Gemfile
gem 'resque'           # Keep existing
gem 'sidekiq'          # Add new
gem 'sidekiq-cron'     # If using resque-scheduler

2. Configure Sidekiq with Different Namespace

# config/initializers/sidekiq.rb
Sidekiq.configure_server do |config|
  config.redis = {
    url: ENV['REDIS_URL'],
    namespace: 'sidekiq'  # Separate from 'resque' namespace
  }
end

This allows both systems to coexist without interfering.

3. Set Up Separate Sidekiq Queues

# config/sidekiq.yml
:queues:
  - critical_sidekiq
  - default_sidekiq
  - low_sidekiq

Name them distinctly to prevent worker misconfiguration.

Phase 2: Pilot Migration

1. Choose Low-Risk Jobs First

Ideal candidates:

✅ Simple, stateless jobs
✅ No shared state or global variables
✅ Pure I/O-bound operations (API calls, file uploads)
✅ Low traffic volume
❌ Avoid: Complex state machines, high-traffic jobs, CPU-intensive work

2. Convert Job Class

# Before: Resque
class EmailNotificationJob
  @queue = :mailers
 
  def self.perform(user_id, template)
    user = User.find(user_id)
    Mailer.send_notification(user, template)
  end
end
 
# After: Sidekiq
class EmailNotificationJob
  include Sidekiq::Job
  sidekiq_options queue: :mailers_sidekiq, retry: 5
 
  def perform(user_id, template)
    user = User.find(user_id)
    Mailer.send_notification(user, template)
  end
end

3. Update Callers Gradually

Use feature flags or gradual rollout:

def send_notification(user_id, template)
  if sidekiq_enabled_for_notifications?
    EmailNotificationJob.perform_async(user_id, template)
  else
    Resque.enqueue(EmailNotificationJob, user_id, template)
  end
end

4. Monitor Closely

Watch for:

Memory usage patterns
Error rates
Job execution times
Redis connection pool exhaustion
Thread safety issues

Phase 3: Identify and Fix Thread Safety Issues

Migration's Real Cost

Thread safety issues are rarely caught in testing—they manifest as intermittent production bugs that disappear when you try to debug them, making the migration’s biggest challenge invisible until it’s already deployed.

The biggest challenge in migration is ensuring thread safety.

Common Thread Safety Problems:

1. Class Variables and Constants Mutation

# UNSAFE - Class variable shared across threads
class ImageProcessor
  @@processed_count = 0
 
  def perform(image_id)
    @@processed_count += 1  # Race condition!
  end
end
 
# SAFE - Use thread-local storage or remove shared state
class ImageProcessor
  def perform(image_id)
    # No shared state
    process_image(image_id)
  end
end

2. Instance Variables in Singleton Objects

# UNSAFE - Singleton with instance variable
class ImageService
  include Singleton
 
  def process(image_id)
    @current_image = Image.find(image_id)  # Race condition!
    apply_filters(@current_image)
  end
end
 
# SAFE - Pass state through method arguments
class ImageService
  include Singleton
 
  def process(image_id)
    image = Image.find(image_id)
    apply_filters(image)
  end
end

3. Non-Thread-Safe Gems

Some older gems aren’t thread-safe. Check documentation or GitHub issues:

# Example: net-sftp had thread safety issues in older versions
# Solution: Upgrade gem or add mutex protection
 
class SftpUploader
  UPLOAD_MUTEX = Mutex.new
 
  def perform(file_path)
    UPLOAD_MUTEX.synchronize do
      Net::SFTP.start(...) do |sftp|
        sftp.upload!(file_path, remote_path)
      end
    end
  end
end

4. Rails AutoLoading in Threads

Rails autoloading isn’t thread-safe in development/test:

# config/environments/development.rb
config.eager_load = true  # Force eager loading in development too

Phase 4: Memory Optimization

Some jobs that worked fine in Resque may cause memory issues in Sidekiq’s threaded environment.

Problem: Job allocates large objects that accumulate across threads

# MEMORY HUNGRY
class DataExporter
  def perform(user_id)
    @user = User.find(user_id)
    @records = @user.records.includes(:everything).all  # Loads 100k records
    csv = generate_csv(@records)  # Creates huge string
    upload_to_s3(csv)
  end
end

Solution: Process in batches, stream data, or use separate process

# MEMORY EFFICIENT
class DataExporter
  def perform(user_id)
    user = User.find(user_id)
 
    # Stream records in batches
    user.records.find_each(batch_size: 1000) do |batch|
      csv_chunk = generate_csv_chunk(batch)
      upload_chunk_to_s3(csv_chunk)
    end
  end
end

Phase 5: Gradual Queue Migration

Migrate jobs in batches, allowing each batch to “bake” in production.

Week 1: Low-traffic, simple jobs (emails, notifications) Week 2: Medium-complexity jobs (reports, data processing) Week 3: High-traffic jobs (critical path operations) Week 4+: Complex jobs (multi-step workflows, integrations)

Phase 6: Optimize Configuration

Once most jobs migrated, tune Sidekiq for your workload.

Concurrency Tuning:

# config/sidekiq.yml
 
# For I/O-heavy workloads (APIs, databases)
:concurrency: 25
 
# For CPU-heavy workloads (image processing, data transformation)
:concurrency: 5
 
# For mixed workloads
:concurrency: 10  # Default

Redis Connection Pool:

Sidekiq.configure_server do |config|
  config.redis = {
    url: ENV['REDIS_URL'],
    size: config.options[:concurrency] + 2  # Minimum pool size
  }
end

Rails Database Connection Pool:

# config/database.yml
production:
  pool: <%= ENV.fetch("RAILS_MAX_THREADS", 10).to_i + 5 %>

Must be ≥ Sidekiq concurrency to avoid connection exhaustion.

Phase 7: Decommission Resque

Once all jobs migrated and stable:

Stop Resque workers (let existing jobs drain)
Remove Resque gem and configuration
Migrate Redis namespace (optional - consolidate keys)
Remove deployment configuration for Resque workers
Update monitoring/alerting to track only Sidekiq

Common Pitfalls and Solutions

Pitfall 1: Connection Pool Exhaustion

Symptom: ActiveRecord::ConnectionTimeoutError or “could not obtain a connection from the pool”

Cause: More Sidekiq threads than database connections

Solution:

# config/database.yml
pool: <%= ENV.fetch("RAILS_MAX_THREADS", 10).to_i + 5 %>
 
# Or use connection_pool gem for fine control
require 'connection_pool'
 
DB = ConnectionPool.new(size: 15, timeout: 5) do
  Sequel.connect(ENV['DATABASE_URL'])
end

Pitfall 2: Memory Bloat

Symptom: Sidekiq processes grow to multiple GB over time

Cause: Memory leaks accumulate across threads

Solutions:

Enable jemalloc (better memory allocator):

# Dockerfile
RUN apt-get install -y libjemalloc2
ENV LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2

Automatic process restart:

# config/initializers/sidekiq.rb
Sidekiq.configure_server do |config|
  config.death_handlers << ->(job, ex) do
    # Restart process after 1000 jobs or 1 hour
    if total_jobs > 1000 || uptime > 3600
      Process.kill('TERM', Process.pid)
    end
  end
end

Profile memory usage:

gem 'memory_profiler'
 
# In job
MemoryProfiler.report do
  perform_work
end.pretty_print

Pitfall 3: GIL Contention

Symptom: High CPU usage but low throughput, jobs take longer than expected

Cause: Too many threads competing for GIL, or CPU-bound jobs

Solution: Reduce concurrency for CPU-heavy jobs

# Use multiple processes with lower concurrency each
# config/sidekiq.yml
 
# Process 1: High I/O jobs
:concurrency: 25
:queues:
  - api_calls
  - uploads
 
# Process 2: CPU-heavy jobs (separate process)
:concurrency: 5
:queues:
  - image_processing
  - data_transformation

Pitfall 4: Silent Failures

Symptom: Jobs fail but aren’t retried or logged

Cause: Exceptions not raised properly, swallowed errors

Solution: Use Sidekiq’s built-in retry mechanism

class RiskyJob
  include Sidekiq::Job
  sidekiq_options retry: 5  # Default is 25
 
  def perform(data)
    process(data)
  rescue SomeRecoverableError => e
    # Let Sidekiq handle retry
    raise e
  rescue UnrecoverableError => e
    # Log and don't retry
    Rails.logger.error("Permanent failure: #{e.message}")
    # Don't raise - job won't retry
  end
end

Pitfall 5: Lost Scheduled Jobs

Symptom: Jobs scheduled in Resque don’t run after migration

Cause: Resque-scheduler and Sidekiq use different mechanisms

Solution: Migrate scheduling logic

# Resque-scheduler (config/resque_schedule.yml)
send_report:
  cron: "0 8 * * *"
  class: "ReportJob"
  args:
  queue: reports
 
# Sidekiq-cron (config/initializers/sidekiq.rb)
Sidekiq::Cron::Job.create(
  name: 'Send daily report',
  cron: '0 8 * * *',
  class: 'ReportJob',
  queue: 'reports'
)

Decision Factors

The Throughput Threshold

Sidekiq’s complexity only pays off above a certain job volume—for applications processing hundreds of jobs per hour (not thousands per minute), Resque’s simplicity often outweighs Sidekiq’s efficiency gains.

When deciding whether to migrate, consider:

Migrate to Sidekiq If:

✅ High job volume - Processing thousands of jobs per minute ✅ Memory constrained - Limited RAM budget for background workers ✅ I/O-bound workloads - Jobs spend time on database/API/file operations ✅ Cost sensitive - Want to reduce server infrastructure costs ✅ Modern codebase - Using thread-safe gems and patterns ✅ Team comfortable with threading - Developers understand concurrency

Stay with Resque If:

❌ Simple, low-volume workloads - Processing dozens of jobs per hour ❌ CPU-bound jobs - Computational work that benefits from process isolation ❌ Legacy dependencies - Using old gems with thread safety issues ❌ Complex state requirements - Jobs need process-level isolation ❌ Team prefers simplicity - Threading complexity outweighs benefits ❌ Working system - Current setup meets needs without issues

Hybrid Approach:

The Two-System Tax

Running both Resque and Sidekiq sidesteps migration risk but doubles operational burden—you now monitor two dashboards, debug two systems, and train developers on two different threading models.

Some teams run both systems:

Sidekiq: High-volume, I/O-bound jobs (emails, API calls, uploads)
Resque: CPU-intensive or problematic jobs (video encoding, data processing)

This maximizes benefits while minimizing risk, though it increases operational complexity.

Real-World Migration Results

Organizations that migrated report significant improvements:

Memory Savings: 60-80% reduction in memory usage for same throughput Cost Reduction: 50-70% fewer servers needed for background processing Throughput Increase: 3-5x more jobs processed per server Operational Simplification: One background job system instead of multiple

However, migrations also encountered challenges:

2-4 weeks of finding and fixing thread safety issues
Memory profiling and optimization for specific jobs
Team learning curve around thread safety patterns
Increased complexity in debugging multi-threaded issues

The consensus: worthwhile for high-throughput applications, overkill for simple use cases.

Conclusion

Migrating from Resque to Sidekiq exchanges process isolation for memory efficiency and throughput. The core trade-off is accepting Thread Safety discipline in exchange for processing more work with fewer resources.

For modern web applications processing significant background job volume, Sidekiq’s threading model typically provides better resource utilization and simpler operational overhead. The migration requires care around thread safety but rewards teams with dramatic performance improvements and cost savings.

The architectural shift reflects broader trends in Ruby Concurrency Mechanisms—moving from heavyweight process isolation toward lighter-weight threading while maintaining Ruby’s developer-friendly abstractions. Understanding both approaches deepens insight into distributed system design patterns and performance optimization strategies.

Recent Writing

Recent Notes

Migrating from Resque to Sidekiq

Architecture Comparison

Resque: Process-Based Isolation

Sidekiq: Thread-Based Efficiency

Performance Comparison

Memory Usage

Throughput Benchmarks

The I/O Advantage

API and Code Comparison

Job Definition

Enqueuing Jobs

Configuration

Hooks and Middleware

Redis Data Structure Differences

Resque Redis Keys

Sidekiq Redis Keys

Migration Strategy

Phase 1: Preparation and Setup

Phase 2: Pilot Migration

Phase 3: Identify and Fix Thread Safety Issues

Phase 4: Memory Optimization

Phase 5: Gradual Queue Migration

Phase 6: Optimize Configuration

Phase 7: Decommission Resque

Common Pitfalls and Solutions

Pitfall 1: Connection Pool Exhaustion

Pitfall 2: Memory Bloat

Pitfall 3: GIL Contention

Pitfall 4: Silent Failures

Pitfall 5: Lost Scheduled Jobs

Decision Factors

Migrate to Sidekiq If:

Stay with Resque If:

Hybrid Approach:

Real-World Migration Results

Conclusion

Recent Writing

Recent Notes

Graph View

Table of Contents