当太多作业待处理时，Ruby、DelayedJob 会变慢

Question

当表 delayed_jobs 开始增长到数百个以上时，工作人员的性能开始呈指数下降。

Answer 1

我已经多次被这个问题困扰，所以我将我的发现公开给 future 的新手来面对这个噩梦。

DelayedJobs 项目中存在与此问题相关的几个问题:

• https://github.com/collectiveidea/delayed_job/issues/581
• https://github.com/collectiveidea/delayed_job/issues/650

问题出在 avery Worker run 中使用的 DelayedJob 查询中:

UPDATE `delayed_jobs` SET `locked_at` = '2014-04-17 22:32:20', `locked_by` = 'host:b38f770a-f3f3-4b2a-8c66-7c8eebdb7fea pid:2' WHERE ((run_at <= '2014-04-17 22:32:20' AND (locked_at IS NULL OR locked_at < '2014-04-17 18:32:20') OR locked_by = 'host:b38f770a-f3f3-4b2a-8c66-7c8eebdb7fea pid:2') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 1

在我的例子中，对于少于 1000 个作业，可能需要近 1 秒的时间......并且随着更多作业待处理而呈指数增长。

我找到的唯一解决方案是 this blog 中公开的解决方案，简而言之:由于问题是初始查询缺乏合适的索引，解决方案是批量拆分表:

-- stop workers
select max(id) from delayed_jobs; -- -> 10010
create table delayed_jobs_backup like delayed_jobs;
insert into delayed_jobs_backup select * from delayed_jobs where id < 10010;
delete from delayed_jobs where id < 10010;
-- start workers
-- while jobs in delayed_jobs_backup do
  -- wait until the batch have been processed
  insert into delayed_jobs select * from delayed_jobs_backup limit 1000;
  delete from delayed_jobs_backup limit 1000;
-- end