ruby-on-rails - 使用线程时处理 Rails 中的循环依赖

Question

以下是添加到 my app 的五个文件(无数据库，无需设置):

lib/tasks/precomputation.rake

namespace :precomputation do
  desc "This fetches data for precomputation"
  task fetch_all: :environment do
      Precomputation.precompute_all_data
    # end
  end
end

app/models/precomputation.rb

class Precomputation
  def self.precompute_all_data
    ad_accounts = [1,2,3]
    bgthread = BackgroundThread::BGThreadPool.new(1)
    tasks = []
    ad_accounts.each do |ad_account_id|
      p = Proc.new do
        begin
          MongoPipeline::FbAdCampaignMongoPipeline.new(ad_account_id).fetch_all
          false
        ensure
          GC.start
        end
      end
      tasks << [p, "test #{ad_account_id}"]
    end
    bgthread.add_randomized_tasks(tasks)
    bgthread.do_work
  end
end

应用程序/模型/mongo_pipeline.rb

module MongoPipeline
  class Base
    def initialize(ad_account_id)
    end

    def insert_data
      puts 'inserting data'
    end

    def fetch_all
      extract_data # Child Class defines this method
      insert_data # Base class defines this method
    end
  end
end

app/models/mongo_pipeline/fb_ad_campaign_mongo_pipeline.rb

module MongoPipeline
  class FbAdCampaignMongoPipeline < MongoPipeline::Base
    def extract_data
      puts 'here is campaign data'
    end
  end
end

和app/models/background_thread.rb

(注意:使用并行 gem 且无后台线程的替代实现也会因类似错误而崩溃 -: https://github.com/pratik60/circular_dependency_havoc/tree/parallel )

错误日志

rake aborted!
Circular dependency detected while autoloading constant MongoPipeline
/Users/pratikbothra/.rvm/gems/ruby-2.2.2/gems/activesupport-4.2.4/lib/active_support/dependencies.rb:492:in `load_missing_constant'
/Users/pratikbothra/.rvm/gems/ruby-2.2.2/gems/activesupport-4.2.4/lib/active_support/dependencies.rb:184:in `const_missing'
/Users/pratikbothra/.rvm/gems/ruby-2.2.2/gems/activesupport-4.2.4/lib/active_support/dependencies.rb:526:in `load_missing_constant'
/Users/pratikbothra/.rvm/gems/ruby-2.2.2/gems/activesupport-4.2.4/lib/active_support/dependencies.rb:184:in `const_missing'
/webapps/circular_dependency_havoc/app/models/precomputation.rb:9:in `block (2 levels) in precompute_all_data'
/webapps/circular_dependency_havoc/app/models/background_thread.rb:91:in `call'
/webapps/circular_dependency_havoc/app/models/background_thread.rb:91:in `block in background'
Tasks: TOP => precomputation:fetch_all
(See full trace by running task with --trace)

关于我做错了什么有什么想法吗？后台线程库被克隆并刚刚修改。如果您认为这是问题所在，请随时更换它。任何建议、任何想法都非常受欢迎。

Answer 1

我在 rake 任务中构建线程循环来轮询外部服务器时遇到了这个问题，并得出了相同的结论，即它与自动加载和不可预测的竞争条件相关。然而，我不想简单地打开急切加载，因此我有两种方法来解决这个问题。一是弄清楚哪些类正在我的任务中加载，并将它们加载到父级中，然后再生成任何线程:

# preload classes that showed circular dependencies just in this task
MediaFile
VideoPlatformIntegration
Tag

10.times.map{Thread.new{...}}

这就像一个魅力，但放入生产代码中有点奇怪，并且对于任务中的任何更改或其调用的任何方法都不稳健。

另一个更不完美，但实际上只是修改开发环境中的行为，而不依赖于了解可能导致这种情况的一切。由于它还处于开发阶段，我认为这是可以接受的:在每个初始线程开始时休眠一段随机时间(0 到 1 秒之间):

sleep(Random.rand) if Rails.env == "development"

这种 sleep 可以防止线程同时启动，并大大减少加载类时出现任何竞争条件的可能性。我在这里使用一批简单的 10 个线程进行了概念验证，在 sleep 时运行脚本 4 次，在不 sleep 时运行脚本 4 次，在不 sleep 时显示可靠的循环依赖错误:

[video_platform_integration] andrew@~/3p/app3$ rails runner '10.times.map{Thread.new{MediaFile.last rescue puts($!)}}'
Circular dependency detected while autoloading constant MediaFile
Circular dependency detected while autoloading constant MediaFile
[video_platform_integration] andrew@~/3p/app3$ rails runner '10.times.map{Thread.new{MediaFile.last rescue puts($!)}}'
Circular dependency detected while autoloading constant MediaFile
[video_platform_integration] andrew@~/3p/app3$ rails runner '10.times.map{Thread.new{MediaFile.last rescue puts($!)}}'
[video_platform_integration] andrew@~/3p/app3$ rails runner '10.times.map{Thread.new{MediaFile.last rescue puts($!)}}'
Circular dependency detected while autoloading constant MediaFile
Circular dependency detected while autoloading constant MediaFile
[video_platform_integration] andrew@~/3p/app3$ rails runner '10.times.map{Thread.new{sleep(Random.rand); MediaFile.last rescue puts($!)}}'
[video_platform_integration] andrew@~/3p/app3$ rails runner '10.times.map{Thread.new{sleep(Random.rand); MediaFile.last rescue puts($!)}}'
[video_platform_integration] andrew@~/3p/app3$ rails runner '10.times.map{Thread.new{sleep(Random.rand); MediaFile.last rescue puts($!)}}'
[video_platform_integration] andrew@~/3p/app3$ rails runner '10.times.map{Thread.new{sleep(Random.rand); MediaFile.last rescue puts($!)}}'