问题:
我有一个 Rails 应用程序,它要求用户上传某种类型的电子表格(csv、xslx、xsl 等)进行处理,这可能是一项成本高昂的操作,因此我们决定将其发送到后台服务作为解决方案到这个问题。我们关心的问题是,因为我们的生产系统是在 Heroku 上,所以我们需要先将文件存储在 AS3 上,然后再检索进行处理。
因为将文件上传到 AS3 本身就是一项成本高昂的操作,所以这也应该作为后台作业来完成。问题在于,由于 Resque 需要将文件数据放入 Redis 或稍后检索,因此使用 Resque 执行此操作可能会占用大量 RAM。如您所知,Redis 仅将其数据存储在 RAM 中,并且更喜欢简单的键值对,因此我们希望尽量避免这种情况。
下面是一些伪代码,作为我们想要尝试做的示例:
workers/AS3Uploader.rb
require 'fog'
class AS3Uploader
@queue = :as3_uploader
def self.perform(some, file, data)
# create a connection
connection = Fog::Storage.new({
:provider => 'AWS',
:aws_access_key_id => APP_CONFIG['s3_key'],
:aws_secret_access_key => APP_CONFIG['s3_secret']
})
# First, a place to contain the glorious details
directory = connection.directories.create(
:key => "catalog-#{Time.now.to_i}", # globally unique name
:public => true
)
# list directories
p connection.directories
# upload that catalog
file = directory.files.create(
:key => 'catalog.xml',
:body => File.open(blah), # not sure how to get file data here with out putting it into RAM first using Resque/Redis
:public => true
end
# make a call to Enqueue the processing of the catalog
Resque.enqueue(CatalogProcessor, some, parameters, here)
end
controllers/catalog_upload_controller.rb
def create
# process params
# call Enqueue to start the file processing
# What do I do here? I could send all of the file data here right now
# but like I said previously that means storing potentially 100s of MB into RAM
Resque.enqueue(AS3Uploader, some, parameters, here)
end
最佳答案
我建议你做的方式是
- 将您的文件存储在您创建的
tmp
目录中并获取file-path
- 告诉
Resque
使用file-path
上传文件 - 制作
Resque
将file-path
存储在redis
而不是整个file-content
(它会很贵) - 现在工作人员将文件上传到
AWS-S3
Note: If you have multiple instances like One instance for background processing, One for database, One as utility instance then your tmp dir may not be available to other instances.. so store the file in the
temp
dir inside the instance holding theresque
关于ruby-on-rails - rails : Uploading a file to AS3 using fog and resque,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/32321015/