我在干净的 AWS 实例中使用默认配置设置 cassandra，并将 10000 列插入一行，每列有 1MB 数据。我使用这个 ruby(版本 1.9.3)脚本:
10000.times do
key = rand(36**8).to_s(36)
value = rand(36**1024).to_s(36) * 1024
Cas_client.insert(TestColumnFamily,TestRow,{key=>value})
end
每次我运行这个脚本，它都会崩溃:
/usr/local/lib/ruby/gems/1.9.1/gems/thrift-0.8.0/lib/thrift/transport/socket.rb:109:in `read': CassandraThrift::Cassandra::Client::TransportException from /usr/local/lib/ruby/gems/1.9.1/gems/thrift-0.8.0/lib/thrift/transport/base_transport.rb:87:in `read_all'
from /usr/local/lib/ruby/gems/1.9.1/gems/thrift-0.8.0/lib/thrift/transport/framed_transport.rb:104:in `read_frame'
from /usr/local/lib/ruby/gems/1.9.1/gems/thrift-0.8.0/lib/thrift/transport/framed_transport.rb:69:in `read_into_buffer'
from /usr/local/lib/ruby/gems/1.9.1/gems/thrift-0.8.0/lib/thrift/client.rb:45:in `read_message_begin'
from /usr/local/lib/ruby/gems/1.9.1/gems/thrift-0.8.0/lib/thrift/client.rb:45:in `receive_message'
from /usr/local/lib/ruby/gems/1.9.1/gems/cassandra-0.15.0/vendor/0.8/gen-rb/cassandra.rb:251:in `recv_batch_mutate'
from /usr/local/lib/ruby/gems/1.9.1/gems/cassandra-0.15.0/vendor/0.8/gen-rb/cassandra.rb:243:in `batch_mutate'
from /usr/local/lib/ruby/gems/1.9.1/gems/thrift_client-0.8.1/lib/thrift_client/abstract_thrift_client.rb:150:in `handled_proxy' from /usr/local/lib/ruby/gems/1.9.1/gems/thrift_client-0.8.1/lib/thrift_client/abstract_thrift_client.rb:60:in `batch_mutate'
from /usr/local/lib/ruby/gems/1.9.1/gems/cassandra-0.15.0/lib/cassandra/protocol.rb:7:in `_mutate'
from /usr/local/lib/ruby/gems/1.9.1/gems/cassandra-0.15.0/lib/cassandra/cassandra.rb:463:in `insert'
from a.rb:6:in `block in <main>'
from a.rb:3:in `times'
from a.rb:3:in `<main>'
但 cassandra 执行正常，然后我运行另一个 ruby 脚本来获取我插入了多少列:
p cas_client.count_columns(TestColumnFamily,TestRow)
此脚本再次崩溃，同样的错误信息。并且 cassandra 进程保持 100% 的 cpu 使用率。
AWS m1.xlarge type instance (15GB mem,800GB harddisk, 4cores cpu)
cassandra-1.1.2
ruby-1.9.3-p194
jdk-7u6-linux-x64
ruby-gems:
cassandra (0.15.0)
thrift (0.8.0)
thrift_client (0.8.1)
问题是什么？
最佳答案
10,000 列，每列 1 MB 是 10 GB 的数据。
Cassandra rpc 使用 thrift，这要求 rpc 调用的整个返回值必须适合内存，因此尝试读取所有列将需要您将 10 gig thrift 对象加载到内存中，这是不切实际的，尤其是在 ruby 中.
关于ruby - 每行有大量列的Cassandra，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/12032427/