scala - 不使用 CPU 时 Cassandra 超时

Question

我使用带有 Datastax Cassandra 驱动程序的 Phantom-DSL 时出现 Cassandra 超时。然而，Cassandra 似乎并没有重载。以下是我得到的异常:

com.datastax.driver.core.exceptions.OperationTimedOutException: [node-0.cassandra.dev/10.0.1.137:9042] Timed out waiting for server response
    at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onTimeout(RequestHandler.java:766)
    at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1267)
    at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:588)
    at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:662)
    at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:385)
    at java.lang.Thread.run(Thread.java:745)

以下是我在这段时间内从 Cassandra Datadog 连接器获得的统计数据:

您可以在顶部中心图表上看到我们的读取速率(每秒)。我们的 CPU 和内存使用率非常低。

以下是我们配置 Datastax 驱动程序的方式:

val points = ContactPoints(config.cassandraHosts)
  .withClusterBuilder(_.withSocketOptions(
    new SocketOptions()
      .setReadTimeoutMillis(config.cassandraNodeTimeout)
  ))
  .withClusterBuilder(_.withPoolingOptions(
    new PoolingOptions()
      .setConnectionsPerHost(
        HostDistance.LOCAL,
        2,
        2
      )
      .setConnectionsPerHost(
        HostDistance.REMOTE,
        2,
        2
      )
      .setMaxRequestsPerConnection(
        HostDistance.LOCAL,
        2048
      )
      .setMaxRequestsPerConnection(
        HostDistance.REMOTE,
        2048
      )
      .setPoolTimeoutMillis(10000)
      .setNewConnectionThreshold(
        HostDistance.LOCAL,
        1500
      )
      .setNewConnectionThreshold(
        HostDistance.REMOTE,
        1500
      )

))

我们的nodetool cfstats看起来像这样:

$ nodetool cfstats alexandria_dev.match_sums
Keyspace : alexandria_dev
        Read Count: 101892
        Read Latency: 0.007479115141522397 ms.
        Write Count: 18721
        Write Latency: 0.012341060840767052 ms.
        Pending Flushes: 0
                Table: match_sums
                SSTable count: 0
                Space used (live): 0
                Space used (total): 0
                Space used by snapshots (total): 0
                Off heap memory used (total): 0
                SSTable Compression Ratio: 0.0
                Number of keys (estimate): 15328
                Memtable cell count: 15332
                Memtable data size: 21477107
                Memtable off heap memory used: 0
                Memtable switch count: 0
                Local read count: 17959
                Local read latency: 0.015 ms
                Local write count: 15332
                Local write latency: 0.013 ms
                Pending flushes: 0
                Percent repaired: 100.0
                Bloom filter false positives: 0
                Bloom filter false ratio: 0.00000
                Bloom filter space used: 0
                Bloom filter off heap memory used: 0
                Index summary off heap memory used: 0
                Compression metadata off heap memory used: 0
                Compacted partition minimum bytes: 0
                Compacted partition maximum bytes: 0
                Compacted partition mean bytes: 0
                Average live cells per slice (last five minutes): 1.0
                Maximum live cells per slice (last five minutes): 1
                Average tombstones per slice (last five minutes): 1.0
                Maximum tombstones per slice (last five minutes): 1
                Dropped Mutations: 0

当我们运行 cassandra-stress 时，我们没有遇到任何问题:每秒稳定读取 50k，符合预期。

每当我进行查询时，Cassandra 都会出现此错误:

INFO  [Native-Transport-Requests-2] 2017-03-10 23:59:38,003 Message.java:611 - Unexpected exception during request; channel = [id: 0x65d7a0cd, L:/10.0.1.98:9042 ! R:/10.0.1.126:35536]
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peer
        at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.39.Final.jar:4.0.39.Final]

为什么我们会超时？

编辑:我上传了错误的仪表板。请查看新图片。

Answer 1

2 个有帮助的问题:

1. 您的超时设置是多少
2. 查询什么？

现在澄清一下我认为你出错的地方:

1. 分辨率太粗略，无法诊断单个查询，我可以让一台服务器什么都不做，做一个昂贵的查询，在整个时间内固定一些瓶颈，在这个规模上看起来没有任何瓶颈，运行 iostat -x 1同时检查服务器，您可能会发现与图表在该分辨率下显示的内容截然不同的内容。
2. 如果我正确地查看了您的 CPU 使用率图表，它看起来像是 50% 使用率。在现代服务器上，由于超线程以及聚合 CPU 使用情况的工作原理，实际上非常繁忙，请参阅 https://www.percona.com/blog/2015/01/15/hyper-threading-double-cpu-throughput/