cassandra - 如何在不重新启动 cassandra 节点的情况下重建 sstable？

Question

当我在一个节点上完成一项紧凑的工作时，它会抛出以下异常:

ERROR [CompactionExecutor:116922] 2016-04-07 12:51:17,291 CassandraDaemon.java:153 - Exception in thread Thread[CompactionExecutor:116922,1,main]
org.apache.cassandra.io.sstable.CorruptSSTableException: org.apache.cassandra.io.compress.CorruptBlockException: (/data1/data/cassandra_uc_log/log_user-2fdda2a03a7f11e58156c78e55b68188/cassandra_uc_log-log_user-ka-7611-Data.db): corruption detected, chunk at 602529 of l
ength 12126.
    at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:92) ~[apache-cassandra-2.1.2.jar:2.1.2]
    at org.apache.cassandra.io.compress.CompressedThrottledReader.reBuffer(CompressedThrottledReader.java:41) ~[apache-cassandra-2.1.2.jar:2.1.2]
    at org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:326) ~[apache-cassandra-2.1.2.jar:2.1.2]
    at java.io.RandomAccessFile.readFully(RandomAccessFile.java:444) ~[na:1.7.0_60]
    at java.io.RandomAccessFile.readFully(RandomAccessFile.java:424) ~[na:1.7.0_60]
    at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:351) ~[apache-cassandra-2.1.2.jar:2.1.2]
    at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:348) ~[apache-cassandra-2.1.2.jar:2.1.2]
    at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:311) ~[apache-cassandra-2.1.2.jar:2.1.2]
    at org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:132) ~[apache-cassandra-2.1.2.jar:2.1.2]
    at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:86) ~[apache-cassandra-2.1.2.jar:2.1.2]
    at org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:52) ~[apache-cassandra-2.1.2.jar:2.1.2]
    at org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:46) ~[apache-cassandra-2.1.2.jar:2.1.2]
    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) ~[guava-16.0.jar:na]
    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-16.0.jar:na]
    at org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:116) ~[apache-cassandra-2.1.2.jar:2.1.2]
    at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:146) ~[apache-cassandra-2.1.2.jar:2.1.2]
    at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:125) ~[apache-cassandra-2.1.2.jar:2.1.2]
    at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:99) ~[apache-cassandra-2.1.2.jar:2.1.2]
    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) ~[guava-16.0.jar:na]
    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-16.0.jar:na]
    at com.google.common.collect.Iterators$7.computeNext(Iterators.java:645) ~[guava-16.0.jar:na]
    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) ~[guava-16.0.jar:na]
    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-16.0.jar:na]
    at org.apache.cassandra.db.ColumnIndex$Builder.buildForCompaction(ColumnIndex.java:165) ~[apache-cassandra-2.1.2.jar:2.1.2]
    at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:110) ~[apache-cassandra-2.1.2.jar:2.1.2]
    at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:200) ~[apache-cassandra-2.1.2.jar:2.1.2]
    at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:115) ~[apache-cassandra-2.1.2.jar:2.1.2]
    at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:183) ~[apache-cassandra-2.1.2.jar:2.1.2]
    at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) ~[apache-cassandra-2.1.2.jar:2.1.2]
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.2.jar:2.1.2]
    at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:75) ~[apache-cassandra-2.1.2.jar:2.1.2]
    at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) ~[apache-cassandra-2.1.2.jar:2.1.2]
    at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:232) ~[apache-cassandra-2.1.2.jar:2.1.2]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_60]
    at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_60]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_60]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_60]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60]
Caused by: org.apache.cassandra.io.compress.CorruptBlockException: (/data1/data/cassandra_uc_log/log_user-2fdda2a03a7f11e58156c78e55b68188/cassandra_uc_log-log_user-ka-7611-Data.db): corruption detected, chunk at 602529 of length 12126.
    at org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:112) ~[apache-cassandra-2.1.2.jar:2.1.2]
    at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:88) ~[apache-cassandra-2.1.2.jar:2.1.2]
    ... 37 common frames omitted

所以，我删除了sstable cassandra_uc_log-log_user-ka-7611-Data.db而不是进行修复，但它抛出了一个新的异常:

 Repair session f33a4b10-ffb7-11e5-8fe3-31b2e5b5b0b2 for range (-5651751204441903619,-5621122634670931727] failed with error java.io.IOException: Failed during snapshot creation.

如何修复或重建 sstable cassandra_uc_log-log_user-ka-7611-Data.db

Answer 1

恐怕您必须在删除文件的情况下重新启动节点。下次您也可以尝试运行 nodetool scrub损坏的 sstable 就位。

您没有提到您的 Cassandra 版本，但假设您在 Cassandra 2.1+ 中使用增量修复，您可能还需要进行完整修复，以防 sstable 已处于修复状态。