ubuntu - 由于 sstable 损坏，无法启动 Cassandra

Question

在我运行之后:- sudo service cassandra start 然后是 sudo service cassandra status 我得到一个无法访问 Cassandra 的 pidfile。

当我检查日志时，它给了我这个，我假设它是 sstable 损坏但找不到任何方法来解决它。

ERROR [SSTableBatchOpen:1] 2016-05-30 23:17:42,301 FileUtils.java:447 - 
Exiting forcefully due to file system exception on startup, disk failure policy "stop"

org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFException
    at org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:131) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:85) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:79) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:72) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:169) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:741) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:692) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:480) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:376) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:523) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_80]
    at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_80]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_80]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_80]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]

Caused by: java.io.EOFException: null
    at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340) ~[na:1.7.0_80]
    at java.io.DataInputStream.readUTF(DataInputStream.java:589) ~[na:1.7.0_80]
    at java.io.DataInputStream.readUTF(DataInputStream.java:564) ~[na:1.7.0_80]
    at org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:106) ~[apache-cassandra-2.1.11.jar:2.1.11]
    ... 14 common frames omitted

我在删除 sstables 后收到了这个额外的错误。

ERROR [SSTableBatchOpen:2] 2016-06-13 22:44:59,177 CassandraDaemon.java:227 - Exception in thread Thread[SSTableBatchOpen:2,5,main]
java.lang.IllegalStateException: Shutdown in progress
    at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) ~[na:1.7.0_80]
    at java.lang.Runtime.removeShutdownHook(Runtime.java:239) ~[na:1.7.0_80]
    at org.apache.cassandra.service.StorageService.removeShutdownHook(StorageService.java:758) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.utils.JVMStabilityInspector$Killer.killCurrentJVM(JVMStabilityInspector.java:119) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.utils.JVMStabilityInspector.killCurrentJVM(JVMStabilityInspector.java:88) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.io.util.FileUtils.handleStartupFSError(FileUtils.java:450) ~[apache-cassandra-2.1.11.jar:2.1.11]

我也意识到其他节点上的错误，但我能够启动它们。 空指针异常

ERROR [GossipStage:1] 2016-06-13 23:06:31,317 CassandraDaemon.java:227 - Exception in thread Thread[GossipStage:1,5,main]
java.lang.NullPointerException: null
    at org.apache.cassandra.service.StorageService.getApplicationStateValue(StorageService.java:1624) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.service.StorageService.getTokensFor(StorageService.java:1632) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:1686) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.service.StorageService.onChange(StorageService.java:1510) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.service.StorageService.onJoin(StorageService.java:2161) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:1042) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1115) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:49) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_80]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_80]
    at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_80]

Answer 1

在ubuntu 16.04+cassandra 3.4.0上，我解决这个问题的方法是删除异常指向的中间文件。我没有丢失任何开发数据，但我现在正在做备份。