Cassandra 启动失败:SSTableBatchOpen IllegalArgumentException
在 Ubuntu 16.04 VM 中运行的实验性 Cassandra 安装发生电源故障，之后 Cassandra 无法启动。
一切都工作正常，直到停电。我不知道还有什么其他原因导致安装无法启动。
Cassandra system.log 在这里:
cat /var/log/cassandra/system.log
INFO [main] 2016-11-29 21:55:25,307 YamlConfigurationLoader.java:89 - Configuration location: file:/etc/cassandra/cassandra.yaml
INFO [main] 2016-11-29 21:55:26,099 DatabaseDescriptor.java:317 - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO [main] 2016-11-29 21:55:26,407 DatabaseDescriptor.java:437 - Global memtable on-heap threshold is enabled at 609MB
INFO [main] 2016-11-29 21:55:26,408 DatabaseDescriptor.java:441 - Global memtable off-heap threshold is enabled at 609MB
WARN [main] 2016-11-29 21:55:26,538 DatabaseDescriptor.java:643 - Only 30.073GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots
INFO [main] 2016-11-29 21:55:26,646 CassandraDaemon.java:465 - Hostname: tiira
INFO [main] 2016-11-29 21:55:26,647 CassandraDaemon.java:472 - JVM vendor/version: OpenJDK 64-Bit Server VM/1.8.0_111
INFO [main] 2016-11-29 21:55:26,647 CassandraDaemon.java:473 - Heap size: 2.379GiB/2.379GiB
INFO [main] 2016-11-29 21:55:26,648 CassandraDaemon.java:478 - Code Cache Non-heap memory: init = 2555904(2496K) used = 4315904(4214K) committed = 4390912(4288K) max = 251658240(245760K)
INFO [main] 2016-11-29 21:55:26,649 CassandraDaemon.java:478 - Metaspace Non-heap memory: init = 0(0K) used = 16194936(15815K) committed = 16777216(16384K) max = -1(-1K)
INFO [main] 2016-11-29 21:55:26,649 CassandraDaemon.java:478 - Compressed Class Space Non-heap memory: init = 0(0K) used = 1950560(1904K) committed = 2097152(2048K) max = 1073741824(1048576K)
INFO [main] 2016-11-29 21:55:26,650 CassandraDaemon.java:478 - Par Eden Space Heap memory: init = 503316480(491520K) used = 110748120(108152K) committed = 503316480(491520K) max = 503316480(491520K)
INFO [main] 2016-11-29 21:55:26,650 CassandraDaemon.java:478 - Par Survivor Space Heap memory: init = 62914560(61440K) used = 0(0K) committed = 62914560(61440K) max = 62914560(61440K)
INFO [main] 2016-11-29 21:55:26,650 CassandraDaemon.java:478 - CMS Old Gen Heap memory: init = 1988100096(1941504K) used = 0(0K) committed = 1988100096(1941504K) max = 1988100096(1941504K)
INFO [main] 2016-11-29 21:55:26,888 CLibrary.java:150 - JNA mlockall successful
WARN [main] 2016-11-29 21:55:26,891 StartupChecks.java:123 - jemalloc shared library could not be preloaded to speed up memory allocations
WARN [main] 2016-11-29 21:55:26,891 StartupChecks.java:156 - JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
WARN [main] 2016-11-29 21:55:26,892 StartupChecks.java:193 - OpenJDK is not recommended. Please upgrade to the newest Oracle Java release
INFO [main] 2016-11-29 21:55:26,898 SigarLibrary.java:44 - Initializing SIGAR library
WARN [main] 2016-11-29 21:55:26,942 SigarLibrary.java:174 - Cassandra server running in degraded mode. Is swap disabled? : false, Address space adequate? : true, nofile limit adequate? : true, nproc limit adequate? : true
INFO [main] 2016-11-29 21:55:27,366 QueryProcessor.java:113 - Initialized prepared statement caches with 10 MB (native) and 10 MB (Thrift)
INFO [main] 2016-11-29 21:55:28,411 ColumnFamilyStore.java:412 - Initializing system.IndexInfo
INFO [SSTableBatchOpen:1] 2016-11-29 21:55:29,537 BufferPool.java:229 - Global buffer pool is enabled, when pool is exhausted (max is 512.000MiB) it will allocate on heap
INFO [main] 2016-11-29 21:55:30,040 CacheService.java:113 - Initializing key cache with capacity of 100 MBs.
INFO [main] 2016-11-29 21:55:30,058 CacheService.java:135 - Initializing row cache with capacity of 0 MBs
INFO [main] 2016-11-29 21:55:30,064 CacheService.java:164 - Initializing counter cache with capacity of 50 MBs
INFO [main] 2016-11-29 21:55:30,067 CacheService.java:175 - Scheduling counter cache save to every 7200 seconds (going to save all keys).
INFO [main] 2016-11-29 21:55:30,152 ColumnFamilyStore.java:412 - Initializing system.batches
...deleted...
INFO [main] 2016-11-29 21:55:32,490 ColumnFamilyStore.java:412 - Initializing system_auth.roles
INFO [main] 2016-11-29 21:55:32,512 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_auth as storage service is not initialized
INFO [main] 2016-11-29 21:55:32,531 ColumnFamilyStore.java:412 - Initializing system_traces.events
INFO [main] 2016-11-29 21:55:32,575 ColumnFamilyStore.java:412 - Initializing system_traces.sessions
INFO [main] 2016-11-29 21:55:32,626 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_traces as storage service is not initialized
INFO [main] 2016-11-29 21:55:32,638 ColumnFamilyStore.java:412 - Initializing my_keyspace.user
INFO [main] 2016-11-29 21:55:32,667 ColumnFamilyStore.java:412 - Initializing my_keyspace.user.user_address_idx
ERROR [SSTableBatchOpen:1] 2016-11-29 21:55:32,679 CassandraDaemon.java:226 - Exception in thread Thread[SSTableBatchOpen:1,5,main]
java.lang.IllegalArgumentException: null
at java.nio.Buffer.limit(Buffer.java:275) ~[na:1.8.0_111]
at org.apache.cassandra.utils.ByteBufferUtil.readBytes(ByteBufferUtil.java:619)
at org.apache.cassandra.db.marshal.TupleType.compareCustom(TupleType.java:122)
at org.apache.cassandra.db.marshal.AbstractType.compare(AbstractType.java:159)
at org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(LocalPartitioner.java:139)
at org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(LocalPartitioner.java:120)
at org.apache.cassandra.db.DecoratedKey.compareTo(DecoratedKey.java:85)
at org.apache.cassandra.io.sstable.format.SSTableReader.validate(SSTableReader.java:1190)
at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:499)
at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:375)
at org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:536)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
INFO [main] 2016-11-29 21:55:32,703 ColumnFamilyStore.java:412 - Initializing my_keyspace.user.user_emails_idx
INFO [SecondaryIndexManagement:1] 2016-11-29 21:55:32,707 CassandraIndex.java:707 - Submitting index build of user_address_idx for data in BigTableReader(path='/var/lib/cassandra/data/my_keyspace/user-0d220340b4d711e6850931fba1c453e0/mc-6-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/my_keyspace/user-0d220340b4d711e6850931fba1c453e0/mc-5-big-Data.db')
INFO [main] 2016-11-29 21:55:32,827 ColumnFamilyStore.java:412 - Initializing my_keyspace.user.user_last_name_idx
INFO [main] 2016-11-29 21:55:32,945 ColumnFamilyStore.java:412 - Initializing my_keyspace.user.user_phone_idx
INFO [main] 2016-11-29 21:55:33,028 ViewManager.java:137 - Not submitting build tasks for views in keyspace my_keyspace as storage service is not initialized
INFO [pool-4-thread-1] 2016-11-29 21:55:33,037 AutoSavingCache.java:196 - reading saved cache /var/lib/cassandra/saved_caches/KeyCache-e.db
INFO [pool-4-thread-1] 2016-11-29 21:55:33,065 AutoSavingCache.java:172 - Completed loading (32 ms; 76 keys) KeyCache cache
ERROR [CompactionExecutor:1] 2016-11-29 21:55:33,072 CassandraDaemon.java:226 - Exception in thread Thread[CompactionExecutor:1,1,main]
java.lang.RuntimeException: null for ks: my_keyspace, table: user.user_address_idx
at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1316)
at org.apache.cassandra.index.internal.CassandraIndex.insert(CassandraIndex.java:531)
at org.apache.cassandra.index.internal.CassandraIndex.access$100(CassandraIndex.java:72)
at org.apache.cassandra.index.internal.CassandraIndex$1.indexCell(CassandraIndex.java:444)
at org.apache.cassandra.index.internal.CassandraIndex$1.indexCells(CassandraIndex.java:436)
Caused by: java.lang.IllegalArgumentException: null
at java.nio.Buffer.limit(Buffer.java:275) ~[na:1.8.0_111]
at org.apache.cassandra.utils.ByteBufferUtil.readBytes(ByteBufferUtil.java:619)
at org.apache.cassandra.db.marshal.TupleType.compareCustom(TupleType.java:122)
... 16 common frames omitted
ERROR [SecondaryIndexManagement:1] 2016-11-29 21:55:33,072 CassandraDaemon.java:226 - Exception in thread Thread[SecondaryIndexManagement:1,5,main]
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: null for ks: my_keyspace, table: user.user_address_idx
at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:403)
at org.apache.cassandra.index.internal.CassandraIndex.buildBlocking(CassandraIndex.java:715)
at org.apache.cassandra.index.internal.CassandraIndex.lambda$getBuildIndexTask$5(CassandraIndex.java:685)
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: null for ks: my_keyspace, table: user.user_address_idx
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_111]
at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_111]
at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:399)
... 6 common frames omitted
Caused by: java.lang.RuntimeException: null for ks: my_keyspace, table: user.user_address_idx
at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1316)
at org.apache.cassandra.index.internal.CassandraIndex.insert(CassandraIndex.java:531)
at org.apache.cassandra.index.internal.CassandraIndex.access$100(CassandraIndex.java:72)
... 4 common frames omitted
Caused by: java.lang.IllegalArgumentException: null
at java.nio.Buffer.limit(Buffer.java:275) ~[na:1.8.0_111]
at org.apache.cassandra.utils.ByteBufferUtil.readBytes(ByteBufferUtil.java:619)
at org.apache.cassandra.db.marshal.TupleType.compareCustom(TupleType.java:122)
... 16 common frames omitted
INFO [main] 2016-11-29 21:55:33,082 CommitLog.java:166 - Replaying /var/lib/cassandra/commitlog/CommitLog-6-1480266920262.log, /var/lib/cassandra/commitlog/CommitLog-6-1480266920263.log, /var/lib/cassandra/commitlog/CommitLog-6-1480428373824.log, /var/lib/cassandra/commitlog/CommitLog-6-1480430610402.log, /var/lib/cassandra/commitlog/CommitLog-6-1480446994136.log, /var/lib/cassandra/commitlog/CommitLog-6-1480447217591.log, /var/lib/cassandra/commitlog/CommitLog-6-1480447657364.log, /var/lib/cassandra/commitlog/CommitLog-6-1480447697557.log, /var/lib/cassandra/commitlog/CommitLog-6-1480448913585.log, /var/lib/cassandra/commitlog/CommitLog-6-1480449134204.log
ERROR [main] 2016-11-29 21:55:33,315 JVMStabilityInspector.java:82 - Exiting due to error while processing commit log during initialization.
org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException: Unexpected error deserializing mutation; saved to /tmp/mutation3325983627363773229dat. This may be caused by replaying a mutation against a table with the same name but incompatible schema. Exception follows: org.apache.cassandra.serializers.MarshalException: Not enough bytes to read 0th field street
我尝试建立二级索引:
nodetool rebuild_index my_keyspace user user_address_idx
nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused (Connection refused)'
然后以 cassandra 用户身份运行 sstablescrub
cassandra@tiira:/etc$ sstablescrub -v --debug my_keyspace user
WARN 12:14:00 Only 29.952GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots
Pre-scrub sstables snapshotted into snapshot pre-scrub-1480594443732
Scrubbing BigTableReader(path='/var/lib/cassandra/data/my_keyspace/user-0d220340b4d711e6850931fba1c453e0/mc-5-big-Data.db') (0.372KiB)
Reading row at 0
row 48656c6c6f6132 is 0.022KiB
Reading row at 32
row 48656c6c6f33 is 0.053KiB
Reading row at 94
row 48656c6c6f is 0.138KiB
Reading row at 242
row 48656c6c6f32 is 0.128KiB
Scrub of BigTableReader(path='/var/lib/cassandra/data/my_keyspace/user-0d220340b4d711e6850931fba1c453e0/mc-5-big-Data.db') complete; looks like all 0 rows were tombstoned
Scrubbing BigTableReader(path='/var/lib/cassandra/data/my_keyspace/user-0d220340b4d711e6850931fba1c453e0/mc-6-big-Data.db') (0.069KiB)
Reading row at 0
row 48656c6c6f is 0.063KiB
WARNING: Error reading row (stacktrace follows):
org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /var/lib/cassandra/data/my_keyspace/user-0d220340b4d711e6850931fba1c453e0/mc-6-big-Data.db
at org.apache.cassandra.db.rows.UnfilteredRowIterators$1Validator.validate(UnfilteredRowIterators.java:306)
at org.apache.cassandra.db.rows.UnfilteredRowIterators$1Validator.applyToRow(UnfilteredRowIterators.java:287)
Caused by: org.apache.cassandra.serializers.MarshalException: Not enough bytes to read 0th field street
at org.apache.cassandra.db.marshal.UserType.validate(UserType.java:194)
at org.apache.cassandra.db.marshal.AbstractType.validateCellValue(AbstractType.java:186)
... 9 more
WARNING: Row starting at position 7 is unreadable; skipping to next
WARNING: No valid rows found while scrubbing BigTableReader(path='/var/lib/cassandra/data/my_keyspace/user-0d220340b4d711e6850931fba1c453e0/mc-6-big-Data.db'); it is marked for deletion now. If you want to attempt manual recovery, you can find a copy in the pre-scrub snapshot
cassandra@tiira:/etc$
cassandra@tiira:/etc$
cassandra@tiira:/etc$ sstablescrub -v --debug my_keyspace user
WARN 12:17:13 Only 29.953GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots
Pre-scrub sstables snapshotted into snapshot pre-scrub-1480594637427
Scrubbing BigTableReader(path='/var/lib/cassandra/data/my_keyspace/user-0d220340b4d711e6850931fba1c453e0/mc-7-big-Data.db') (0.372KiB)
Reading row at 0
row 48656c6c6f6132 is 0.022KiB
Reading row at 32
row 48656c6c6f33 is 0.053KiB
Reading row at 94
row 48656c6c6f is 0.138KiB
Reading row at 242
row 48656c6c6f32 is 0.128KiB
Scrub of BigTableReader(path='/var/lib/cassandra/data/my_keyspace/user-0d220340b4d711e6850931fba1c453e0/mc-7-big-Data.db') complete; looks like all 0 rows were tombstoned
cassandra@tiira:/etc$
尝试启动 cassandra:
INFO [main] 2016-12-01 14:56:18,945 YamlConfigurationLoader.java:89 - Configuration location: file:/etc/cassandra/cassandra.yaml
INFO [main] 2016-12-01 14:56:19,329 DatabaseDescriptor.java:317 - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO [main] 2016-12-01 14:56:19,492 DatabaseDescriptor.java:437 - Global memtable on-heap threshold is enabled at 609MB
INFO [main] 2016-12-01 14:56:19,493 DatabaseDescriptor.java:441 - Global memtable off-heap threshold is enabled at 609MB
WARN [main] 2016-12-01 14:56:19,516 DatabaseDescriptor.java:643 - Only 29.949GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots
INFO [main] 2016-12-01 14:56:19,590 CassandraDaemon.java:465 - Hostname: tiira
INFO [main] 2016-12-01 14:56:19,591 CassandraDaemon.java:472 - JVM vendor/version: OpenJDK 64-Bit Server VM/1.8.0_111
INFO [main] 2016-12-01 14:56:19,591 CassandraDaemon.java:473 - Heap size: 2.379GiB/2.379GiB
INFO [main] 2016-12-01 14:56:19,592 CassandraDaemon.java:478 - Code Cache Non-heap memory: init = 2555904(2496K) used = 4371136(4268K) committed = 4390912(4288K) max = 251658240(245760K)
INFO [main] 2016-12-01 14:56:19,592 CassandraDaemon.java:478 - Metaspace Non-heap memory: init = 0(0K) used = 15802936(15432K) committed = 16252928(15872K) max = -1(-1K)
INFO [main] 2016-12-01 14:56:19,593 CassandraDaemon.java:478 - Compressed Class Space Non-heap memory: init = 0(0K) used = 1910488(1865K) committed = 2097152(2048K) max = 1073741824(1048576K)
INFO [main] 2016-12-01 14:56:19,593 CassandraDaemon.java:478 - Par Eden Space Heap memory: init = 503316480(491520K) used = 100681584(98321K) committed = 503316480(491520K) max = 503316480(491520K)
INFO [main] 2016-12-01 14:56:19,593 CassandraDaemon.java:478 - Par Survivor Space Heap memory: init = 62914560(61440K) used = 0(0K) committed = 62914560(61440K) max = 62914560(61440K)
INFO [main] 2016-12-01 14:56:19,593 CassandraDaemon.java:478 - CMS Old Gen Heap memory: init = 1988100096(1941504K) used = 0(0K) committed = 1988100096(1941504K) max = 1988100096(1941504K)
WARN [main] 2016-12-01 14:56:19,699 StartupChecks.java:123 - jemalloc shared library could not be preloaded to speed up memory allocations
WARN [main] 2016-12-01 14:56:19,699 StartupChecks.java:156 - JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
WARN [main] 2016-12-01 14:56:19,700 StartupChecks.java:193 - OpenJDK is not recommended. Please upgrade to the newest Oracle Java release
INFO [main] 2016-12-01 14:56:19,701 SigarLibrary.java:44 - Initializing SIGAR library
WARN [main] 2016-12-01 14:56:19,729 SigarLibrary.java:174 - Cassandra server running in degraded mode. Is swap disabled? : false, Address space adequate? : true, nofile limit adequate? : true, nproc limit adequate? : true
INFO [main] 2016-12-01 14:56:19,893 QueryProcessor.java:113 - Initialized prepared statement caches with 10 MB (native) and 10 MB (Thrift)
INFO [main] 2016-12-01 14:56:20,523 ColumnFamilyStore.java:412 - Initializing system.IndexInfo
INFO [SSTableBatchOpen:1] 2016-12-01 14:56:21,449 BufferPool.java:229 - Global buffer pool is enabled, when pool is exhausted (max is 512.000MiB) it will allocate on heap
INFO [main] 2016-12-01 14:56:21,790 CacheService.java:113 - Initializing key cache with capacity of 100 MBs.
INFO [main] 2016-12-01 14:56:21,811 CacheService.java:135 - Initializing row cache with capacity of 0 MBs
INFO [main] 2016-12-01 14:56:21,918 CacheService.java:164 - Initializing counter cache with capacity of 50 MBs
INFO [main] 2016-12-01 14:56:21,922 CacheService.java:175 - Scheduling counter cache save to every 7200 seconds (going to save all keys).
INFO [main] 2016-12-01 14:56:21,983 ColumnFamilyStore.java:412 - Initializing system.batches
INFO [main] 2016-12-01 14:56:22,004 ColumnFamilyStore.java:412 - Initializing system.paxos
INFO [main] 2016-12-01 14:56:22,035 ColumnFamilyStore.java:412 - Initializing system.local
INFO [main] 2016-12-01 14:56:22,057 ColumnFamilyStore.java:412 - Initializing system.peers
INFO [main] 2016-12-01 14:56:22,070 ColumnFamilyStore.java:412 - Initializing system.peer_events
INFO [main] 2016-12-01 14:56:22,079 ColumnFamilyStore.java:412 - Initializing system.range_xfers
INFO [main] 2016-12-01 14:56:22,089 ColumnFamilyStore.java:412 - Initializing system.compaction_history
INFO [main] 2016-12-01 14:56:22,118 ColumnFamilyStore.java:412 - Initializing system.sstable_activity
INFO [main] 2016-12-01 14:56:22,145 ColumnFamilyStore.java:412 - Initializing system.size_estimates
INFO [main] 2016-12-01 14:56:22,173 ColumnFamilyStore.java:412 - Initializing system.available_ranges
INFO [main] 2016-12-01 14:56:22,188 ColumnFamilyStore.java:412 - Initializing system.views_builds_in_progress
INFO [main] 2016-12-01 14:56:22,218 ColumnFamilyStore.java:412 - Initializing system.built_views
INFO [main] 2016-12-01 14:56:22,238 ColumnFamilyStore.java:412 - Initializing system.hints
INFO [main] 2016-12-01 14:56:22,261 ColumnFamilyStore.java:412 - Initializing system.batchlog
INFO [main] 2016-12-01 14:56:22,269 ColumnFamilyStore.java:412 - Initializing system.schema_keyspaces
INFO [main] 2016-12-01 14:56:22,277 ColumnFamilyStore.java:412 - Initializing system.schema_columnfamilies
INFO [main] 2016-12-01 14:56:22,284 ColumnFamilyStore.java:412 - Initializing system.schema_columns
INFO [main] 2016-12-01 14:56:22,292 ColumnFamilyStore.java:412 - Initializing system.schema_triggers
INFO [main] 2016-12-01 14:56:22,300 ColumnFamilyStore.java:412 - Initializing system.schema_usertypes
INFO [main] 2016-12-01 14:56:22,308 ColumnFamilyStore.java:412 - Initializing system.schema_functions
INFO [main] 2016-12-01 14:56:22,315 ColumnFamilyStore.java:412 - Initializing system.schema_aggregates
INFO [main] 2016-12-01 14:56:22,317 ViewManager.java:137 - Not submitting build tasks for views in keyspace system as storage service is not initialized
INFO [main] 2016-12-01 14:56:22,700 JMXServerUtils.java:111 - Configured JMX server at: service:jmx:rmi://127.0.0.1/jndi/rmi://127.0.0.1:7199/jmxrmi
INFO [main] 2016-12-01 14:56:23,382 StorageService.java:535 - Populating token metadata from system tables
INFO [main] 2016-12-01 14:56:23,406 StorageService.java:542 - Token metadata: Normal Tokens:
INFO [main] 2016-12-01 14:56:23,422 ColumnFamilyStore.java:412 - Initializing system_schema.keyspaces
INFO [main] 2016-12-01 14:56:23,470 ColumnFamilyStore.java:412 - Initializing system_schema.tables
INFO [main] 2016-12-01 14:56:23,493 ColumnFamilyStore.java:412 - Initializing system_schema.columns
INFO [main] 2016-12-01 14:56:23,512 ColumnFamilyStore.java:412 - Initializing system_schema.triggers
INFO [main] 2016-12-01 14:56:23,529 ColumnFamilyStore.java:412 - Initializing system_schema.dropped_columns
INFO [main] 2016-12-01 14:56:23,574 ColumnFamilyStore.java:412 - Initializing system_schema.views
INFO [main] 2016-12-01 14:56:23,596 ColumnFamilyStore.java:412 - Initializing system_schema.types
INFO [main] 2016-12-01 14:56:23,615 ColumnFamilyStore.java:412 - Initializing system_schema.functions
INFO [main] 2016-12-01 14:56:23,626 ColumnFamilyStore.java:412 - Initializing system_schema.aggregates
INFO [main] 2016-12-01 14:56:23,653 ColumnFamilyStore.java:412 - Initializing system_schema.indexes
INFO [main] 2016-12-01 14:56:23,669 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_schema as storage service is not initialized
INFO [main] 2016-12-01 14:56:23,950 ColumnFamilyStore.java:412 - Initializing hotel.amenities_by_room
INFO [main] 2016-12-01 14:56:23,956 ColumnFamilyStore.java:412 - Initializing hotel.available_rooms_by_hotel_date
INFO [main] 2016-12-01 14:56:23,962 ColumnFamilyStore.java:412 - Initializing hotel.hotels
INFO [main] 2016-12-01 14:56:23,967 ColumnFamilyStore.java:412 - Initializing hotel.hotels_by_poi
INFO [main] 2016-12-01 14:56:23,973 ColumnFamilyStore.java:412 - Initializing hotel.pois_by_hotel
INFO [main] 2016-12-01 14:56:23,974 ViewManager.java:137 - Not submitting build tasks for views in keyspace hotel as storage service is not initialized
INFO [main] 2016-12-01 14:56:23,981 ColumnFamilyStore.java:412 - Initializing system_distributed.parent_repair_history
INFO [main] 2016-12-01 14:56:23,986 ColumnFamilyStore.java:412 - Initializing system_distributed.repair_history
INFO [main] 2016-12-01 14:56:23,992 ColumnFamilyStore.java:412 - Initializing system_distributed.view_build_status
INFO [main] 2016-12-01 14:56:24,030 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_distributed as storage service is not initialized
INFO [main] 2016-12-01 14:56:24,037 ColumnFamilyStore.java:412 - Initializing reservation.guests
INFO [main] 2016-12-01 14:56:24,043 ColumnFamilyStore.java:412 - Initializing reservation.reservations_by_guest
INFO [main] 2016-12-01 14:56:24,048 ColumnFamilyStore.java:412 - Initializing reservation.reservations_by_hotel_date
INFO [main] 2016-12-01 14:56:24,052 ColumnFamilyStore.java:412 - Initializing reservation.reservations_by_confirmation
INFO [main] 2016-12-01 14:56:24,054 ViewManager.java:137 - Not submitting build tasks for views in keyspace reservation as storage service is not initialized
INFO [main] 2016-12-01 14:56:24,059 ColumnFamilyStore.java:412 - Initializing system_auth.resource_role_permissons_index
INFO [main] 2016-12-01 14:56:24,064 ColumnFamilyStore.java:412 - Initializing system_auth.role_members
INFO [main] 2016-12-01 14:56:24,069 ColumnFamilyStore.java:412 - Initializing system_auth.role_permissions
INFO [main] 2016-12-01 14:56:24,074 ColumnFamilyStore.java:412 - Initializing system_auth.roles
INFO [main] 2016-12-01 14:56:24,087 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_auth as storage service is not initialized
INFO [main] 2016-12-01 14:56:24,093 ColumnFamilyStore.java:412 - Initializing system_traces.events
INFO [main] 2016-12-01 14:56:24,137 ColumnFamilyStore.java:412 - Initializing system_traces.sessions
INFO [main] 2016-12-01 14:56:24,190 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_traces as storage service is not initialized
INFO [main] 2016-12-01 14:56:24,204 ColumnFamilyStore.java:412 - Initializing my_keyspace.user
INFO [main] 2016-12-01 14:56:24,222 ColumnFamilyStore.java:412 - Initializing my_keyspace.user.user_address_idx
INFO [main] 2016-12-01 14:56:24,387 ColumnFamilyStore.java:412 - Initializing my_keyspace.user.user_emails_idx
INFO [main] 2016-12-01 14:56:24,448 ColumnFamilyStore.java:412 - Initializing my_keyspace.user.user_last_name_idx
INFO [main] 2016-12-01 14:56:24,501 ColumnFamilyStore.java:412 - Initializing my_keyspace.user.user_phone_idx
INFO [main] 2016-12-01 14:56:24,556 ViewManager.java:137 - Not submitting build tasks for views in keyspace my_keyspace as storage service is not initialized
INFO [pool-4-thread-1] 2016-12-01 14:56:24,561 AutoSavingCache.java:196 - reading saved cache /var/lib/cassandra/saved_caches/KeyCache-e.db
INFO [pool-4-thread-1] 2016-12-01 14:56:24,575 AutoSavingCache.java:172 - Completed loading (15 ms; 43 keys) KeyCache cache
INFO [main] 2016-12-01 14:56:24,592 CommitLog.java:166 - Replaying /var/lib/cassandra/commitlog/CommitLog-6-1480266920262.log, /var/lib/cassandra/commitlog/CommitLog-6-1480266920263.log, /var/lib/cassandra/commitlog/CommitLog-6-1480428373824.log, /var/lib/cassandra/commitlog/CommitLog-6-1480430610402.log, /var/lib/cassandra/commitlog/CommitLog-6-1480446994136.log, /var/lib/cassandra/commitlog/CommitLog-6-1480447217591.log, /var/lib/cassandra/commitlog/CommitLog-6-1480447657364.log, /var/lib/cassandra/commitlog/CommitLog-6-1480447697557.log, /var/lib/cassandra/commitlog/CommitLog-6-1480448913585.log, /var/lib/cassandra/commitlog/CommitLog-6-1480449134204.log, /var/lib/cassandra/commitlog/CommitLog-6-1480449329313.log, /var/lib/cassandra/commitlog/CommitLog-6-1480527540752.log, /var/lib/cassandra/commitlog/CommitLog-6-1480584423323.log, /var/lib/cassandra/commitlog/CommitLog-6-1480594224164.log, /var/lib/cassandra/commitlog/CommitLog-6-1480594243517.log, /var/lib/cassandra/commitlog/CommitLog-6-1480594321703.log, /var/lib/cassandra/commitlog/CommitLog-6-1480594441672.log, /var/lib/cassandra/commitlog/CommitLog-6-1480594635525.log, /var/lib/cassandra/commitlog/CommitLog-6-1480596178586.log, /var/lib/cassandra/commitlog/CommitLog-6-1480596689253.log
ERROR [main] 2016-12-01 14:56:24,878 JVMStabilityInspector.java:82 - Exiting due to error while processing commit log during initialization.
org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException: Unexpected error deserializing mutation; saved to /tmp/mutation2085286393633679945dat. This may be caused by replaying a mutation against a table with the same name but incompatible schema. Exception follows: org.apache.cassandra.serializers.MarshalException: Not enough bytes to read 0th field street
at org.apache.cassandra.db.commitlog.CommitLogReader.readMutation(CommitLogReader.java:410)
at org.apache.cassandra.db.commitlog.CommitLogReader.readSection(CommitLogReader.java:343)
at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:202)
at org.apache.cassandra.db.commitlog.CommitLogReader.readAllFiles(CommitLogReader.java:85)
jk@tiira:~$
我该如何解决这个问题？
最佳答案
您有一些损坏的 sstable，但由于它是二级索引，因此修复起来非常简单。二级索引实际上只是本地数据的非规范化，因此所有数据都在您的 user 表中，没有任何丢失。
只需删除 user_address_idx 数据目录中的 sstable，然后使用 nodetool rebuild_index my_keyspace user user_address_idx 重建它们即可。
关于Cassandra启动失败: SSTableBatchOpen IllegalArgumentException，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/40876443/