MySQL 分区与更改为 MongoDB

Question

我们在 MySQL 数据库中有 4 个相当大的表。它们大约有 50、35、6 和 5 Gb，其他表没有那么大。这些表充满了分析数据，每 10 分钟由 cron 任务附加一次。随着时间的推移，这些表格将继续增长。

这是一个数据表的架构

CREATE TABLE `instpld` (
  `id` int(20) NOT NULL AUTO_INCREMENT,
  `insID` varchar(100) NOT NULL,
  `dbID` int(10) NOT NULL,
  `type` varchar(1) NOT NULL,
  `timestamp` int(11) NOT NULL,
  `count` text NOT NULL,
  `comment_count` int(10) NOT NULL,
  PRIMARY KEY (`id`),
  KEY `insID` (`insID`(50)),
  KEY `dbID` (`dbID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

我知道字段类型可能会好得多。问题是什么更好 - 在表上添加一些分区或将所有内容切换到 MongoDB 因为它更快？

我正在寻找每个选项的优缺点。

# Misc Settings
# -------------
datadir=/var/lib/mysql
tmpdir=/var/lib/mysqltmp
socket=/var/lib/mysql/mysql.sock
#skip-locking
skip-name-resolve
#table_cache=2048
thread_cache_size=16
back_log=100
max_connect_errors=10000
open-files-limit=20000
interactive_timeout=3600
wait_timeout=600
#max_connections=200
# Added to prevent DNS lookups from causing performance issues
skip-name-resolve

# Set this to change the way MySQL handles validation, data
# conversion, etc. Be careful with this setting as it can
# cause unexpected results and horribly break some applications!
# Note, too, that it can be set per-session and can be hard set
# in stored procedures.
#sql_mode=NO_ENGINE_SUBSTITUTION

# Slow Query Log Settings
# -----------------------
#log-slow-queries=/var/lib/mysqllogs/slow-log
#long_query_time=2
#log-queries-not-using-indexes

# Global, Non Engine-Specific Buffers
# -----------------------------------
max_allowed_packet=16M
tmp_table_size=64M
max_heap_table_size=64M

# Generally, it is unwise to set the query cache to be
# larger than 64-128M as this can decrease performance
# since the penalty for flushing the cache can become
# significant.
query_cache_size=32M
skip-name-resolve

# Set this to change the way MySQL handles validation, data
# conversion, etc. Be careful with this setting as it can
# cause unexpected results and horribly break some applications!
# Note, too, that it can be set per-session and can be hard set
# in stored procedures.
#sql_mode=NO_ENGINE_SUBSTITUTION

# Slow Query Log Settings
# -----------------------
#log-slow-queries=/var/lib/mysqllogs/slow-log
#long_query_time=2
#log-queries-not-using-indexes

# Global, Non Engine-Specific Buffers
# -----------------------------------
max_allowed_packet=16M
tmp_table_size=64M
max_heap_table_size=64M

# Generally, it is unwise to set the query cache to be
# larger than 64-128M as this can decrease performance
# since the penalty for flushing the cache can become
# significant.
query_cache_size=32M

# Per-Thread Buffers
# ------------------
sort_buffer_size=1M
read_buffer_size=1M
read_rnd_buffer_size=8M
join_buffer_size=1M
key_buffer_size=64M

# This setting controls the size of the buffer that is allocated when
# sorting MyISAM indexes during a REPAIR TABLE or when creating indexes
# with CREATE INDEX or ALTER TABLE.
myisam_sort_buffer_size=64M

# InnoDB
# ------
# Note: While most settings in MySQL can be set at run-time, InnoDB
# variables require restarting MySQL to apply.

# If the customer already has InnoDB tables and wants to change the
# size of the InnoDB tablespace and InnoDB logs, then:
# 1. Run a full backup with mysqldump
# 2. Stop MySQL
# 3. Move current ibdata and ib_logfiles out of /var/lib/mysql
# 4. Uncomment the below innodb_data_file_path and innodb_log_file_size
# 5. Start MySQL (it will recreate new InnoDB files)
# 6. Restore data from backup
#innodb_data_file_path=ibdata1:2000M;ibdata2:10M:autoextend
innodb_log_file_size=100M

innodb_buffer_pool_size=2G

........

Answer 1

您有 128GB，请使用它! innodb_buffer_pool_size=2G -- 更改为大约 70% 的 RAM。

我敢打赌你不能给我看一个使用 KEY instaID (instaID(50)) 的 EXPLAIN。前缀索引几乎总是未使用。

打开慢日志，收集一些数据，运行 pt-query-digest，然后向我们展示“最差”的查询。为其提供 EXPLAIN SELECT ...。

id int(20) NOT NULL AUTO_INCREMENT, -- 希望您不要期待 20 位数字。这将超过 20 亿。

如果我们无法优化您的查询，那么我们将转向数据仓库技术，例如汇总表——它们往往会提供 10 倍的加速。