加快mydumper与myloader导出导入

有关mydumper与myloader其他事项可以参考：
mydumper安装及安装故障汇总
mydumper备份mysql数据库示例
myloader恢复mysql数据库示例

1、基于MyIsam引擎导出导入
a、表不分块导出及导入
[root@GZAPP tmp]# mydumper -u inno -p xxx -B bsom -T tb_access_log -o /backup/tmp/
[root@GZAPP tmp]# ls -hltr ###导出的数据文件为单个文件，大小在2.6GB
total 2.6G
-rw-r--r-- 1 root root 1.6K Jul 24 08:51 bsom.tb_access_log-schema.sql
-rw-r--r-- 1 root root 214 Jul 24 08:52 metadata
-rw-r--r-- 1 root root 2.6G Jul 24 08:52 bsom.tb_access_log.sql

###基于缺省线程数导入，且设定每个事务查询数为10000，此参数此时其实作用不大，因为表为myisam引擎

[root@GZAPP tmp]# myloader -u inno -p xxx -B tempdb -d /backup/tmp -v 3 -q 10000
[root@GZAPP tmp]# myloader -u inno -p xxx -B tempdb -d /backup/tmp -v 3 -q 10000
** Message: 4 threads created
** Message: Creating table `tempdb`.`tb_access_log`
** Message: Thread 4 shutting down
** Message: Thread 1 restoring `bsom`.`tb_access_log` part 0
** Message: Thread 3 shutting down
** Message: Thread 2 shutting down

root@localhost[tempdb]> show processlist;
+---------+---------+----------+--------+---------+--------+---------+------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+---------+---------+----------+--------+---------+--------+---------+------------------------------------------------+
| 4452079 | root | localhost| tempdb | Query | 0 | init | show processlist |
| 4453793 | inno | localhost| tempdb | Sleep | 420 | | NULL |
| 4453794 | inno | localhost| tempdb | Query | 4 | update | INSERT INTO `tb_access_log` VALUES (506873,"325|
+---------+---------+----------+--------+---------+--------+---------+------------------------------------------------+
###从上面的线程数可以看出只有一个单线程在执行insert操作

b、表分块导出及导入
###下面的示例中使用500MB进行分块

[root@GZAPP tmp]# mydumper -u inno -p xxx -B bsom -T tb_access_log -F 500 -o /backup/tmp/
[root@GZAPP tmp]# ls -hltr
total 2.6G      
-rw-r--r-- 1 root root 1.6K Jul 24 08:21 bsom.tb_access_log-schema.sql
-rw-r--r-- 1 root root 478M Jul 24 08:21 bsom.tb_access_log.00001.sql
-rw-r--r-- 1 root root 478M Jul 24 08:21 bsom.tb_access_log.00002.sql
-rw-r--r-- 1 root root 478M Jul 24 08:21 bsom.tb_access_log.00003.sql
-rw-r--r-- 1 root root 478M Jul 24 08:21 bsom.tb_access_log.00004.sql
-rw-r--r-- 1 root root 478M Jul 24 08:22 bsom.tb_access_log.00005.sql
-rw-r--r-- 1 root root  214 Jul 24 08:22 metadata
-rw-r--r-- 1 root root 241M Jul 24 08:22 bsom.tb_access_log.00006.sql

###由上可知，大表tb_access_log按接近500M被分割成了多个文件

[root@GZAPP tmp]# myloader -u inno -p xxx -B tempdb -t 6 -d /backup/tmp -v 3
** Message: 6 threads created
** Message: Creating database `tempdb`
** Message: Creating table `tempdb`.`tb_access_log`
** Message: Thread 1 restoring `bsom`.`tb_access_log` part 3
** Message: Thread 2 restoring `bsom`.`tb_access_log` part 5
** Message: Thread 5 restoring `bsom`.`tb_access_log` part 4
** Message: Thread 3 restoring `bsom`.`tb_access_log` part 6
** Message: Thread 4 restoring `bsom`.`tb_access_log` part 1
** Message: Thread 6 restoring `bsom`.`tb_access_log` part 2

#在下面的processlist可以看到，存在表级锁等待
+---------+-------+-----------+---------+---------+--------+-----------------------------+------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+---------+-------+-----------+---------+---------+--------+-----------------------------+------------------------------------------------+
| 4452079 | root | localhost | bsom | Query | 0 | init | show processlist |
| 4452167 | inno | localhost | tempdb | Sleep | 769 | | NULL |
| 4452168 | inno | localhost | tempdb | Query | 36 | update | INSERT INTO `tb_access_log` VALUES (6367402,"0,|
| 4452169 | inno | localhost | tempdb | Query | 21 | Waiting for table level lock| INSERT INTO `tb_access_log` VALUES (12593865," |
| 4452170 | inno | localhost | tempdb | Query | 26 | Waiting for table level lock| INSERT INTO `tb_access_log` VALUES (15643029,""|
| 4452171 | inno | localhost | tempdb | Query | 6 | Waiting for table level lock| INSERT INTO `tb_access_log` VALUES (173947,"70 |
| 4452172 | inno | localhost | tempdb | Query | 15 | Waiting for table level lock| INSERT INTO `tb_access_log` VALUES (9490507,"7 |
| 4452173 | inno | localhost | tempdb | Query | 30 | Waiting for table level lock| INSERT INTO `tb_access_log` VALUES (3271602,"4 |
+---------+---------+-----------+---------+---------+--------+-----------------------------+----------------------------------------------+

c、调整myisam有关参数后导入

[root@GZ-APP-BAK01 tmp]# time myloader -u innobk -p InnoBK -B tempdb -t 6 -d /backup/tmp -v 3
** Message: 6 threads created
** Message: Creating table `tempdb`.`tb_mobile_access_log`
** Message: Thread 1 restoring `blossom`.`tb_mobile_access_log` part 3
** Message: Thread 6 restoring `blossom`.`tb_mobile_access_log` part 6
** Message: Thread 2 restoring `blossom`.`tb_mobile_access_log` part 5
** Message: Thread 3 restoring `blossom`.`tb_mobile_access_log` part 4
** Message: Thread 4 restoring `blossom`.`tb_mobile_access_log` part 1
** Message: Thread 5 restoring `blossom`.`tb_mobile_access_log` part 2
** Message: Thread 6 shutting down
** Message: Thread 5 shutting down
** Message: Thread 1 shutting down
** Message: Thread 2 shutting down
** Message: Thread 4 shutting down
** Message: Thread 3 shutting down

real 266m28.903s
user 0m6.008s
sys 0m1.681s

###调整以下相关参数，后尝试再次导入，
concurrent_insert AUTO 改成 ALWAYS
bulk_insert_buffer_size 8388608 改成 256M
myisam_sort_buffer_size 67108864 改成 128M

[root@GZ-APP-BAK01 tmp]# time myloader -u innobk -p InnoBK -B tempdb -t 6 -o -d /backup/tmp -v 3
** Message: 6 threads creat