但是multi-master –> slave的repliction确实非常有用,例如一台集中备份机备份所有master的数据。
实现multi-master有几种思路:
1. 修改mysql源码:修改sql_yacc.yy, sql_lex.cc支持多master的change master to语法,然后修改slave相关的slave.cc,支持开启多个slave, 将slave io/ slave thread线程扩展为一个slave_list。
2. 利用mysqlbinlog之类的工具,远程注册到master获取binlog,导入本地slave服务器。
从效率看,肯定第一种方式效率高,但是风险太大了,并且mysql版本更新,可能需要变动自己的代码以适应新的mysql source, mysql官方的实现方式肯定是第一种,从源码中的注释可以看出他们的设计思路。但是他们考虑的问题可能是多个master复制如何处理冲突等异常,因而迟迟不发布。
为了避免过多的入侵mysql,我采用第二种方式,用一个脚本或者程序等等,去调用mysqlbinlog,用-r远程请求到–to-last-log,然后稍微修改一下啊mysqlbinlog的源码,在日志切换后计数一下,在输出文件末尾打上切换日志的个数,例如:
insert into a values (8)/*!*/; # at 1070#110114 16:16:11 server id 3 end_log_pos 1097 xid = 36commit/*!*/; delimiter ; # end of log file rollback /* added by mysqlbinlog */;/*!50003 set completion_type=@old_completion_type*/;-- rorate binlog count: 1
— rorate binlog count: 1就是日志切换信息,表示切换了一次日志(即传入master的日志号没有用完)然后tail末尾的end_pos来查看本次同步到哪里了,写到*.info的文件中。
我的脚本需要配置一个multi_master.conf文件,配好每个master的信息,例如:
#cat multi_master.conf [master1]master_host=1.2.3.4 master_user=plx master_password=plx master_port=3306master_log_name=mysql-bin master_log_idx=000002 master_log_pos=521relay_log_dir=/tmp/relay_log_name=1-relay-bin [master2]master_host=2.3.4.5 master_user=plx master_password=plx master_port=3306master_log_name=mysql-bin master_log_idx=000002 master_log_pos=581relay_log_dir=/tmp/relay_log_name=2-relay-bin [slave]slave_user=plx slave_password=plx
slave默认导入本地,所以没有提供主机选项。
配置文件的含义是,定义了master1和master2两个master,名称其实只要不是slave都行,[slave]中定义了本地导入的用户名和密码。
特有的参数我解释下,没解释的跟mysql一样,
master_log_name和master_log_idx组成mysql中的master_log_file,relay_log_dir表示取回的binlog文件放哪个目录,relay_log_name是relay文件的文件名,会加上标号,跟mysql一样,这个脚本会自动处理。
一旦执行过一次,就会生成master1.info之类的文件,来表示当前同步到哪里了,例如下面这个例子:
master_log_pos=1482name=master1 master_user=plx relay_log_name=1-relay-bin master_log_idx=2master_host=1.2.3.4 master_log_name=mysql-bin master_port=3306relay_log_dir=/tmp/master_password=plx relay_log_idx=3
只有找不到*.info的时候,才会使用multi_master.conf。
现在每次调度multi_master_repl.pl都只会运行一次,可以不断的调度multi_master_repl.pl,因为还没有完全搞定kill信号在perl脚本的处理,用c重写后会解决,不能暴力kill -9,会导致不知道复制到哪里了。
这是下载地址,切勿用在生产环境,这只是个验证想法的程序。
note: there is a file embedded within this post, please visit this post to download the file.
下一步我想用c重新实现,在mysqlbinlog源码基础上修改,获取到的日志直接写入到sock或直接导入远程mysql,避免多写一次文件,也欢迎提供新思路。
这是一次执行的日志:
#./multi_master_repl.pl (debug) enter: get_config() info: begin (debug) get_config --> master1 (debug) get_config --> multi_master.conf --> master1:master_host=1.2.3.4 (debug) get_config --> multi_master.conf --> master1:master_user=plx (debug) get_config --> multi_master.conf --> master1:master_password=plx (debug) get_config --> multi_master.conf --> master1:master_port=3306 (debug) get_config --> multi_master.conf --> master1:master_log_name=mysql-bin (debug) get_config --> multi_master.conf --> master1:master_log_idx=000002 (debug) get_config --> multi_master.conf --> master1:master_log_pos=521 (debug) get_config --> multi_master.conf --> master1:relay_log_dir=/tmp/ (debug) get_config --> multi_master.conf --> master1:relay_log_name=1-relay-bin (debug) get_config --> found master1.info, read it (debug) get_config --> master1.info --> master1:master_log_pos=1097 (debug) get_config --> master1.info --> master1:name=master1 (debug) get_config --> master1.info --> master1:master_user=plx (debug) get_config --> master1.info --> master1:relay_log_name=1-relay-bin (debug) get_config --> master1.info --> master1:master_log_idx=2 (debug) get_config --> master1.info --> master1:master_host=1.2.3.4 (debug) get_config --> master1.info --> master1:master_log_name=mysql-bin (debug) get_config --> master1.info --> master1:master_port=3306 (debug) get_config --> master1.info --> master1:relay_log_dir=/tmp/ (debug) get_config --> master1.info --> master1:master_password=plx (debug) get_config --> master1.info --> master1:relay_log_idx=2 (debug) get_config --> push[master1] to master_info_list (debug) get_config --> master2 (debug) get_config --> multi_master.conf --> master2:master_host=2.3.4.5 (debug) get_config --> multi_master.conf --> master2:master_user=plx (debug) get_config --> multi_master.conf --> master2:master_password=plx (debug) get_config --> multi_master.conf --> master2:master_port=3306 (debug) get_config --> multi_master.conf --> master2:master_log_name=mysql-bin (debug) get_config --> multi_master.conf --> master2:master_log_idx=000002 (debug) get_config --> multi_master.conf --> master2:master_log_pos=581 (debug) get_config --> multi_master.conf --> master2:relay_log_dir=/tmp/ (debug) get_config --> multi_master.conf --> master2:relay_log_name=2-relay-bin (debug) get_config --> found master2.info, read it (debug) get_config --> master2.info --> master2:master_log_pos=1541 (debug) get_config --> master2.info --> master2:name=master2 (debug) get_config --> master2.info --> master2:master_user=plx (debug) get_config --> master2.info --> master2:relay_log_name=2-relay-bin (debug) get_config --> master2.info --> master2:master_log_idx=2 (debug) get_config --> master2.info --> master2:master_host=2.3.4.5 (debug) get_config --> master2.info --> master2:master_log_name=mysql-bin (debug) get_config --> master2.info --> master2:master_port=3306 (debug) get_config --> master2.info --> master2:relay_log_dir=/tmp/ (debug) get_config --> master2.info --> master2:master_password=plx (debug) get_config --> master2.info --> master2:relay_log_idx=2 (debug) get_config --> push[master2] to master_info_list (debug) get_config --> multi_master.conf --> slave:slave_user=plx (debug) get_config --> multi_master.conf --> slave:slave_password=plx (debug) enter: get_config() info: exit (debug) enter: create_slave_threads() info: begin (debug) create_slave_threads --> creating run_slave threads... (debug) enter: run_slave() info: begin [tid: 1] (debug) run_slave(0) --> no kill signal --> g_is_killed =>0 (debug) run_slave --> mysqlbinlog: ./mysqlbinlog -h1.2.3.4 -uplx -pplx -r -t --start-position=1097 mysql-bin.000002 > /tmp/1-relay-bin.000002 warning: ./mysqlbinlog: unknown variable 'loose_default-character-set=utf8' (debug) run_slave(0) --> no kill signal --> g_is_killed =>0 (debug) enter: import_to_slave() info: begin [param: p_master_idx=>0] (debug) import_to_slave(0) --> no kill signal --> g_is_killed =>0 (debug) import_to_slave(0) --> importing relay log /tmp/1-relay-bin.000002 to slave... (debug) create_slave_threads --> created 2 run_slave threads (debug) enter: run_slave() info: begin [tid: 2] (debug) run_slave(1) --> no kill signal --> g_is_killed =>0 (debug) run_slave --> mysqlbinlog: ./mysqlbinlog -h2.3.4.5 -uplx -pplx -r -t --start-position=1541 mysql-bin.000002 > /tmp/2-relay-bin.000002 warning: ./mysqlbinlog: unknown variable 'loose_default-character-set=utf8' (debug) run_slave(1) --> no kill signal --> g_is_killed =>0 (debug) enter: import_to_slave() info: begin [param: p_master_idx=>1] (debug) import_to_slave(1) --> no kill signal --> g_is_killed =>0 (debug) import_to_slave(1) --> importing relay log /tmp/2-relay-bin.000002 to slave... (debug) enter: update_master_info() info: begin [param: p_master_idx=>0] (debug) enter: update_master_info() info: begin [param: p_master_idx=>1] (debug) update_master_info(0) --> now master-log is mysql-bin.000002 pos is 1482 (debug) enter: update_master_info_file() info: begin [param: p_master_idx=>0] (debug) update_master_info_file(0) --> no kill signal --> g_is_killed =>0 (debug) update_master_info_file(0) --> writing master1.info --> master_log_pos=1482 (debug) update_master_info_file(0) --> writing master1.info --> name=master1 (debug) update_master_info_file(0) --> writing master1.info --> master_user=plx (debug) update_master_info_file(0) --> writing master1.info --> relay_log_name=1-relay-bin (debug) update_master_info_file(0) --> writing master1.info --> master_log_idx=2 (debug) update_master_info_file(0) --> writing master1.info --> master_host=1.2.3.4 (debug) update_master_info_file(0) --> writing master1.info --> master_log_name=mysql-bin (debug) update_master_info_file(0) --> writing master1.info --> master_port=3306 (debug) update_master_info_file(0) --> writing master1.info --> relay_log_dir=/tmp/ (debug) update_master_info_file(0) --> writing master1.info --> master_password=plx (debug) update_master_info_file(0) --> writing master1.info --> relay_log_idx=3 (debug) update_master_info_file(0) --> created master1.info (debug) enter: update_master_info_file(0) info: exit (debug) enter: update_master_info(0) info: exit (debug) enter: import_to_slave(0) info: exit (debug) enter: run_slave(0) info: exit (debug) update_master_info(1) --> now master-log is mysql-bin.000002 pos is 2120 (debug) enter: update_master_info_file() info: begin [param: p_master_idx=>1] (debug) update_master_info_file(1) --> no kill signal --> g_is_killed =>0 (debug) update_master_info_file(1) --> writing master2.info --> master_log_pos=2120 (debug) update_master_info_file(1) --> writing master2.info --> name=master2 (debug) update_master_info_file(1) --> writing master2.info --> master_user=plx (debug) update_master_info_file(1) --> writing master2.info --> relay_log_name=2-relay-bin (debug) update_master_info_file(1) --> writing master2.info --> master_log_idx=2 (debug) update_master_info_file(1) --> writing master2.info --> master_host=2.3.4.5 (debug) update_master_info_file(1) --> writing master2.info --> master_log_name=mysql-bin (debug) update_master_info_file(1) --> writing master2.info --> master_port=3306 (debug) update_master_info_file(1) --> writing master2.info --> relay_log_dir=/tmp/ (debug) update_master_info_file(1) --> writing master2.info --> master_password=plx (debug) update_master_info_file(1) --> writing master2.info --> relay_log_idx=3 (debug) update_master_info_file(1) --> created master2.info (debug) enter: update_master_info_file(1) info: exit (debug) enter: update_master_info(1) info: exit (debug) enter: import_to_slave(1) info: exit (debug) enter: run_slave(1) info: exit (debug) enter: create_slave_threads() info: exit
