Tungsten Replication from Amazon RDS


//=== http://docs.continuent.com/tungsten-replicator-3.0/deployment-fromamazonrds.html

Roles: RDS instance, Tungsten Replicator, destination database instance(dest_db)

* no ssh access to RDS instance is provided yet?

* install Tungsten Replicator on a certain host
The replicator can be installed either within a standard (non-RDS) Amazon EC2 instance
or
on another host which can write to the dest_db

* read bin_log from RDS, transfer format and write to dest_db
- "Service Alpha" extracts the binary log information directly from RDS instance and loads into THL.
-- ("Service Alpha" is a utility running on Tungsten Replicator)
-- (THL: Transaction History Log )

- Tungsten Replicator then writes the information from the THL and writes the data to the target database.



//=== http://docs.continuent.com/tungsten-replicator-3.0/deployment-fromamazonrds-preparation.html

dest_db data format prerequisites : [take mysql for example]

* Table format should be updated to UTF8 by updating the MySQL configuration (/etc/mysql/my.cnf):
- character-set-server=utf8
- collation-server=utf8_general_ci

* To prevent timezone configuration storing zone adjusted values,
fix the timezone configuration to use UTC within the configuration file (my.cnf):
- default-time-zone='+00:00'


RDS instance config
http://docs.continuent.com/tungsten-replicator-3.0/deployment-fromamazonrds-rdsconfig.html

to change RDS config by RDS client
http://docs.continuent.com/tungsten-replicator-3.0/deployment-amazonrds-rdsconfig.html


//=== download and install Tungsten
http://docs.continuent.com/tungsten-replicator-3.0/deployment-fromamazonrds-installation.html

* Download the latest version of Tungsten Replicator.
https://code.google.com/p/tungsten-replicator/wiki/Downloads


Expand the release:
shell> tar zxf tungsten-replicator-3.0.1-64

Change to the staging directory:
shell> cd tungsten-replicator-3.0.1-64


Run tpm to install the replicator:
[shd create repli-user first]

shell> ./tools/tpm install alpha \
--install-directory=/opt/continuent \
--replication-user=tungsten \
--replication-password=secret \
--replication-port=3306 \
--direct-replication-port=3306 \
--privileged-master=false \
--skip-validation-check=MySQLDumpCheck \
--topology=direct \
--master=db1 \
--direct-datasource-host=xxxx.cnlhon44f2wq.eu-west-1.rds.amazonaws.com \
--direct-datasource-user=rds_user \
--direct-datasource-password=rds_password \
--start-and-report



//=== http://docs.continuent.com/continuent-tungsten-4.0/terminology-thl.html
Tungsten Replicator --> Continuent-Tungsten

THL: Transaction History Log

"... stores transactional data from different data servers in a universal format
that is then used to exchange and transfer the information between replicator instances.

Because the THL is stored and 'independently' managed from the data servers that it reads and writes,
the data can be moved, exchanged, and transmuted during processing.
..."""




.


mysql create user


mysql> CREATE USER 'repl'@'%.myslaves.com' IDENTIFIED BY 'slavepass';

//=== https://dev.mysql.com/doc/refman/5.1/en/create-user.html
mysql> CREATE USER 'jeff'@'localhost' IDENTIFIED BY 'mypass';

* to enable the user to connect with no password, remove "IDENTIFIED BY" clause:
mysql> CREATE USER 'jeff'@'localhost';

* to assign the hashed password directly, use "IDENTIFIED BY PASSWORD" clause:
mysql> CREATE USER 'jeff'@'localhost' IDENTIFIED BY PASSWORD '*90Exfydsajd12324959dfare5ere895989';




show granted privileges for a certain user in mysql


https://dev.mysql.com/doc/refman/5.0/en/show-grants.html

mysql> SHOW GRANTS FOR 'root'@'localhost';
mysql> SHOW GRANTS;
mysql> SHOW GRANTS FOR CURRENT_USER;
mysql> SHOW GRANTS FOR CURRENT_USER();

mysql> select user,host from mysql.user;
mysql> show grants for repli_user@3hd.me ;

GRANT REPLICATION SLAVE ON *.* TO 'repli_user'@'3hd.me' IDENTIFIED BY PASSWORD 'xvf1E7fda4dsf9xxxxx'


Tungsten Replicator for heterogeneous databases


http://docs.continuent.com/tungsten-replicator-3.0/deployment-hetero-operation.html

*** All heterogeneous replication deployments use row-based replication.

"""...
Heterogeneous replication works slightly differently compared to the native MySQL to MySQL replication.
... The SQL dialects are different, so that an SQL statement on MySQL is not the same as an SQL statement on Oracle, and differences in the dialects mean that either the statement would fail, or would perform an incorrect operation.

On targets that do not support SQL of any kind, such as MongoDB, replicating SQL statements would achieve nothing since they cannot be executed at all.

All heterogeneous replication deployments therefore use row-based replication.
This extracts only the raw row data, not the statement information.
... raw-data, it can be easily re-assembled or constructed into another format,
including statements in other SQL dialects,
native appliers for alternative formats, such as JSON or BSON, or external CSV formats ...

..."""

* Native applier:
to replicate to MongoDB, data is reformatted into BSON and then applied to MongoDB
using the native insert/update/delete API calls.






nodejs fork, spawn, exec


http://www.hacksparrow.com/difference-between-spawn-and-exec-of-node-js-child_process.html

Difference between spawn and exec of Node.js child_process
"""...
The most significant difference between child_process.spawn and child_process.exec is
in what they return -
spawn returns a stream and
exec returns a buffer.

... child_process.spawn is "asynchronously asynchronous",
meaning it starts sending back data ... as soon as the child process starts executing


...
child_process.exec returns the whole buffer output from the child process.
By default the buffer size is set at 200k.

If the child process returns anything more than that, your program will
crash with the error message "Error: maxBuffer exceeded".

... child_process.exec is "synchronously asynchronous", ...
it waits for the child process to end and tries to return all the buffered data at once



..."""



nodejs exec and execFile


https://nodejs.org/api/child_process.html#child_process_child_process_exec_command_options_callback

* child_process.exec(command[, options], callback)
* child_process.execFile(file[, args][, options][, callback])


child_process.execFile() is similar to child_process.exec() except it does not execute a subshell
but rather the specified file directly.
This makes it slightly leaner than child_process.exec.


* child_process.fork(modulePath[, args][, options])
This is a special case of the spawn() functionality for spawning Node processes.

In addition to having all the methods in a normal ChildProcess instance,
the returned object has a communication channel built-in.

These child Nodes are still whole new instances of V8.
Assume at least 30ms startup and 10mb memory for each new Node.
That is, cannot create many thousands of them.




mysql examples


//===
show index from table1;

ALTER DATABASE db1
DEFAULT CHARACTER SET = 'utf8' DEFAULT COLLATE 'utf8_unicode_ci' ;

alter table phrase add column freq int unsigned;

show create table t1;


http://www.w3schools.com/sql/sql_alter.asp
ALTER TABLE table_name MODIFY COLUMN column_name datatype;
alter table phrase modify column val char(8);

//=== mysql data type char(8)
varchar(500)
char(254)


https://dev.mysql.com/doc/refman/5.0/en/string-type-overview.html

CREATE TABLE t
(
c1 VARCHAR(20) CHARACTER SET utf8,
c2 TEXT CHARACTER SET latin1 COLLATE latin1_general_cs
);



//=== http://www.w3schools.com/sql/sql_unique.asp
CREATE TABLE Persons
(
P_Id int NOT NULL,
LastName varchar(255) NOT NULL,
FirstName varchar(255),
Address varchar(255),
City varchar(255),
UNIQUE (P_Id)
)

ALTER TABLE Persons
ADD UNIQUE (P_Id)


ALTER TABLE Persons
DROP INDEX uc_PersonID



//=== http://dev.mysql.com/doc/refman/5.1/en/create-table.html
CREATE TABLE new_tbl LIKE orig_tbl;
CREATE TABLE new_tbl SELECT * FROM orig_tbl;

CREATE TABLE IF NOT EXISTS t1 (
`id` int unsigned NOT NULL AUTO_INCREMENT,
`val` varchar(200) NOT NULL,
PRIMARY KEY (`id`),
index `val` (`val`)
) DEFAULT CHARACTER SET = 'utf8' DEFAULT COLLATE 'utf8_general_ci'


//=== mysql datatype int
https://dev.mysql.com/doc/refman/5.0/en/integer-types.html

INTEGER (or INT) , SMALLINT, TINYINT, MEDIUMINT, BIGINT.

http://stackoverflow.com/questions/5562322/difference-between-int-and-int3-data-types-in-my-sql
Difference between “int” and “int(3)” data types in my sql

INT(10) specifies an INT with a display width of 10 digits
and doesn't affect the size of the integer.






//====
http://dba.stackexchange.com/questions/6570/setting-default-charset-collation-for-mysql-database

"""...
CREATE DATABASE IF NOT EXISTS foo
DEFAULT CHARACTER SET = 'utf8' DEFAULT COLLATE 'utf8_general_ci'


ALTER DATABASE foo
DEFAULT CHARACTER SET = 'utf8' DEFAULT COLLATE 'utf8_general_ci'
..."""


http://stackoverflow.com/questions/766809/whats-the-difference-between-utf8-general-ci-and-utf8-unicode-ci

utf8_general_ci , utf8_unicode_ci ,


"""...
You should never, ever use utf8_general_ci: it simply doesn’t work. It’s a throwback to the bad old days of ASCII stooopeeedity from fifty years ago

...
In short: utf8_unicode_ci uses the Unicode Collation Algorithm as defined in the Unicode standards,
whereas utf8_general_ci is a more simple sort order which results in "less accurate" sorting results.


..."""


vim indent for python, ftplugin


[ex1.py]
#!/usr/bin/env python

"""
Python docString :description about ex1.py
"""

# vim: tabstop=8 expandtab shiftwidth=4 softtabstop=4

# ======= write your code below =========

....

# =======================================




//=== https://github.com/tarmack/vim-python-ftplugin
https://github.com/tarmack/vim-python-ftplugin.git

need pyflakes for python syntax check
$ sudo apt-get install pyflakes



//=== https://wiki.python.org/moin/Vim

"""...
A useful addition to Python source files is this comment:
# vim: tabstop=8 expandtab shiftwidth=4 softtabstop=4

This may need the modeline option enabled in your ~/.vimrc file:
set modeline
(In Debian and Ubuntu, for example, the modeline option has been disabled for security reasons.)

The above # vim: ... text, when embedded in a source file, tells Vim that when the file is loaded, tabs are always expanded to spaces and that the width of each tab is four characters. Type the following in command mode to achieve the same effect:

:set tabstop=8 expandtab shiftwidth=4 softtabstop=4

Or:
:set ts=8 et sw=4 sts=4


... An alternative method is adding...

set tabstop=8
set expandtab
set softtabstop=4
set shiftwidth=4
filetype indent on
...to your ~/.vimrc file.
[but this will effect other types of files such as .js, .php, .c ...]


..."""

----->

"""...
add the following to ~/.vim/ftplugin/python.vim. Create the directory and/or file if it does not already exist.

set tabstop=8
set expandtab
set shiftwidth=4
set softtabstop=4

..."""


http://vim.wikia.com/wiki/Keep_your_vimrc_file_clean

"""...
move all the language-specific stuff from your .vimrc file into a file named
.vim/ftplugin/language.vim
(or $HOME/vimfiles/ftplugin/language.vim on Windows).


" File ~/.vimrc
" ($HOME/_vimrc on Windows)
" Global settings for all files (but may be overridden in ftplugin).
set tabstop=2
set shiftwidth=2
set noexpandtab



" File ~/.vim/ftplugin/python.vim
" ($HOME/vimfiles/ftplugin/python.vim on Windows)
" Python specific settings.
setlocal tabstop=4
setlocal shiftwidth=4
setlocal expandtab
setlocal autoindent
setlocal smarttab
setlocal formatoptions=croql


...
if there is a filetype plugin distributed with Vim that you want to completely disable, make your own (perhaps empty) settings file and adding this line:

let b:did_ftplugin = 1


..."""




//=== ftplugin !== ftp plugin

ftplugin: file type plugin
--> http://vim.wikia.com/wiki/File_type_plugins

"""...
A file type plugin (ftplugin) is a script that is run automatically
when Vim detects the type of file when the file is created or opened.

The type can be detected from the file name (for example, file sample.c has file type c),
or from the file contents.


...
to disable all ftplugins, or disable a particular default ftplugin,
see :help :filetype and :help ftplugin-overrule.




..."""



ftp plugin for vim
--> http://www.marksanborn.net/software/modify-remote-files-with-vims-built-in-ftp-plugin/

$ vim ftp://username@yourdomain.com//the/path/to/yourfile.php








vim plugin management, pathogen


$ git clone https://github.com/tpope/vim-pathogen.git
$ cd vim-pathogen/autoload
$ mkdir -p ~/.vim/autoload ~/.vim/bundle
$ cp pathogen.vim ~/.vim/autoload

Add the following to your ~/.vimrc
-- execute pathogen#infect()
-- syntax on
-- filetype plugin indent on


*** install plugins you want to a subdirectory under ~/.vim/bundle,
and they will be added to the 'runtimepath'

[ex] js syntax plugin for vim
$ git clone https://github.com/pangloss/vim-javascript.git ~/.vim/bundle/vim-javascript

or step by step
$ git clone https://github.com/pangloss/vim-javascript.git
$ cp -r vim-javascript ~/.vim/bundle/


*** vim7.4 problem with js regex
If you are stuck on an older version of Vim 7.4 with no way to update,
then simply perform the following commands to fix your current buffer:

:set regexpengine=1
:syntax enable


[ref]
https://github.com/tpope/vim-pathogen
https://github.com/pangloss/vim-javascript