Another HBase clone: Voldermort project

http://project-voldemort.com/

JBoss tools 3 release candidate

JBoss tools 3 release candidate was released, could be taken from here

Mount new disk on CentOS linux

1. Check what is the new hard disk device name with "fdisk -l", it should be something like /dev/sdb. You can easily identify which is the new drive by running "mount" and finding the drive that exists in "fdisk -l" but is not mounted.

   mount; fdisk -l;
   

2. Create a partition on the new drive, (the sample code assumes the disk is /dev/sdb)

   echo -ne "n\np\n1\n\n\nw\n" | fdisk /dev/sdb1
   

3. Create a filesystem on the new partition, we use ext3 file system.

    mkfs.ext3 /dev/sdb1
   

4. Create a directory where to mount the partition

   mkdir /mount/data2
   

5. Edit /etc/fstab an add a record for the new drive at the end of the file. This will make the server mount the drive automatically after reboot. Mount options (like noatime and nodiratime) can be added as a comma separated list of values after "defaults": "defaults,noatime,nodiratime"

   echo "/dev/sdb1  /mount/data2  ext3  defaults 0 0" >> /etc/fstab

Linux records information about when files were created and last modified as well as when it was last accessed. There is a cost associated with recording the last access time. Linux has a special mount option for file systems called noatime that can be added to each line that addresses one file system in the /etc/fstab file. If a file system has been mounted with this option, reading accesses to the file system will no longer result in an update to the atime information associated with the file. The importance of the noatime setting is that it eliminates the need by the system to make writes to the file system for files which are simply being read. Since writes can be somewhat expensive, this can result in measurable performance gains."

nodiratime does the same thing but for directories. I know the beginners guide says to use both mount options on filesystems, but from others I've talked to and places I've read it seems noatime implies nodiratime because noatime is a superset and nodiratime is a subset used specifically to disable it for directories but leave it on for files, and when you use noatime, it does it for everything, files/dirs

   echo "/dev/sdb1  /mount/data2  ext3  rw,noatime,nodiratime 0 0" >> /etc/fstab

Mount the drive. "mount -a" just mounts everything according to /etc/fstab.

1. mount -a
   

2. Restart the server to make sure it starts ok with the new drive mounted.

   shutdown -r now
   

An Introduction to ZooKeeper from Yahoo DN

An introduction to ZooKeeper from YDN

Tasks management on Linux using crontab

Crontab

The crontab (cron derives fromchronos, Greek for time; tab stands fortable) command, found in Unix and Unix-like operating systems, is used to schedule commands to be executed periodically. To see what crontabs are currently running on your system, you can open a terminal and run:

crontab -l

To edit the list of cronjobs you can run:

crontab -e

This will open a the default editor ( vi or pico) to manipulate the crontab settings. If you save and exit the editor, all your cronjobs are saved into crontab. Cronjobs are written in the following format:

* * * * * /bin/execute/this/script.sh

Scheduling explained

As you can see there are 5 stars. The stars represent different date parts in the following order:

1. minute (from 0 to 59)
2. hour (from 0 to 23)
3. day of month (from 1 to 31)
4. month (from 1 to 12)
5. day of week (from 0 to 6) (0=Sunday)

Execute every minute

If you leave the star, or asterisk, it means every. Maybe that's a bit unclear. Let's use the the previous example again:

* * * * * /bin/execute/this/script.sh

They are all still asterisks! So this means execute /bin/execute/this/script.sh:

1. every minute
2. of every hour
3. of every day of the month
4. of every month
5. and every day in the week.

In short: This script is being executed every minute. Without exception.

Execute every Friday 1AM

So if we want to schedule the script to run at 1AM every Friday, we would need the following cronjob:

0 1 * * 5 /bin/execute/this/script.sh

Get it? The script is now being executed when the system clock hits:

1. minute: 0
2. of hour: 1
3. of day of month: * (every day of month)
4. of month: * (every month)
5. and weekday: 5 (=Friday)

Execute 10 past after every hour on the 1st of every month

Here's another one, just for practicing

10 * 1 * * /bin/execute/this/script.sh

Fair enough, it takes some getting used to, but it offers great flexibility.

If you want to run something every 10 minutes

0,10,20,30,40,50 * * * * /bin/execute/this/script.sh

Mailing the crontab output of just one cronjob

And change the cronjob like this:*/10 * * * * /bin/execute/this/script.sh 2>&1 | mail -s "Cronjob ouput" yourname@

Running Hadoop with HBase On CentOS Linux (Multi-Node Cluster)

1.Prerequisites

1.1 Configure single nodes

Use my tutorial "Running Hadoop with HBase On CentOS Linux (Single-Node Cluster)"

1.2 SSH public access

Add mapping to /etc/hosts for each your nodes:

10.1.0.55 master
10.1.0.56 slave

To do that you need to add public key from master to slave ~/.ssh/authorized_keys and via verse, so eventually tou should be able to do the following:

[root@37 /usr/local/bin/hbase]ssh master
The authenticity of host 'master (10.1.0.55)' can't be established.
RSA key fingerprint is 09:e2:73:ac:6f:42:d1:da:13:20:76:10:36:29:c4:62.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'master,10.1.0.55' (RSA) to the list of known hosts.
Last login: Thu Dec 25 07:19:16 2008 from slave
[root@37 ~]exit
logout
Connection to master closed.
[root@37 /usr/local/bin/hbase]ssh slave
The authenticity of host 'slave (10.1.0.56)' can't be established.
RSA key fingerprint is 5c:4f:81:19:ad:f3:78:02:ce:64:f1:67:10:ca:c5:b8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'slave,10.1.0.56' (RSA) to the list of known hosts.
Last login: Thu Dec 25 07:16:47 2008 from 38.d.de.static.xlhost.com
[root@38 ~]exit
logout
Connection to slave closed.

2.Cluster Overview

The master node will also act as a slave because we only have two machines available in our cluster but still want to spread data storage and processing to multiple machines.

The master node will run the "master" daemons for each layer: namenode for the HDFS storage layer, and jobtracker for the MapReduce processing layer. Both machines will run the "slave" daemons: datanode for the HDFS layer, and tasktracker for MapReduce processing layer. Basically, the "master" daemons are responsible for coordination and management of the "slave" daemons while the latter will do the actual data storage and data processing work.

3. Hadoop core Configuration

3.1 conf/masters (master only)

The conf/masters file defines the master nodes of our multi-node cluster.

On master, update $HADOOP_HOME/conf/masters that it looks like this:

  
master

3.2 conf/slaves (master only)

This conf/slaves file lists the hosts, one per line, where the Hadoop slave daemons (datanodes and tasktrackers) will run. We want both the master box and the slave box to act as Hadoop slaves because we want both of them to store and process data.

On master, update $HADOOP_HOME/conf/slaves that it looks like this:

master
slave

If you have additional slave nodes, just add them to the conf/slaves file, one per line (do this on all machines in the cluster).

 master
slave
slave01
slave02
slave03

3.3 conf/hadoop-site.xml (all machines)

Following a sample configuration for all hosts, explanation of all parameters link

hadoop.tmp.dir
/usr/local/bin/hadoop/datastore/hadoop-${user.name}
A base for other temporary directories.



fs.default.name
hdfs://master:54310
The name of the default file system.  A URI whose
scheme and authority determine the FileSystem implementation.  The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class.  The uri's authority is used to
determine the host, port, etc. for a filesystem.



mapred.job.tracker
master:54311
The host and port that the MapReduce job tracker runs
at.  If "local", then jobs are run in-process as a single map
and reduce task.




dfs.replication
1
Default block replication.
The actual number of replications can be specified when the file is created.
The default is used if replication is not specified in create time.



mapred.reduce.tasks
8
The default number of reduce tasks per job.  Typically set
to a prime close to the number of available hosts.  Ignored when
mapred.job.tracker is "local".



mapred.tasktracker.reduce.tasks.maximum
8
The maximum number of reduce tasks that will be run
simultaneously by a task tracker.



mapred.child.java.opts
-Xmx500m
Java opts for the task tracker child processes.
The following symbol, if present, will be interpolated: @taskid@ is replaced
by current TaskID. Any other occurrences of '@' will go unchanged.
For example, to enable verbose gc logging to a file named for the taskid in
/tmp and to set the heap maximum to be a gigabyte, pass a 'value' of:
-Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc

The configuration variable mapred.child.ulimit can be used to control the
maximum virtual memory of the child processes.

Very important: now format each node data storage before proceeding.

3.4 Start/Stop Hadoop cluster

HDFS daemons: start/stop namenode command bin/start-dfs.sh or bin/stop-dfs.sh
MapReduce daemons: run the command $HADOOP_HOME/bin/start-mapred.sh on the machine you want the jobtracker to run on. This will bring up the MapReduce cluster with the jobtracker running on the machine you ran the previous command on, and tasktrackers on the machines listed in the conf/slaves file.

[root@37 /usr/local/bin/hadoop]bin/start-dfs.sh
starting namenode, logging to /usr/local/bin/hadoop/bin/../logs/hadoop-root-namenode-37.c3.33.static.xlhost.com.out
master: starting datanode, logging to /usr/local/bin/hadoop/bin/../logs/hadoop-root-datanode-37.c3.33.static.xlhost.com.out
slave: starting datanode, logging to /usr/local/bin/hadoop/bin/../logs/hadoop-root-datanode-38.c3.33.static.xlhost.com.out
master: starting secondarynamenode, logging to /usr/local/bin/hadoop/bin/../logs/hadoop-root-secondarynamenode-37.c3.33.static.xlhost.com.out
[root@37 /usr/local/bin/hadoop]bin/start-mapred.sh
starting jobtracker, logging to /usr/local/bin/hadoop/bin/../logs/hadoop-root-jobtracker-37.c3.33.static.xlhost.com.out
master: starting tasktracker, logging to /usr/local/bin/hadoop/bin/../logs/hadoop-root-tasktracker-37.c3.33.static.xlhost.com.out
slave: starting tasktracker, logging to /usr/local/bin/hadoop/bin/../logs/hadoop-root-tasktracker-38.c3.33.static.xlhost.com.out
[root@37 /usr/local/bin/hadoop]jps]
-bash: jps]: command not found
[root@37 /usr/local/bin/hadoop]jps
28638 DataNode
29052 Jps
28527 NameNode
28764 SecondaryNameNode
28847 JobTracker
28962 TaskTracker
[root@37 /usr/local/bin/hadoop]bin/hadoop dfs -copyFromLocal LICENSE.txt testWordCount
[root@37 /usr/local/bin/hadoop]bin/hadoop dfs -ls
Found 1 items
-rw-r--r--   1 root supergroup      11358 2008-12-25 08:20 /user/root/testWordCount
[root@37 /usr/local/bin/hadoop]bin/hadoop jar hadoop-0.18.2-examples.jar wordcount testWordCount testWordCount-out
08/12/25 08:21:46 INFO mapred.FileInputFormat: Total input paths to process : 1
08/12/25 08:21:46 INFO mapred.FileInputFormat: Total input paths to process : 1
08/12/25 08:21:47 INFO mapred.JobClient: Running job: job_200812250818_0001
08/12/25 08:21:48 INFO mapred.JobClient:  map 0% reduce 0%
08/12/25 08:21:53 INFO mapred.JobClient:  map 50% reduce 0%
08/12/25 08:21:55 INFO mapred.JobClient:  map 100% reduce 0%
08/12/25 08:22:01 INFO mapred.JobClient:  map 100% reduce 12%
08/12/25 08:22:02 INFO mapred.JobClient:  map 100% reduce 25%
08/12/25 08:22:05 INFO mapred.JobClient:  map 100% reduce 41%
08/12/25 08:22:09 INFO mapred.JobClient:  map 100% reduce 52%
08/12/25 08:22:14 INFO mapred.JobClient:  map 100% reduce 64%
08/12/25 08:22:19 INFO mapred.JobClient:  map 100% reduce 66%
08/12/25 08:22:24 INFO mapred.JobClient:  map 100% reduce 77%
08/12/25 08:22:29 INFO mapred.JobClient:  map 100% reduce 79%
08/12/25 08:25:30 INFO mapred.JobClient:  map 100% reduce 89%
08/12/25 08:26:12 INFO mapred.JobClient: Job complete: job_200812250818_0001
08/12/25 08:26:12 INFO mapred.JobClient: Counters: 17
08/12/25 08:26:12 INFO mapred.JobClient:   Job Counters
08/12/25 08:26:12 INFO mapred.JobClient:     Data-local map tasks=1
08/12/25 08:26:12 INFO mapred.JobClient:     Launched reduce tasks=9
08/12/25 08:26:12 INFO mapred.JobClient:     Launched map tasks=2
08/12/25 08:26:12 INFO mapred.JobClient:     Rack-local map tasks=1
08/12/25 08:26:12 INFO mapred.JobClient:   Map-Reduce Framework
08/12/25 08:26:12 INFO mapred.JobClient:     Map output records=1581
08/12/25 08:26:12 INFO mapred.JobClient:     Reduce input records=593
08/12/25 08:26:12 INFO mapred.JobClient:     Map output bytes=16546
08/12/25 08:26:12 INFO mapred.JobClient:     Map input records=202
08/12/25 08:26:12 INFO mapred.JobClient:     Combine output records=1292
08/12/25 08:26:12 INFO mapred.JobClient:     Map input bytes=11358
08/12/25 08:26:12 INFO mapred.JobClient:     Combine input records=2280
08/12/25 08:26:12 INFO mapred.JobClient:     Reduce input groups=593
08/12/25 08:26:12 INFO mapred.JobClient:     Reduce output records=593
08/12/25 08:26:12 INFO mapred.JobClient:   File Systems
08/12/25 08:26:12 INFO mapred.JobClient:     HDFS bytes written=6117
08/12/25 08:26:12 INFO mapred.JobClient:     Local bytes written=19010
08/12/25 08:26:12 INFO mapred.JobClient:     HDFS bytes read=13872
08/12/25 08:26:12 INFO mapred.JobClient:     Local bytes read=8620

4. HBase configuration

4.1 $HBASE_HOME/hbase-site.sh

hbase.rootdir
hdfs://master:54310/hbase
The directory shared by region servers.



hbase.master
master:60000
The host and port that the HBase master runs at.

4.2 $HBASE_HOME/conf/regionservers

master
slave

4.3 Start HBase on master

Use command $HBASE_HOME/bin/hbase-start.sh

[root@37 /usr/local/bin/hbase]bin/start-hbase.sh
starting master, logging to /usr/local/bin/hbase/bin/../logs/hbase-root-master-37.c3.33.static.xlhost.com.out
slave: starting regionserver, logging to /usr/local/bin/hbase/bin/../logs/hbase-root-regionserver-38.c3.33.static.xlhost.com.out
master: starting regionserver, logging to /usr/local/bin/hbase/bin/../logs/hbase-root-regionserver-37.c3.33.static.xlhost.com.out
[root@37 /usr/local/bin/hbase]jps
30362 SecondaryNameNode
30455 JobTracker
30570 TaskTracker
30231 DataNode
32003 Jps
31772 HMaster
30115 NameNode
31919 HRegionServer

On slave you suppose to see the following output:

[root@38 /usr/local/bin/hbase]jps
513 DataNode
6360 Jps
6214 HRegionServer
628 TaskTracker

Code highlighting in blogspot

Adding syntax highlight to blogger is very simple, first of all you need to download SyntaxHighlighter.

Now extract the contents of the package and upload the Scripts and Styles folder to any host or website which can be linked from your blog.

To make it work, you will need to edit your blog's template and add the following code after the -- end outer-wrapper -- tag:

To add some java code to your blog use:

The list of supported languages and their aliases can be found at the syntaxhighlighter wiki