Java frameworks and tools overview

Realtime Apache Hadoop at Facebook

2011-12-20T23:49:00.000-08:00

Facebook use Hadoop/Hbase for a following products Titan( Facebook messages), Puma( real-time publisher insights), ODS( Facebook internal metrics).

Currently in use 4 data centers, about 2000 nodes.

Data center replication is not supported as part of HBase, so they wrote some internal product to do that.

All data LZO compressed.

Each data center has 2PB of data replicated x3, total 6PB.

Each node have 12 discs each 1Tb.

Load on each node 55% read, 45% write.

To collect data from front end servers they're using Scribe, which is open sourced.

Slides here.

WHY HADOOP AND HBASE( from facebook paper)

The requirements for the storage system for our workloads can be summarized as follows:

1. Elasticity: We need to be able to add incremental capacity to our storage systems with minimal overhead and no downtime. In some cases we may want to add capacity rapidly and the system should automatically balance load and utilization across new hardware.

2. High write throughput: Most of the applications store (and optionally index) tremendous amounts of data and require high aggregate write throughput.

3. Efficient and low-latency strong consistency semantics within a data center: There are important applications like Messages that require strong consistency within a data center. This requirement often arises directly from user expectations. For example ‘‘unread’’ message counts displayed on the home page and the messages shown in the inbox page view should be consistent with respect to each other. While a globally distributed strongly consistent system is practically impossible, a system that could at least provide strong consistency within a data center would make it possible to provide a good user experience. We also knew that (unlike other Facebook applications), Messages was easy to federate so that a particular user could be served entirely out of a single data center making strong consistency within a single data center a critical requirement for the Messages project. Similarly, other projects, like realtime log aggregation, may be deployed entirely within one data center and are much easier to program if the system provides strong consistency guarantees.

4. Efficient random reads from disk: In spite of the widespread use of application level caches (whether embedded or via memcached), at Facebook scale, a lot of accesses miss the cache and hit the back-end storage system. MySQL is very efficient at performing random reads from disk and any new system would have to be comparable.

5. High Availability and Disaster Recovery: We need to provide a service with very high uptime to users that covers both planned and unplanned events (examples of the former being events like software upgrades and addition of hardware/capacity and the latter exemplified by failures of hardware components). We also need to be able to tolerate the loss of a data center with minimal data loss and be able to serve data out of another data center in a reasonable time frame.

6. Fault Isolation: Our long experience running large farms of MySQL databases has shown us that fault isolation is critical. Individual databases can and do go down, but only a small fraction of users are affected by any such event. Similarly, in our warehouse usage of Hadoop, individual disk failures affect only a small part of the data and the system quickly recovers from such faults.

7. Atomic read-modify-write primitives: Atomic increments and compare-and-swap APIs have been very useful in building lockless concurrent applications and are a must have from the underlying storage system.

8. Range Scans: Several applications require efficient retrieval of a set of rows in a particular range. For example all the last 100 messages for a given user or the hourly impression counts over the last 24 hours for a given advertiser.

It is also worth pointing out non-requirements:

1. Tolerance of network partitions within a single data center: Different system components are often inherently centralized. For example, MySQL servers may all be located within a few racks, and network partitions within a data center would cause major loss in serving capabilities therein. Hence every effort is made to eliminate the possibility of such events at the hardware level by having a highly redundant network design.

2. Zero Downtime in case of individual data center failure: In our experience such failures are very rare, though not impossible. In a less than ideal world where the choice of system design boils down to the choice of compromises that are acceptable, this is one compromise that we are willing to make given the low occurrence rate of such events. We might revise this non-requirement at a later time.

3. Active-active serving capability across different data centers: As mentioned before, we were comfortable making the assumption that user data could be federated across different data centers (based ideally on user locality). Latency (when user and data locality did not match up) could be masked by using an application cache close to the user.

Some less tangible factors were also at work. Systems with existing production experience for Facebook and in-house expertise were greatly preferred. When considering open-source projects, the strength of the community was an important factor. Given the level of engineering investment in building and maintaining systems like these –– it also made sense to choose a solution that was broadly applicable (rather than adopt point solutions based on differing architecture and codebases for each workload).

After considerable research and experimentation, we chose Hadoop and HBase as the foundational storage technology for these next generation applications. The decision was based on the state of HBase at the point of evaluation as well as our confidence in addressing the features that were lacking at that point via in- house engineering. HBase already provided a highly consistent, high write-throughput key-value store. The HDFS NameNode stood out as a central point of failure, but we were confident that our HDFS team could build a highly-available NameNode (AvatarNode) in a reasonable time-frame, and this would be useful for our warehouse operations as well. Good disk read-efficiency seemed to be within striking reach (pending adding Bloom filters to HBase’’s version of LSM Trees, making local DataNode reads efficient and caching NameNode metadata). Based on our experience operating the Hive/Hadoop warehouse, we knew HDFS was stellar in tolerating and isolating faults in the disk subsystem. The failure of entire large HBase/HDFS clusters was a scenario that ran against the goal of fault-isolation, but could be considerably mitigated by storing data in smaller HBase clusters. Wide area replication projects, both in-house and within the HBase community, seemed to provide a promising path to achieving disaster recovery.

HBase is massively scalable and delivers fast random writes as well as random and streaming reads. It also provides row-level atomicity guarantees, but no native cross-row transactional support. From a data model perspective, column-orientation gives extreme flexibility in storing data and wide rows allow the creation of billions of indexed values within a single table. HBase is ideal for workloads that are write-intensive, need to maintain a large amount of data, large indices, and maintain the flexibility to scale out quickly.

A Day Made of Glass... Made possible by Corning.

2011-02-24T23:44:00.000-08:00

An Introduction to RDF and the Jena RDF API

2010-09-04T06:58:00.001-07:00

http://jena.sourceforge.net/tutorial/RDF_API/

XAuth - an introduction from Chris Messina

2010-09-04T00:28:00.001-07:00

XAuth - an introduction from Chris Messina on Vimeo.

CentOS: find out what network ports are in use

2010-05-10T00:33:00.001-07:00

To get a quick idea of what ports you have open on your local box, you can use nmap.

[root@api1dev /usr/local/bin]nmap localhost
Starting Nmap 5.00 ( http://nmap.org ) at 2010-05-10 03:02 EDT
Interesting ports on vmlinux-testing1 (127.0.0.1):
Not shown: 991 closed ports
PORT     STATE SERVICE
21/tcp   open ftp
22/tcp   open ssh
80/tcp   open http
111/tcp open rpcbind
389/tcp open ldap
631/tcp open ipp
726/tcp open unknown
8009/tcp open ajp13
8080/tcp open http-proxy

For more detailed information, try netstat:

netstat -an

You’ll get a breakdown of every socket open on your machine – useful for figuring out who’s connected and from where.

Interesting articles about Ad-Exchange and RTB

2010-04-08T01:34:00.000-07:00

The Ad-Exchange Model (Part I)
RTB Part I Followup
RTB Part II: Supply supply supply!
RTB Part III: Cookies & User Data!

Alternative to HBase + Map-Reduce

2010-03-15T08:11:00.001-07:00

mongoDB

Combining the best features of document databases, key-value stores, and RDBMSes.

MongoDB (from "humongous") is a scalable, high-performance, open source, schema-free, document-oriented database. Written in C++, MongoDB features:

Document-oriented storage (the simplicity and power of JSON-like data schemas)
Dynamic queries
Full index support, including secondary indexes, inner-objects, embedded arrays, geospatial
Query profiling
Fast, in-place updates
Efficient storage of binary data large objects (e.g. photos and videos)
Replication and fail-over support
Auto-sharding for cloud-level scalability
MapReduce for complex aggregation
Commercial Support, Training, and Consulting

MongoDB bridges the gap between key-value stores (which are fast and highly scalable) and traditional RDBMS systems. Many companies are using MongoDB in production today.

Quick Links

Support

LAMP on EC2 : Setting up Amazon Web Services

2010-02-18T00:31:00.001-08:00

http://goo.gl/z1Zb

http://goo.gl/xhqB

Web performance in seven steps: Summary and Conclusions

2010-02-08T01:05:00.000-08:00

Web performance in seven steps: Summary and Conclusions

Previous time I blogged about the last step of the seven steps, step 7: Share the responsibility for the whole chain, a non-technical but rather a communication and behavior thing which I found crucial for success. We now have reached the end of this series and I'll sum up the topics we've dealt with and draw some conclusions.

In this growing on line world with demanding customers it has become essential that services provided on the web are always available and always fast enough. This is often challenging to developers and operators: performance problems manifest themselves in various ways, like in frustration, loss of revenue and disruption of development; and just adding hardware is a doubtful solution.

The question is: how can we as developers and operators assure that our web site is always available and always fast? My answer is: you need the right approach. I present that approach: measure, don’t guess; seven steps to performance success. These seven steps are as follows:

This approach provides a pro-active way of working which my customers appreciate as valuable. It can actually be leveraged to assure high performance all the time, for virtually any on- and off-line application.

This blog series has been an interesting journey for me. Some time ago we presented our EJAPP Top 10 of performance problems. Now we have added this approach of seven steps to help assure your applications performance.

It has worked for us and our customers. How does this all work for you in practice? We’d like to hear your feedback.

Location based DB schema in mySQL

2009-12-12T07:34:00.000-08:00

http://dev.mysql.com/tech-resources/articles/4.1/gis-with-mysql.html

http://dev.mysql.com/doc/refman/4.1/en/spatial-extensions.html

http://howto-use-mysql-spatial-ext.blogspot.com/

http://forge.mysql.com/wiki/GIS_Functions#Introduction

http://forums.mysql.com/read.php?23,40909,43079#msg-43079

OpenInviter

2009-11-15T23:12:00.000-08:00

Easy access to your visitors address book in all major email providers and social networks around the world.

Completely painless and easy way of integrating in your website. It takes virtually not more than 5 minutes to have your own OpenInviter^TM up and running on your site.

Constant updates so that you can sit back and relax and always have access to the latest ways to get your visitor's address book.

WGET-ready! Yes, you read right! OpenInviter^TM is the only contacts importer supporting both WGET and cURL as methods of handling requests (since version 1.2) so now you can use it on ANY server you want without the hassle of installing libcurl!

Real time access to the service statuses so you can know if there is an email provider that is not working right with OpenInviter^TM.

http://openinviter.com/faq.php

GO: Google Launches Its Own Programming Language

2009-11-12T03:12:00.001-08:00

GO: Google Launches Its Own Programming Language

Posted using ShareThis

Upgrading mySQL from 5.0 to 5.1.on CentOS 5

2009-10-26T23:37:00.000-07:00

My first attempt was to use yum:


[root@linux /] yum info mysql
Loading "fastestmirror" plugin
Loading mirror speeds from cached hostfile
* base: mirror.sanctuaryhost.com
* updates: mirror.fdcservers.net
* addons: mirror.steadfast.net
* extras: mirror.trouble-free.net
Installed Packages
Name   : mysql
Arch   : i386
Version: 5.0.45
Release: 7.el5
Size   : 7.3 M
Repo   : installed
Summary: MySQL client programs and shared libraries.
Description:
MySQL is a multi-user, multi-threaded SQL database server. MySQL is a
client/server implementation consisting of a server daemon (mysqld)
and many different client programs and libraries. The base package
contains the MySQL client programs, the client shared libraries, and
generic MySQL files.


Available Packages
Name   : mysql
Arch   : i386
Version: 5.0.77
Release: 3.el5
Size   : 4.8 M
Repo   : base
Summary: MySQL client programs and shared libraries
Description:
MySQL is a multi-user, multi-threaded SQL database server. MySQL is a
client/server implementation consisting of a server daemon (mysqld)
and many different client programs and libraries. The base package
contains the MySQL client programs, the client shared libraries, and
generic MySQL files.

The version 5.0.77 is good, but it not last updated release, which is 5.1.40 up today, which contains an important mysql feature named
Pationing: Enables distributing portions of individual tables across a file system, according to rules which can be set when the table is created. In effect, different portions of a table are stored as separate tables in different locations, but from the user point of view, the partitioned table is still a single table.

Basically it won’t do an upgrade as the vendor has changed from being MySQL to Sun Microsystems, and as a result I have to do a complete uninstall and re-install manually, first backup your data, stop mysql service and uninstall the current installation( 5.0.45 in my case):


[root@16 /]service mysqld stop
[root@16 /]rpm -qa | grep -i '^mysql-'
mysql-server-5.0.45-7.el5
mysql-5.0.45-7.el5
mysql-devel-5.0.45-7.el5
[root@16 /]rpm -e mysql-server-5.0.45-7.el5
[root@16 /]rpm -e mysql-5.0.45-7.el5
[root@16 /]rpm -e mysql-devel-5.0.45-7.el5

Now download all the current MySQL packages you need and install all with rpm -i( links from mySQL site):


[root@linux /usr/local/bin] mkdir mysql_5.1.40
[root@linux /usr/local/bin] cd mysql_5.1.40
wget http://dev.mysql.com/get/Downloads/MySQL-5.1/MySQL-server-community-5.1.40-0.rhel5.i386.rpm/from/http://mirror.mirimar.net/mysql/
wget http://dev.mysql.com/get/Downloads/MySQL-5.1/MySQL-client-community-5.1.40-0.rhel5.i386.rpm/from/http://mirror.mirimar.net/mysql/
wget http://dev.mysql.com/get/Downloads/MySQL-5.1/MySQL-shared-community-5.1.40-0.rhel5.i386.rpm/from/http://mirror.mirimar.net/mysql/
wget http://dev.mysql.com/get/Downloads/MySQL-5.1/MySQL-shared-compat-5.1.40-0.rhel5.i386.rpm/from/http://mirror.mirimar.net/mysql/
wget http://dev.mysql.com/get/Downloads/MySQL-5.1/MySQL-devel-community-5.1.40-0.rhel5.i386.rpm/from/http://mirror.mirimar.net/mysql/
wget http://dev.mysql.com/get/Downloads/MySQL-5.1/MySQL-embedded-community-5.1.40-0.rhel5.i386.rpm/from/http://mirror.mirimar.net/mysql/
wget http://dev.mysql.com/get/Downloads/MySQL-5.1/MySQL-test-community-5.1.40-0.rhel5.i386.rpm/from/http://mirror.mirimar.net/mysql/
[root@linux /mysql_5.1.40] rpm -i MySQL-shared-community-5.1.40-0.rhel5
[root@linux /mysql_5.1.40] rpm -i MySQL-embedded-community-5.1.40-0.rhel5
[root@linux /mysql_5.1.40] rpm -i MySQL-server-community-5.1.40-0.rhel5
[root@linux /mysql_5.1.40] rpm -i MySQL-client-community-5.1.40-0.rhel5
[root@linux /mysql_5.1.40] rpm -i MySQL-test-community-5.1.40-0.rhel5
[root@linux /mysql_5.1.40] rpm -i MySQL-devel-community-5.1.40-0.rhel5
[root@linux /mysql_5.1.40] mysql_upgrade
[root@linux /mysql_5.1.40] service mysql start
[root@linux /mysql_5.1.40] mysqladmin -V
 mysqladmin  Ver 8.42 Distrib 5.1.40, for pc-linux-gnu on i686

The Cloud Dilemma for Developers( From Tikal Community Event)

2009-10-19T06:01:00.000-07:00

Introduction to Cloud Computing by Yanai Franchi

Google App Engine Intro By Andrew Skiba

Amazon AWS Case Study by: Dudi Landau , “Thomson Reuters", "ClearForest CTO”

Automated Deployment with Maven - going the whole nine yards

2009-10-03T04:13:00.001-07:00

Check out this SlideShare Presentation:

Automated Deployment with Maven - going the whole nine yards

View more documents from wakaleo.

A Watercolor Film about the Evolution of Life

2009-09-09T02:07:00.000-07:00

A Record Of Life from Owen Gatley and Luke Jinks on Vimeo.

Real World Web: Performance & Scalability

2009-08-27T02:51:00.000-07:00

Real World Web: Performance & Scalability

Best GWT widget library(by stackoverflow)

2009-08-23T02:07:00.000-07:00

Original post.

Benchmarks in java

2009-08-19T06:21:00.000-07:00

First one deals with all the nasty and important details one has to consider to start benchmarking (compiler optimizations, JVM specifics, etc) and gives an example of how they can bite you in your behind. Second one is a real-life example of an attempt of performance analysis that at first glance appears reasonable. After a close look it turns out to have tons of problems and flaws that basically obscure the intended measurements.

Approaching Scalability, and Building a Biz on Hadoop, HBase, and Open Source Distributed Computing

2009-08-06T07:10:00.000-07:00

Building a Business on Hadoop, HBase, and Open Source Distributed Computing

View more presentations from Bradford Stephens.

The Best of Business Intelligence: Innovation at the Fringe

2009-08-05T13:23:00.000-07:00

http://www.juiceanalytics.com/writing/best-business-intelligence/

Good Data Architecture Overview at Cloud Expo

2009-08-05T12:35:00.001-07:00

Good Data: Collaborative Analytics On Demand

View more documents from zsvoboda.

Web 20 Business Models

2009-08-05T12:28:00.001-07:00

Web 20 Business Models

View more documents from reynolds.

OpenCV + Java + Linux

2009-07-26T07:38:00.001-07:00

OpenCV + Java + Linux

Shared via AddThis

WEB4J. Because simple is beautiful.

2009-07-25T11:33:00.000-07:00

The important things about WEB4J are :

it's an open source, full-stack Java web app framework
it let's you build a browser front-end to a relational database
it's small, and has a philosophy of deep simplicity and minimalism
it lets you put SQL in plain .sql text files
it lets you implement forms with plain old HTML
it can significantly boost your productivity compared to other tools
it allows package-by-feature, where closely related items are placed in the same directory
it protects you from common hacks
it has no reliance on javascript
it has no custom annotations
it has no custom .xml files (only web.xml is used)
it has no object-relational mapping
it can be used with any relational database that has a JDBC driver
it lets your Model Objects be immutable
it lets your Model Objects be responsible for their own validation (as they should be)
it lets your Model Objects avoid the Java Beans anti-pattern
it has nice building block classes
it uses a minimal toolset
it doesn't take you long to learn (and you don't need to buy a big fat book)
it uses convention over configuration in several important ways
it makes your application classes simple, and easy to maintain
it requires minimal configuration
it doesn't impose thread-safety constraints on your classes
it lets multilingual apps have almost the same style as a single language app
it lets your apps assist in their own translation
it lets you place translations in the database
it lets you avoid ResourceBundle and its defects
it allows apps to share a fake system clock with the framework, so they remain in sync
it starts up quickly

Introduction to WEB4J by JavaWorld

faint - The Face Annotation Interface

2009-07-25T11:10:00.000-07:00

This project is a flexible Java framework for face detection and face recognition technologies, that is based on different plugin and filter types. A suitable graphical interface can be used to set up pipelines for detection and recognition by combining these plugins and filters. Moreover an integrated photo browser allows users to apply the face detection and recognition process on personal images.

Usage example

nVidia CUDA

2009-07-25T11:04:00.000-07:00

NVIDIA® CUDA™ technology leverages the massively parallel processing power of NVIDIA GPUs. The CUDA architecture is a revolutionary parallel computing architecture that delivers the performance of NVIDIA’s world-renowned graphics processor technology to general purpose GPU Computing. Applications that run on the CUDA architecture can take advantage of an installed base of over one hundred million CUDA-enabled GPUs in desktop and notebook computers, professional workstations, and supercomputer clusters.

With the CUDA architecture and tools, developers are achieving dramatic speedups in fields such as medical imaging and natural resource exploration, and creating breakthrough applications in areas such as image recognition and real-time HD video playback and encoding. CUDA enables this unprecedented performance via standard APIs such as the soon to be released OpenCL™ and DirectX® Compute, and high level programming languages such as C/C++, Fortran, Java, Python, and the Microsoft .NET Framework.

The CUDA Developer SDK provides examples with source code to help you get started with CUDA. Examples include:

Parallel bitonic sort
Matrix multiplication
Matrix transpose
Performance profiling using timers
Parallel prefix sum (scan) of large arrays
Image convolution
1D DWT using Haar wavelet
OpenGL and Direct3D graphics interoperation examples
CUDA BLAS and FFT library usage examples
CPU-GPU C- and C++-code integration
Binomial Option Pricing
Black-Scholes Option Pricing
Monte-Carlo Option Pricing
Parallel Mersenne Twister (random number generation)
Parallel Histogram
Image Denoising
Sobel Edge Detection Filter

HAT -- The Java Heap Analysis Tool

2009-07-19T00:16:00.000-07:00

Explanations and examples could found here

Tweaking hard disk on Linux

2009-07-09T05:35:00.000-07:00

 
hdparm -Tt /dev/hda

/dev/hda:
Timing buffer-cache reads:   128 MB in  1.34 seconds =95.52 MB/sec
Timing buffered disk reads:  64 MB in 17.86 seconds = 3.58 MB/sec

hdparm /dev/hda

/dev/hda:
multcount    =  0 (off)
I/O support  =  0 (default 16-bit)
unmaskirq    =  0 (off)
using_dma    =  0 (off)
keepsettings =  0 (off)
nowerr       =  0 (off)
readonly     =  0 (off)
readahead    =  8 (on)
geometry     = 1870/255/63, sectors = 30043440, start = 0

multcount: Short for multiple sector count. This controls how many sectors are fetched from the disk in a single I/O interrupt. Almost all modern IDE drives support this. The man page claims: when this feature is enabled, it typically reduces operating system overhead for disk I/O by 30-50%. On many systems, it also provides increased data throughput of anywhere from 5% to 50%.
I/O support: This is a big one. This flag controls how data is passed from the PCI bus to the controller. Almost all modern controller chipsets support mode 3, or 32-bit mode w/sync. Some even support 32-bit async. Turning this on will almost certainly double your throughput (see below.)
unmaskirq: Turning this on will allow Linux to unmask other interrupts while processing a disk interrupt. What does that mean? It lets Linux attend to other interrupt-related tasks (i.e., network traffic) while waiting for your disk to return with the data it asked for. It should improve overall system response time, but be warned: Not all hardware configurations will be able to handle it. See the manpage.
using_dma: DMA can be a tricky business. If you can get your controller and drive using a DMA mode, do it. But I have seen more than one machine hang while playing with this option.

 
hdparm -X66 -d1 -u1 -m16 -c3 /dev/hda:
setting 32-bit I/O support flag to 3
setting multcount to 16
setting unmaskirq to 1 (on)
setting using_dma to 1 (on)
setting xfermode to 66 (UltraDMA mode2)
multcount    = 16 (on)
I/O support  =  3 (32-bit w/sync)
unmaskirq    =  1 (on)
using_dma    =  1 (on)

hdparm -tT /dev/hda

/dev/hda:
Timing buffer-cache reads:   128 MB in  1.43 seconds = 89.51 MB/sec
Timing buffered disk reads:  64 MB in  3.18 seconds = 20.13 MB/sec

Semantic Applications For Web

2009-07-06T05:00:00.000-07:00

Alphageeks meetup - Semantic Applications For Web

View more presentations from yuvalgo.

Scaling Hibernate

2009-07-06T04:09:00.000-07:00

http://www.infoq.com/presentations/Scaling-Hibernate-Emmanuel-Bernard-Max-Ross

An Eclipse 3.5 Galileo is here !

2009-07-04T11:21:00.000-07:00

An Eclipse Galileo 3.5

What's new in the latest version of the open source multipurpose IDE and application platform

Galileo is the simultaneous release of 33 major Eclipse projects. The Eclipse Foundation states that the Galileo release train consists of 33 projects. However, some are subprojects that are rolled up into projects, and not all projects are highlighted in the Eclipse Foundation's marketing push. Regardless, Galileo represents the largest single release of new technology to date.
The important thing to remember about Galileo in particular and Eclipse release trains in general is that even though it's a simultaneous release, it doesn't mean these projects are unified. Each project is a separate open source project, operating with its own project leadership, its own committers, and its own development plan. The release train concept is designed to provide a transparent and predictable development cycle.
Get Galileo
There are two main ways to get Galileo. The first — and recommended — way is to just grab a package relevant to you. The other way to get Galileo is to use an update site.
Packages
Go to the Eclipse Galileo Packages site. The packages site contains nine pre-bundled versions of Galileo specific for your needs.
Galileo update site
To get Galileo using an update site, download the Eclipse V3.5 SDK. Once this is done, you can launch Eclipse and access the software-update mechanism via Help > Software Updates (see Figure 2). Enter the proper Galileo update site information, if it isn't already available as the Galileo Discovery Site. Once you are connected to the Galileo update site, you should see the list of available features that are part of the Galileo release train. It's as simple as that. Once you're connected, you can simply choose what features to install into your Eclipse.

Figure 2. Software updates

The projects
The Eclipse ecosystem is a large and sometimes intimidating place. About 100 projects are being overseen by the Eclipse Foundation, and the Galileo release only represents a snapshot of that. The Galileo release train showcases Eclipse technology and helps adopters integrate Eclipse technology into their products. For more information about the Galileo projects, see the links below.

Project Synopsis Web site
Accessibility Tools Framework (ACTF) Build applications and content for people with disabilities http://www.eclipse.org/actf/
Business Intelligence and Reporting Tools (BIRT) Generate reports http://www.eclipse.org/birt
C/C++ Development Tooling (CDT) Code C/C++ http://www.eclipse.org/cdt
Data Tools Platform (DTP) Extensible frameworks and tools http://www.eclipse.org/datatools/
Eclipse Modeling Framework (EMF) Modeling framework and code generation facility http://www.eclipse.org/modeling/emf/
Eclipse Packaging Project Create, download, and install packages http://www.eclipse.org/epp/
Eclipse Platform Core frameworks and services http://www.eclipse.org/platform/
Equinox Implementation of the OSGi R4 core framework spec http://www.eclipse.org/equinox/
Graphical Editor Framework (GEF) Develop graphical applications http://www.eclipse.org/gef/
Graphical Modeling Framework (GMF) Develop graphical editors http://www.eclipse.org/gmf/
Java™ Workflow Tooling (JWT) Toolset for workflows and processes from design to monitoring http://www.eclipse.org/jwt/
Java Development Tools (JDT) Develop Java applications http://www.eclipse.org/jdt/
Java Emitter Templates (M2T JET) Generate textual artifacts from models http://www.eclipse.org/modeling/m2t/
Memory Analyzer Find memory leaks and reduce memory consumption http://www.eclipse.org/mat/
Mobile Tools for Java (MTJ) Extend Eclipse frameworks to support mobile device Java application development http://www.eclipse.org/dsdp/mtj/
Mylyn Monitors your work to make the GUI relevant to what you're doing http://www.eclipse.org/mylyn/
PHP Development Tools (PDT) Code PHP http://www.eclipse.org/pdt/
Rich Ajax Platform (RAP) Code Ajax http://www.eclipse.org/rap/
SCA Tools Tools for the Service Component Architecture standard http://www.eclipse.org/stp/sca/
SOA Tools Code Service-Oriented Architecture apps http://www.eclipse.org/stp/
Swordfish Extensible SOA framework http://www.eclipse.org/swordfish/
Target Management Configure and manage remote systems http://www.eclipse.org/dsdp/tm/
Test and Performance Tools Platform Project (TPTP) Tooling for profiling and testing applications http://www.eclipse.org/tptp/
Textual Modeling Framework (Xtext) Code external textual DSLs http://www.eclipse.org/modeling/tmf/
Tools for mobile Linux (TmL) Code mobile applications http://www.eclipse.org/dsdp/tml/
Web Tools Platform (WTP) Code Web and Java EE applications http://www.eclipse.org/webtools/

Google Developer Days Brazil 2009 - Keynote

2009-07-04T00:22:00.000-07:00

Google Developer Days Brazil 2009 - Keynote

View more presentations from chanezon.

Neo4j a graph database

2009-06-30T23:41:00.000-07:00

Neo4j is a graph database. It is an embedded, disk-based, fully transactional Java persistence engine that stores data structured in graphs rather than in tables. A graph (mathematical lingo for a network) is a flexible data structure that allows a more agile and rapid style of development.

According Emil Eifrem the neo4j database outperforms relational backends with >1000x for many increasingly important use cases.

How to generate XSD schema from XML

2009-06-29T22:47:00.000-07:00

xmlbeans: http://xmlbeans.apache.org/docs/2.0.0/guide/tools.html#inst2xsd
online tool: http://www.flame-ware.com/xml2xsd/
oxygenxml: http://www.oxygenxml.com/
(eclipse plugin: http://www.oxygenxml.com/eclipse_plugin.html)
stylus studio: http://www.stylusstudio.com/autogen_xsd.html

How to Exploit Multiple Cores

2009-06-28T06:38:00.000-07:00

How to Exploit Multiple Cores for Better Performance and Scalability( by Todd Hoff)

Learn How to Exploit Multiple Cores for Better Performance and Scalability (350)

InfoQueue has this excellent talk by Brian Goetz on the new features being added to Java SE 7 that will allow programmers to fully exploit our massively multi-processor future. While the talk is about Java it's really more general than that and there's a lot to learn here for everyone.

Brian starts with a short, coherent, and compelling explanation of why programmers can't expect to be saved by ever faster CPUs and why we must learn to exploit the strengths of multiple core computers to make our software go faster.

Some techniques for exploiting multiple cores are given in an equally short, coherent, and compelling explanation of why divide and conquer as the secret to multi-core bliss, fork-join, how the Java approach differs from map-reduce, and lots of other juicy topics.

The multi-core "problem" is only going to get worse. Tilera founder Anant Agarwal estimates by 2017 embedded processors could have 4,096 cores, server CPUs might have 512 cores and desktop chips could use 128 cores. Some disagree saying this is too optimistic, but Agarwal maintains the number of cores will double every 18 months.

An abstract of the talk follows though I would highly recommend watching the whole thing. Brian does a great job.

Why is Parallelism More Important Now?

Coarse grain concurrency was all the rage for Java 5. The hardware reality has changed. The number of cores is increasing so applications must now search for fine grain parallelism (fork-join)

As hardware becomes more parallel, more and more cores, software has to look for techniques to find more and more parallelism to keep the hardware busy.

Clock rates have been increasing exponentially over the last 30 years or so. Allowed programmers to be lazy because a faster processor would be released that saved your butt. There wasn't a need to tune programs.

That wait for faster processor game is up. Around 2003 clock rates stopped increasing. Hit the power wall. Faster processors require more power. Thinner chip conductor lines were required and the thinner lines can't dissipate the increased power without causing overheating which effects the resistance characteristics of the conductors. So you can't keep increasing clock rate.

Fastest Intel CPU 4 or 5 years ago was 3.2 Ghz. Today it's about the same or even slower.

Easier to build 2.6 Ghz or 2.8 Ghz chips. Moore's law wasn't repealed so we can cram more transistors on each wafer. So more processing power could be put on a chip which leads to putting more and more processing cores on a chip. This is multicore.

Multicore systems are the trend. The number of cores will grow at exponential rate for the next 10 years. 4 cores at the low end. The high end 256 (Sun) and 800 (Azul) core systems.

More cores per chip instead of faster chips. Moore's law has been redirected to multicore.

The problem is it's harder to make a program go faster on a multicore system. A faster chip will run your program faster. If you have a 100 cores you program won't go faster unless you explicitly design it to take advantage of those chips.

No free lunch anymore. Must now be able to partition your program so it can run faster by running on multiple cores. And you must be able keep doing that as the number of cores keeps improving.

We need a way to specify programs so they can be made parallel as topologies change by adding more cores.

As hardware evolves platforms must evolve to take advantage of the new hardware. Started off with course grain tasks which was sufficient given the number of cores. This approach won't work as the number cores increase.

Must find finer-grained parallelism. Example sorting and searching data. Opportunities around data. The data can for sorting can be chunked and sorted and the brought together with a merge sort. Searching can be done in parallel by searching subregions of the data and merging the results.

Parallel solutions use more CPU in aggregate because of the coordination needed and that data needs to be handled more than once (merge). But the result is faster because it's done in parallel. This adds business value. Faster is better for humans.

What has Java 7 Added to Support Parallelism?

Example problem is to find the max number from a list.

The course grained threading approach is to use a thread pool, divide up the numbers, and let the task pool compute the sub problems. A shared task pool is slow as the number increases which forces the work to be more course grained. No way to load balance. Code is ugly. Doesn't match the problem well. The runtime is dominated by how long it takes the longest subtask to run. Had to decide up front how many pieces to divide the problem into.

Solution using divide and conquer. Divide set into pieces recursively until the problem is so small the sequential solution is more efficient. Sort the pieces. Merge the results. 0(n log n), but problem is parallelizable. Scales well and can keep many CPUs busy.

Divide and conquer uses fork-join to fork off subtasks and wait for them to complete and then join the results. A typical thread pool solution is not efficient. Creates too many threads and creating threads are expensive and use a lot of memory.

This approach portable because it's abstract. It doesn't know how many processors are available It's independent of the topology.

The fork-join pool is optimized for fine grained operations whereas the thread pool is optimized for course grained operations. Best used for problems without IO. Just computations using CPU that tend to fork off sub problems. Allows data to be shared read-only and used across different computations without copying.

This approach scales nearly linearly with the number of hardware threads.

The goal for fork-join: Avoid context switches; Have as many threads as hardware threads and keep them all busy; Minimize queue lock contention for data structures. Avoid common task queue.

Implementation uses Work-Stealing. Each thread has a work queue that is a double ended queue. Each thread pulls work from the head of queue and processes it. When there's nothing do it steals work from the tail of another queue. No contention for the head because only one thread access it. Rare contention on tail because stealing is infrequent as the stolen work is large which takes them time to process. Process starts with one task. It breaks up the work. Other tasks steal work and start the same process. Load balances without central coordination, few context switches, little coordination.

The same approach also works for graph traversal, matrix operations, linear algebra, modeling, generate moves and evaluate the result. Latent parallelism can be found in a lot of places once you start looking.

Support higher level operations like ParallelArray. Can specify filtering, transformation, and aggregation options. Not a generalized in-memory database, but has a very transparent cost model. It's clear how many parallel operations are happening. Can look at the code and quickly know what's a parallel operation so you will know the cost.

Looks like map reduce except this is scaling across a multicore system, one single JVM, whereas map reduce is across a cluster. The strategy is the same: divide and conquer.

Idea is to make specifying parallel operations so easy you wouldn't even think of the serial approach.

Google Shares Its Need For Speed

2009-06-23T23:39:00.001-07:00

Google Shares Its Need For Speed

Posted using ShareThis

Restlet resources

2009-06-23T13:16:00.000-07:00

User guide v1.1
FAQ
GAE integration
Tutorial for v2.0

Creating Adobe Flex 3 widgets in 5 minutes

2009-06-23T11:52:00.000-07:00

See a Flex app created with Flex Builder 3

Flex in a week:

http://www.adobe.com/devnet/flex/videotraining/

Intro To Media APIs

2009-06-23T00:23:00.001-07:00

Check out this SlideShare Presentation:

Intro To Media APIs

View more OpenOffice presentations from Sachin Rekhi.

How to control services in linux with chkconfig

2009-06-16T23:18:00.000-07:00

Linux / Unix Command: chkconfig

chkconfig - updates and queries runlevel information for system services:

chkconfig --list [name]
chkconfig --add name
chkconfig --del name
chkconfig [--level levels] name <on|off|reset>
chkconfig [--level levels] name

DESCRIPTION

chkconfig provides a simple command-line tool for maintaining the /etc/rc[0-6].d directory hierarchy by relieving system administrators of the task of directly manipulating the numerous symbolic links in those directories.

This implementation of chkconfig was inspired by the chkconfig command present in the IRIX operating system. Rather than maintaining configuration information outside of the /etc/rc[0-6].d hierarchy, however, this version directly manages the symlinks in /etc/rc[0-6].d. This leaves all of the configuration information regarding what services init
starts in a single location. chkconfig has five distinct functions: adding new services for
management, removing services from management, listing the current startup information for services, changing the startup information for services, and checking the startup state of a particular service. When chkconfig is run without any options, it displays usage
information. If only a service name is given, it checks to see if the service is configured to be started in the current runlevel. If it is, chkconfig returns true; otherwise it returns false. The
--level option may be used to have chkconfig query an alternative runlevel rather than the current one. If one of on, off, or reset is specified after the service name, chkconfig changes the startup information for the specified service. The on and off flags cause the service
to be started or stopped, respectively, in the runlevels being changed. The reset flag resets the startup information for the service to whatever is specified in the init script in question.

By default, the on and off options affect only runlevels 2, 3, 4, and 5, while reset affects all of the runlevels. The --level option may be used to specify which runlevels are affected.

Note that for every service, each runlevel has either a start script or a stop script. When switching runlevels, init will not re-start an already-started service, and will not re-stop a service that is not running.

OPTIONS

--level levels

Specifies the run levels an operation should pertain to. It is given as a string of numbers from 0 to 7. For example, --level 35 specifies runlevels 3 and 5.

--add name

This option adds a new service for management by chkconfig.When a new service is added, chkconfig ensures that the servicehas either a start or a kill entry in every runlevel. If any runlevel is missing such an entry, chkconfig creates the appropriate entryas specified by the default values in the init script. Note thatdefault entries in LSB-delimited 'INIT INFO' sections take precedence over the default runlevels in the initscript.

--del name

The service is removed from chkconfig management, and any symboliclinks in /etc/rc[0-6].d which pertain to it are removed.

--list name

This option lists all of the services which chkconfig knows about, and whether they are stopped or started in each runlevel. If name is specified, information in only display about service name.

RUNLEVEL FILES

Each service which should be manageable by chkconfig needs two or more commented lines added to its init.d script. The first linetells chkconfig what runlevels the service should be started in by default, as well as the start and stop priority levels. If the serviceshould not, by default, be started in any runlevels, a - should be used in place of the runlevels list. The second line contains a description for the service, and may be extended acrossmultiple lines with backslash continuation.

For example, random.init has these three lines:

# chkconfig: 2345 20 80
# description: Saves and restores system entropy pool for \
#              higher quality random number generation.

This says that the random script should be started in levels 2, 3, 4, and 5, that its start priority should be 20, and that its stop priority should be 80. You should be able to figure out what the description says; the \ causes the line to be continued. The
extra space in front of the line is ignored.

For instance take a look on service configuration to run tomcat:


#startup script for Jakarta Tomcat
#
# chkconfig: 345 84 16
# description: Jakarta Tomcat Java Servlet/JSP Container

TOMCAT_HOME=/usr/local/bin/apache-tomcat-6.0.16
TOMCAT_START=/usr/local/bin/apache-tomcat-6.0.16/bin/startup.sh
TOMCAT_STOP=/usr/local/bin/apache-tomcat-6.0.16/bin/shutdown.sh
TOMCAT_RUN=/usr/local/bin/apache-tomcat-6.0.16/bin/catalina.sh
#Necessary environment variables
export CATALINA_HOME=/usr/local/bin/apache-tomcat-6.0.16

# Source function library.
. /etc/rc.d/init.d/functions

# Source networking configuration.
. /etc/sysconfig/network

# Check that networking is up.
[ ${NETWORKING} = "no" ] && exit 0

#Check for tomcat script
if [ ! -f $TOMCAT_HOME/bin/catalina.sh ]
then
 echo "Tomcat not available..."
 exit
fi

start() {
 echo -n "Starting Tomcat: "
 su - root -c $TOMCAT_START
 echo
 touch /var/lock/subsys/tomcatd
# We may need to sleep here so it will be up for apache
#    sleep 5
#Instead should check to see if apache is up by looking for http.pid
}
run() {
 echo -n "Starting Tomcat: "
 su - root -c $TOMCAT_START
 echo
 touch /var/lock/subsys/tomcatd
# We may need to sleep here so it will be up for apache
#    sleep 5
#Instead should check to see if apache is up by looking for http.pid
}

stop() {
 echo -n $"Shutting down Tomcat: "
 su - root -c $TOMCAT_STOP
 rm -f /var/lock/subsys/tomcatd.pid
 echo
}

status() {
 ps ax --width=1000 | grep "[o]rg.apache.catalina.startup.Bootstrap
start" | awk '{printf $1 " "}' | wc | awk '{print $2}' > /tmp/tomcat_process_count.txt
read line < /tmp/tomcat_process_count.txt if [ $line -gt 0 ]; then     echo -n "tomcatd ( pid "     ps ax --width=1000 | grep "[o]rg.apache.catalina.startup.Bootstrap start" | awk '{printf $1 " "}'     echo -n ") is running..." else     echo -n "Tomcat is stopped" fi }  case "$1" in     start)         start         ;;       run)         run         ;;       stop)         stop         ;;      restart)         stop         sleep 3         start         ;;      status)         status         ;;      *)         echo "Usage: tomcatd {start|stop|restart|status}"      exit 1 esac

Java service wrapper fro Linux

2009-06-07T06:53:00.000-07:00

http://wrapper.tanukisoftware.org/doc/english/introduction.html

How to send e-mail using GMail on CentOS from shell script

2009-06-07T04:11:00.000-07:00

sSMTP is a very simple and straightforward alternative to big MTAs like sendmail or Exim. Unfortunately CentOS repositories don’t come with it, so you have to fetch it from Fedora’s EPEL repo.


rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-3.noarch.rpm
yum install ssmtp

Configuration is rather simple. Just change the following values in your /etc/ssmtp/ssmtp.conf:


root=noreply@youdomain.com
AuthUser=noreply@yourdomain.com
AuthPass=password
FromLineOverride=YES
mailhub=smtp.gmail.com:587
UseSTARTTLS=YES

To test just run a following command("-v" should be used for debugging):


echo "test" | ssmtp -v -s "Test" genadyg@exelate.com

To use sSmtp in unix script:


ssmtp myemailaddress@gmail.com < msg.txt

msg.txt is a simple text using the proper formatting for sSMTP:


To: toaddress@gmail.com
From: fromaddress@gmail.com
Subject: alert
Message text

Google wave: implementation notes

2009-06-03T23:54:00.000-07:00

It's an HTML 5 app, built on Google Web Toolkit. It includes a rich text editor and other functions like desktop drag-and-drop (which, for example, lets you drag a set of photos right into a wave).

Went Walkabout. Brought back Google Wave

Google Wave Drips With Ambition. A New Communication Platform For A New Web

Google wave is here !

2009-06-03T07:40:00.000-07:00

Google Wave is a new communication and collaboration platform based on hosted XML documents (called waves) supporting concurrent modifications and low-latency updates. This platform enables people to communicate and work together in new, convenient and effective ways. We will offer these benefits to users of Google Wave and we also want to share them with everyone else by making waves an open platform that everybody can share. We welcome others to run wave servers and become wave providers, for themselves or as services for their users, and to "federate" waves, that is, to share waves with each other and with Google Wave. In this way users from different wave providers can communicate and collaborate using shared waves. We are introducing the Google Wave Federation Protocol for federating waves between wave providers on the Internet.

Here are the initial white papers that are available to complement the Google Wave Federation Protocol:

Google Wave Federation Architecture
Google Wave Data Model and Client-Server Protocol
Google Wave Operational Transform
General Verifiable Federation

The Google Wave APIs are documented here.

Amazon s3 java client

2009-06-02T01:12:00.000-07:00

J2ME and J2SE Toolkit for Amazon S3

http://developer.amazonwebservices.com/connect/entry.jspa?externalID=816&categoryID=47

Introduction to Amazon S3 with Java and REST

http://www.onjava.com/pub/a/onjava/2007/11/07/introduction-to-amazon-s3-with-java-and-rest.html

SmartGWT

2009-06-02T00:57:00.000-07:00

SmartGWT is a GWT based framework that allows you to not only utilize
its comprehensive widget library for your application UI, but also tie
these widgets in with your server-side for data management. SmartGWT is
based on the powerful and mature SmartClient library.
http://code.google.com/p/smartgwt/

Common REST Mistakes

2009-06-02T00:04:00.000-07:00

Common REST Mistakes( from Paul Prescod)

When designing your first REST system there are a variety of mistakes people often make. I want to summarize them so that you can avoid them. If any are unclear, ask for more information on rest-discuss.

Using HTTP is not enough. You can use HTTP in a Web service without SOAP or XML-RPC and still do the logical equivalent of SOAP or XML-RPC. If you're going to use HTTP wrong you would actually be better off doing it in a standard way! Most of these other points describe ways in which people abuse HTTP.
Do not overuse POST. POST is in some senses the "most flexible" of HTTP's methods. It has a slightly looser definition than the other methods and it supports sending information in and getting information out at the same time. Therefore there is a tendency to want to use POST for everything. In your first REST Web Service, I would say that you should only use POST when you are creating a new URI. Pretend POST means "create new URI as child of the current URI." As you get more sophisticated, you may decide to use POST for other kinds of mutations on a resource. One rule of thumb is to ask yourself whether you are using POST to do something that is really a GET, DELETE or PUT, or could be decomposed into a combination of
methods.
Do not depend on URI's internal structure. Some people think about REST design in terms of setting up a bunch of URIs. "I'll put purchase orders in /purchases and I'll give them all numbers like /purchases/12132 and customer records will be in
/customers..." That can be a helpful way to think while you are whiteboarding
and chatting, but should not be your final public interface to the service. According to Web Architectural principles, most URIs are opaque to client software most of the time. In other words, your public API should not depend on the structure of your URIs. Instead there would typically be a single XML file that points to the components of your service. Those components would have hyperlinks that point to other components and so forth. Then you can introduce people to your service with a single URI and you can distribute the actual components across computers and domains however you want. My rule of thumb is that clients only construct URIs when they are building queries (and thus using query strings). Those queries return references to objects with opaque URIs.
Do not put actions in URIs.
This follows naturally from the previous point. But a particularly pernicious abuse of URIs is to have query strings like "someuri?action=delete". First, you are using GET to do something unsafe. Second, there is no formal relationship between this "action URI" and the "object" URI. After all your "action=" convention is something specific to your application. REST is about driving as many "application conventions" out of the protocol as possible.
Services are seldom resources. In a REST design, a "stock quote service" is not very
interesting. In a REST design you would instead have a "stock" resources and a
service would just be an index of stock resources.
Sessions are irrelevant.
There should be no need for a client to "login" or "start a connection." HTTP authentication is done automatically on every message. Client applications are consumers of resources, not services. Therefore there is nothing to log in to! Let's say that you are booking a flight on a REST web service. You don't create a new "session" connection to the service. Rather you ask the "itinerary creator object" to create you a new itinerary. You can start filling in the blanks but then get some totally different component elsewhere on the web to fill in some other blanks. There is no session so there is no problem of migrating session state between clients. There is also no issue of "session affinity" in the server (though there are still load balancing issues to continue).
Do not invent proprietary object identifiers. Use URIs. URIs are important because you can always associate information with them in two ways. The simplest way is to put data on a web server so that the URI can be dereferenced in order to get the data. Note that this technique only works with URIs that can be dereferenced so these URIs (http URIs!) are strongly preferred to URN or UUID-based URIs. Another way is to use RDF and other techniques that allow you to project metadata onto a URI
that may not be under your control. If you use URI syntax with UUIDs or something like that then you get half of the benefit of URIs. You get a standardized syntax but have no standardized dereferencing capability. If you use an HTTP URI then you get the other half of the benefit because you then also have a standardized
derferencing mechanism.
Do not worry about protocol independence. There exists only one protocol whichsupports the proper resource manipulation semantics. If another one arises in the future, it will be easy to keep your same design and merely support the alternate
protocol's interface. On the other hand, what people usually mean by "protocol
independence" is to abandon resource modelling and therefore abandon both REST
and the Web.

Overall, the thing to keep in mind is that REST is about exposing resources through URIs, not services through messaging interfaces.

PHP/Java Rest frameworks finalists

2009-06-01T23:45:00.000-07:00

A RESTful Web App Development PHP Library

http://tonic.sourceforge.net/docs.html

Lightweight REST framework

The mission of this open source project is to bring the simplicity and
efficiency of the REST architectural style to Java developers.
Concretely, it is composed of two parts:

1) Restlet API

Supports all REST concepts (resource, representation, data, connector, components, etc.)
Suitable for both client and server REST applications
Maplets support the concept of URIs as UI with advanced pattern matching features
Chainlets filter calls to implement features like logging, authentication or compression
Complete alternative to Servlet API with no external dependency (JAR < 50kb)
Supports blocking and non-blocking NIO modes

2) Noelios Restlet Engine (NRE)

Reference implementation of the Restlet API provided by Noelios Consulting (core JAR < 60kb)
Server connector provided: HTTP (via Jetty connectors)
Client connectors provided: HTTP, JDBC, SMTP (via JavaMail)
Support for logging (LoggerChainlet) and cool URIs rewriting (RedirectRestlet)
Static files serving (DirectoryRestlet) with metadata association based on file extensions
FreeMarker template representations as an alternative to JSP pages
Automatic server-side content negotiation based on media type and language

Also, an introduction paper as well as a detailled tutorial are available.

http://www.restlet.org/

How reset the auto_increment propertie in mysql table.

2009-05-24T11:15:00.000-07:00

Recently encountered on a 'odd' behaviour of mySQL table with primary key that defined as auto-increment, in a first look the table had only 5 records, which was far away from limit(256), then I've remebered that the table was used by junit runs for inserting/deleting records and apparently the limit was reached, I think that most expected behavior would be to generate an id for primary key that most close to maximum value of primary key.
Anyway in order to reset the auto_increment, in a situation where some of the most recently added rows were deleted, use:

ALTER TABLE theStuckedTable AUTO_INCREMENT=1234

Use command-line tools in PHP

2009-05-23T06:34:00.000-07:00

Gennady recommends this from developerWorks(TM)
---------------------------------------------------------------------

Title: Use command-line tools in PHP

Learn how to better integrate scripts with command-line tools. Emphasis is placed on using shell_exec(), exec(), passthru(), and system(); safely passing information to the command line; and safely retrieving information from it.

Learn more:
http://www.ibm.com/developerworks/opensource/library/os-php-commandline/index.html?ca=drs-

developerWorks
IBM's resource for developers.
http://www.ibm.com/developerworks/

100 Tips and Tools to Research the Social Web

2009-05-20T06:08:00.000-07:00

http://www.linkedin.com/news?actionBar=&sik=1242824767100&aIdx=0&articleID=34853642

50 Useful JavaScript Tools

2009-05-20T05:58:00.000-07:00

http://www.smashingmagazine.com/2009/02/08/50-extremely-useful-javascript-tools/

The Downfall of Agile Hitler

2009-05-19T00:20:00.000-07:00

The Downfall of Agile Hitler

Luke - Lucene Index Toolbox

2009-05-17T05:42:00.000-07:00

Lucene is an Open Source, mature and high-performance Java search engine. It is highly flexible, and scalable from hundreds to millions of documents.

Luke is a handy development and diagnostic tool, which accesses already existing Lucene indexes and allows you to display and modify their content in several ways:

browse by document number, or by term
view documents / copy to clipboard
retrieve a ranked list of most frequent terms
execute a search, and browse the results
analyze search results
selectively delete documents from the index
reconstruct the original document fields, edit them and re-insert to the index
optimize indexes
and much more...

2009-05-16T23:23:00.000-07:00

Finally was launched alpha of computational knowledge engine Wolfram, that did a lot of 'noise' recently, it's has a lot of useful features and expose API for external use:
http://www61.wolframalpha.com/developers.html

The Regex Coach - interactive regular expressions

2009-05-06T01:29:00.000-07:00

An open source tool to check and write regular expressions:

http://www.weitz.de/regex-coach

High Performance Kick Ass Web Apps (JavaScript edition)

2009-05-05T00:53:00.001-07:00

Check out this SlideShare Presentation:

High Performance Kick Ass Web Apps (JavaScript edition)

View more presentations from Stoyan Stefanov.

A memcached implementation in JGroups - PartitionedHashMap

2009-05-04T22:54:00.000-07:00

PartitionedHashMap is an implementation of memcached on top of JGroups, written completely in Java. It has a couple of advantages over memcached:

Java clients and PartitionedHashMap can run in the same address space and therefore don't need to use the memcached protocol to communicate. The latter is text based² and slow, due to serialization. This allows servlets to access the cache directly, without serialization overhead.
All PartitionedHashMap processes know about each other, and can therefore make intelligent decisions as to what to do when a cluster membership change occurs. For example, a server to be stopped can migrate all of the keys it manages to the next server. With memcached, the entries hosted by a server S are lost when S goes down. Of course, this doesn't work when S crashes.
Similat to the above point, when a cluster membership change occurs (e.g. a new server S is started), then all servers check whether an entry hosted by them should actually be hosted by S. They will move all entries to be hosted by S to S. This has the advantage that entries don't have to be re-read from the DB (for example) and inserted into the cache (as in memcached's case), but the cache rebalances itself automatically.
PartitionedHashMap has a level 1 cache (L1 cache). This allows for caching of data near to where it is really needed. For example, if we have servers A, B, C, D and E and a client adds a (to be heavily accessed) news article to C, then memcached would always redirect every single request for the article to C. So, a client accessing D would always trigger a GET request from D to C and then return the article. JGroups caches the article in D's L1 cache on the first access, so all other clients accessing the article from D would get the cached article, and we can avoid a round trip to C. Note that each entry has an expiration time, which will cause the entry to be removed from the L1 cache on expiry, and the next access would have to fetch it again from C and place it in D's L1 cache. The expiration time is defined by the submitter of the article.
Since the RPCs for GETs, SETs and REMOVEs use JGroups as transport, the type of transport and the quality of service can be controlled and customized through the underlying XML file defining the transport. For example, we could add compression, or decide to encrypt all RPC traffic. It also allows for use of either UDP (IP multicasting and/or UDP datagrams) or TCP.
The connector (org.jgroups.blocks.MemcachedConnector) which is responsible for parsing the memcached protocol and invoking requests on PartitionedHashMap, PartitionedHashMap (org.jgroups.blocks.PartitionedHashMap) which represents the memcached implementation, the server (org.jgroups.demos.MemcachedServer) and the L1 and L2 caches (org.jgroups.blocks.Cache) can be assembled or replaced at will. Therefore it is simple to customize the JGroups memcached implementation; for example to use a different MemcachedConnector which parses a binary protocol (requiring matching client code of course).
All management information and operations are exposed via JMX.

http://www.jgroups.org/memcached/memcached.html

Want SEO Results? Implement These 8 Important Techniques(from LinkedIn forum)

2009-05-04T22:43:00.000-07:00

Implementing more than a handful of the Search Engine

Optimization methods we encounter can be a never-ending

struggle, hence the reason for this list. Included below are

the methods that will get you the most favor in the eyes of
the search engines. When you implement these SEO methods,
you will be able to see a noticable difference in your
traffic and wonder why you didn't impletment them sooner.

1. Use your 'title' tag effectively by including your
keywords. Each page should have a specific theme and be
optimized for a certain key phrase anyway, so including your
keywords in the page title should be natural. It just so
happens that search engines eat it up.

2. Get your keywords in your inbound links. If another
page is pointing at you, whatever text is part of that link
goes to your credit in the eyes of the search engines. If
your blogger friend has a link to your website on his, it
probably says "check out Joes business." Try to get him to
put "auto mechanic" or "car repair" as the link, or at least
"Joes Auto Repair" - your business name.

3. Have Unique Content and Update Often. Add something to
the Internet. Become a resource for potential clients and
those in your industry. If you get more page views and
clicks through a search engine, the search engine will value
your site higher. As a side benefit, you are seen as an
expert in the community, which never hurts business.

4. Put your Keywords in your Filenames. Instead of titling
your Auto Repair Services page "arserv.html" or another
shorthand title, use long, descriptive filenames like
"car-auto-repair-services.html." This is a great technique
that is easy to implement and yields great results.

5. Create an XML Sitemap. If you're not used to writing
code, this can seem daunting, but you can always hire
someone. Creating and submitting the sitemap will help
Google find your site and all its content much faster. You
can find the protocol here
( https://www.google.com/webmasters/tools/docs/en/protocol.html ).
Save it as 'sitemap.xml' in the root of your website (the
same directory as your index page).

6. Build a Large Number of Backlinks. There are entire
companies that do nothing but provide backlink generation
services, which should tell you how important this process
is. Simply put, search engines favor sites that are
referenced by other well-referenced sites (the basis of
Google's PageRank). There are countless ways to get
backlinks, but arguably the best way is to use the content
you generated in Step 3. Submit your articles to
directories and content libraries using special programs or
manual submissions. You can even delegate this a
knowledgeable employee (or one that knows how to use Google)
for their downtime. Additionally, search engines favor links
from .gov and .edu domains, so use your connections if you
have them!

7. Keep your Keyword Density Between 3-7%. When youre
writing all that great, unique content, make sure to seed
your keywords and phrases in there so that they comprise
3-7% of the text on a page. Any more, and you look like
youre keyword stuffing (whether youre trying to or not), any
less, and the page doesnt seem relevant.

8. Use your Keywords in your Headings. Many times a section
heading can be seen as irrelevant for a search engine, so
always make sure to fill your 'h1' and 'h2' tags with
important keywords. Dont leave them out or treat them as
'implied' - thats the worst thing you can do.

If you follow these 8 tips, you will rank high for your
target keyword. If your keyword is very competitive, add
more backlinks with your keyword phrase by creating
articles like this, submitting them to article directories,
and placing your link in the 'About' box below. Well, what
are you waiting for?

Choosing the Right Scrum Management Tool

2009-05-04T22:34:00.001-07:00

Just read a two interesting articles about choosing a scrum management tools:

http://tommynorman.blogspot.com/2009/05/choosing-right-scrum-management-tool_04.html

http://tommynorman.blogspot.com/2009/05/choosing-right-scrum-management-tool.html

Yahoo development session notes: YQL, Flickr, OAuth, YAP

2009-03-31T23:37:00.001-07:00

A few notes regarding Yahoo session:
YQL - very nice API, successor of the Yahoo pipes. With you could read and parse user social data, parse RSS or Twiter feeds. Feature that I most liked is possibility to parse recent multiple RSS feeds and extract information you need with a simple query like this:

select title from rss where url="http://rss.news.yahoo.com/rss/topstories" and title like '%GM%'
Currently it's limited up to 10000 requests a day from a single IP address.

OAth - yahoo platform for authentication, standard that all big web guys( Facebook, Google, etc. ) are going to adopt. Very nice feature for web sites that don't want to 'outsource' authentication.

YQL, Flickr, OAuth, YAP

View more presentations from Erik Eldridge.

SMTP properties

2009-03-29T22:51:00.000-07:00

Name	Type	Description
mail.smtp.user	String	Default user name for SMTP.
mail.smtp.host	String	The SMTP server to connect to.
mail.smtp.port	int	The SMTP server port to connect to, if the connect() method doesn't explicitly specify one. Defaults to 25.
mail.smtp.connectiontimeout	int	Socket connection timeout value in milliseconds. Default is infinite timeout.
mail.smtp.timeout	int	Socket I/O timeout value in milliseconds. Default is infinite timeout.
mail.smtp.from	String	Email address to use for SMTP MAIL command. This sets the envelope return address. Defaults to msg.getFrom() or InternetAddress.getLocalAddress(). NOTE: mail.smtp.user was previously used for this.
mail.smtp.localhost	String	Local host name used in the SMTP HELO or EHLO command. Defaults to `InetAddress.getLocalHost().getHostName()`. Should not normally need to be set if your JDK and your name service are configured properly.
mail.smtp.localaddress	String	Local address (host name) to bind to when creating the SMTP socket. Defaults to the address picked by the Socket class. Should not normally need to be set, but useful with multi-homed hosts where it's important to pick a particular local address to bind to.
mail.smtp.localport	int	Local port number to bind to when creating the SMTP socket. Defaults to the port number picked by the Socket class.
mail.smtp.ehlo	boolean	If false, do not attempt to sign on with the EHLO command. Defaults to true. Normally failure of the EHLO command will fallback to the HELO command; this property exists only for servers that don't fail EHLO properly or don't implement EHLO properly.
mail.smtp.auth	boolean	If true, attempt to authenticate the user using the AUTH command. Defaults to false.
mail.smtp.auth.mechanisms	String	If set, lists the authentication mechanisms to consider, and the order in which to consider them. Only mechanisms supported by the server and supported by the current implementation will be used. The default is `"LOGIN PLAIN DIGEST-MD5"`, which includes all the authentication mechanisms supported by the current implementation.
mail.smtp.submitter	String	The submitter to use in the AUTH tag in the MAIL FROM command. Typically used by a mail relay to pass along information about the original submitter of the message. See also the `setSubmitter` method of `SMTPMessage`. Mail clients typically do not use this.
mail.smtp.dsn.notify	String	The NOTIFY option to the RCPT command. Either NEVER, or some combination of SUCCESS, FAILURE, and DELAY (separated by commas).
mail.smtp.dsn.ret	String	The RET option to the MAIL command. Either FULL or HDRS.
mail.smtp.allow8bitmime	boolean	If set to true, and the server supports the 8BITMIME extension, text parts of messages that use the "quoted-printable" or "base64" encodings are converted to use "8bit" encoding if they follow the RFC2045 rules for 8bit text.
mail.smtp.sendpartial	boolean	If set to true, and a message has some valid and some invalid addresses, send the message anyway, reporting the partial failure with a SendFailedException. If set to false (the default), the message is not sent to any of the recipients if there is an invalid recipient address.
mail.smtp.sasl.realm	String	The realm to use with DIGEST-MD5 authentication.
mail.smtp.quitwait	boolean	If set to false, the QUIT command is sent and the connection is immediately closed. If set to true (the default), causes the transport to wait for the response to the QUIT command.
mail.smtp.reportsuccess	boolean	If set to true, causes the transport to include an `SMTPAddressSucceededException` for each address that is successful. Note also that this will cause a `SendFailedException` to be thrown from the `sendMessage` method of `SMTPTransport` even if all addresses were correct and the message was sent successfully.
mail.smtp.socketFactory	SocketFactory	If set to a class that implements the `javax.net.SocketFactory` interface, this class will be used to create SMTP sockets. Note that this is an instance of a class, not a name, and must be set using the `put` method, not the `setProperty` method.
mail.smtp.socketFactory.class	String	If set, specifies the name of a class that implements the `javax.net.SocketFactory` interface. This class will be used to create SMTP sockets.
mail.smtp.socketFactory.fallback	boolean	If set to true, failure to create a socket using the specified socket factory class will cause the socket to be created using the `java.net.Socket` class. Defaults to true.
mail.smtp.socketFactory.port	int	Specifies the port to connect to when using the specified socket factory. If not set, the default port will be used.
mail.smtp.ssl.enable	boolean	If set to true, use SSL to connect and use the SSL port by default. Defaults to false for the "smtp" protocol and true for the "smtps" protocol.
mail.smtp.ssl.checkserveridentity	boolean	If set to true, check the server identity as specified by RFC 2595. These additional checks based on the content of the server's certificate are intended to prevent man-in-the-middle attacks. Defaults to false.
mail.smtp.ssl.socketFactory	SSLSocketFactory	If set to a class that extends the `javax.net.ssl.SSLSocketFactory` class, this class will be used to create SMTP SSL sockets. Note that this is an instance of a class, not a name, and must be set using the `put` method, not the `setProperty` method.
mail.smtp.ssl.socketFactory.class	String	If set, specifies the name of a class that extends the `javax.net.ssl.SSLSocketFactory` class. This class will be used to create SMTP SSL sockets.
mail.smtp.ssl.socketFactory.port	int	Specifies the port to connect to when using the specified socket factory. If not set, the default port will be used.
mail.smtp.ssl.protocols	string	Specifies the SSL protocols that will be enabled for SSL connections. The property value is a whitespace separated list of tokens acceptable to the `javax.net.ssl.SSLSocket.setEnabledProtocols` method.
mail.smtp.ssl.ciphersuites	string	Specifies the SSL cipher suites that will be enabled for SSL connections. The property value is a whitespace separated list of tokens acceptable to the `javax.net.ssl.SSLSocket.setEnabledCipherSuites` method.
mail.smtp.mailextension	String	Extension string to append to the MAIL command. The extension string can be used to specify standard SMTP service extensions as well as vendor-specific extensions. Typically the application should use the `SMTPTransport` method `supportsExtension` to verify that the server supports the desired service extension. See RFC 1869 and other RFCs that define specific extensions.
mail.smtp.starttls.enable	boolean	If true, enables the use of the `STARTTLS` command (if supported by the server) to switch the connection to a TLS-protected connection before issuing any login commands. Note that an appropriate trust store must configured so that the client will trust the server's certificate. Defaults to false.
mail.smtp.starttls.required	boolean	If true, requires the use of the `STARTTLS` command. If the server doesn't support the STARTTLS command, or the command fails, the connect method will fail. Defaults to false.
mail.smtp.userset	boolean	If set to true, use the RSET command instead of the NOOP command in the `isConnected` method. In some cases sendmail will respond slowly after many NOOP commands; use of RSET avoids this sendmail issue. Defaults to false.

Disk-Based Parallel Computation, Rubik's Cube, and Checkpointing

2009-03-19T07:28:00.001-07:00

Nice one.

The Promise and Peril of Jumbo Frames

2009-03-19T01:54:00.000-07:00

The Promise and Peril of Jumbo Frames

via Coding Horror by Jeff Atwood on 3/1/09

We sit at the intersection of two trends:

Most home networking gear, including routers, has safely transitioned to gigabit ethernet.
The generation, storage, and transmission of large high definition video files is becoming commonplace.

If that sounds like you, or someone you know, there's one tweak you should know about that can potentally improve your local network throughput quite a bit -- enabling Jumbo Frames.

The typical UDP packet looks something like this:

But the default size of that data payload was established years ago. In the context of gigabit ethernet and the amount of data we transfer today, it does seem a bit.. anemic.

The original 1,518-byte MTU for Ethernet was chosen because of the high error rates and low speed of communications. If a corrupted packet is sent, only 1,518 bytes must be re-sent to correct the error. However, each frame requires that the network hardware and software process it. If the frame size is increased, the same amount of data can be transferred with less effort. This reduces CPU utilization (mostly due to interrupt reduction) and increases throughput by allowing the system to concentrate on the data in the frames, instead of the frames around the data.

I use my beloved energy efficient home theater PC as an always-on media server, and I'm constantly transferring gigabytes of video, music, and photos to it. Let's try enabling jumbo frames for my little network.

The first thing you'll need to do is update your network hardware drivers to the latest versions. I learned this the hard way, but if you want to play with advanced networking features like Jumbo Frames, you need the latest and greatest network hardware drivers. What was included with the OS is unlikely to cut it. Check on the network chipset manufacturer's website.

Once you've got those drivers up to date, look for the Jumbo Frames setting in the advanced properties of the network card. Here's what it looks like on two different ethernet chipsets:

That's my computer, and the HTPC, respectively. I was a little disturbed to notice that neither driver recognizes exactly the same data payload size. It's named "Jumbo Frame" with 2KB - 9KB settings in 1KB increments on the Realtek, and "Jumbo Packet" with 4088 or 9014 settings on the Marvell. I know that technically, for jumbo frames to work, all the networking devices on the subnet have to agree on the data payload size. I couldn't tell quite what to do, so I set them as you see above.

(I didn't change anything on my router / switch, which at the moment is the D-Link DGL-4500; note that most gigabit switches support jumbo frames, but you should always verify with the manufacturer's website to be sure.)

I then ran a few tests to see if there was any difference. I started with a simple file copy.

Default network settings

Jumbo Frames enabled

My file copy went from 47.6 MB/sec to 60.0 MB/sec. Not too shabby! But this is a very ad hoc sort of testing. Let's see what the PassMark Network Benchmark has to say.

Default network settings

Jumbo Frames enabled

This confirms what I saw with the file copy. With jumbo frames enabled, we go from 390,638 kilobits/sec to 477,927 kilobits/sec average. A solid 20% improvement.

Now, jumbo frames aren't a silver bullet. There's a reason jumbo frames are never enabled by default: some networking equipment can't deal with the non-standard frame sizes. Like all deviations from default settings, it is absolutely possible to make your networking worse by enabling jumbo frames, so proceed with caution. This SmallNetBuilder article outlines some of the pitfalls:

1) For a large frame to be transmitted intact from end to end, every component on the path must support that frame size.

The switch(es), router(s), and NIC(s) from one end to the other must all support the same size of jumbo frame transmission for a successful jumbo frame communication session.

2) Switches that don't support jumbo frames will drop jumbo frames.

In the event that both ends agree to jumbo frame transmission, there still needs to be end-to-end support for jumbo frames, meaning all the switches and routers must be jumbo frame enabled. At Layer 2, not all gigabit switches support jumbo frames. Those that do will forward the jumbo frames. Those that don't will drop the frames.

3) For a jumbo packet to pass through a router, both the ingress and egress interfaces must support the larger packet size. Otherwise, the packets will be dropped or fragmented.

If the size of the data payload can't be negotiated (this is known as PMTUD, packet MTU discovery) due to firewalls, the data will be dropped with no warning, or "blackholed". And if the MTU isn't supported, the data will have to be fragmented to a supported size and retransmitted, reducing throughput.

In addition to these issues, large packets can also hurt latency for gaming and voice-over-IP applications. Bigger isn't always better.

Still, if you regularly transfer large files, jumbo frames are definitely worth looking into. My tests showed a solid 20% gain in throughput, and for the type of activity on my little network, I can't think of any downside.

280 North’s Atlas Bridges The Gap Between Web Apps And Native iPhone Applications

2009-03-05T03:35:00.000-08:00

FOWA: Atlas from mobilecrunch on Vimeo

http://cappuccino.org/learn/
http://280atlas.com/

System calls analysys tool

2009-03-03T07:07:00.000-08:00

strace -cp 18875

% time     seconds usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
77.61    0.019695           0    782458       read
12.31    0.003123           0      9456        write
3.60    0.000913           0     47291        fcntl64
3.32    0.000843           0      9463         open
1.34    0.000341           0     18920        close
1.27    0.000323           0     18914        dup2
0.55    0.000140           0      9457         rt_sigprocmask
0.00    0.000000           0         6           fstat64
0.00    0.000000           0        12          getdents64
------ ----------- ----------- --------- --------- ----------------
100.00    0.025378                895977           total

Numbers Everyone Should Know

2009-03-03T06:43:00.000-08:00

Google AppEngine Numbers

This group of numbers is from Brett Slatkin in Building Scalable Web Apps with Google App Engine.

Writes are expensive!

Datastore is transactional: writes require disk access

Disk access means disk seeks

Rule of thumb: 10ms for a disk seek

Simple math: 1s / 10ms = 100 seeks/sec maximum

Depends on:
* The size and shape of your data
* Doing work in batches (batch puts and gets)

Reads are cheap!

Reads do not need to be transactional, just consistent

Data is read from disk once, then it's easily cached

All subsequent reads come straight from memory

Rule of thumb: 250usec for 1MB of data from memory

Simple math: 1s / 250usec = 4GB/sec maximum
* For a 1MB entity, that's 4000 fetches/sec

Numbers Miscellaneous

This group of numbers is from a presentation Jeff Dean gave at a Engineering All-Hands Meeting at Google.

L1 cache reference 0.5 ns

Branch mispredict 5 ns

L2 cache reference 7 ns

Mutex lock/unlock 100 ns

Main memory reference 100 ns

Compress 1K bytes with Zippy 10,000 ns

Send 2K bytes over 1 Gbps network 20,000 ns

Read 1 MB sequentially from memory 250,000 ns

Round trip within same datacenter 500,000 ns

Disk seek 10,000,000 ns

Read 1 MB sequentially from network 10,000,000 ns

Read 1 MB sequentially from disk 30,000,000 ns

Send packet CA->Netherlands->CA 150,000,000 ns

The Lessons

Writes are 40 times more expensive than reads.

Global shared data is expensive. This is a fundamental limitation of distributed systems. The lock contention in shared heavily written objects kills performance as transactions become serialized and slow.

Architect for scaling writes.

Optimize for low write contention.

Optimize wide. Make writes as parallel as you can.

Web 2.0 Tools For Project Management

2009-03-03T05:46:00.001-08:00

Web 2.0 Tools For Project Management

View more presentations from Douglas Tokuno. (tags: productivity 2.0)

10 Papers Every Programmer Should Read

2009-03-01T22:34:00.000-08:00

On the criteria to be used in decomposing systems into modules – David Parnas
A Note On Distributed Computing – Jim Waldo, Geoff Wyant, Ann Wollrath, Sam Kendall
The Next 700 Programming Languages – P. J. Landin
Can Programming Be Liberated from the von Neumann Style? – John Backus
Reflections on Trusting Trust – Ken Thompson
Lisp: Good News, Bad News, How to Win Big – Richard Gabriel
An experimental evaluation of the assumption of independence in multiversion programming – John Knight and Nancy Leveson
Arguments and Results – James Noble
A Laboratory For Teaching Object-Oriented Thinking – Kent Beck, Ward Cunningham
Programming as an Experience: the inspiration for Self – David Ungar, Randall B. Smith

How LDAP Error Codes Map to JNDI Exceptions

2009-02-17T04:30:00.000-08:00

How LDAP Error Codes Map to JNDI Exceptions

The LDAP defines a set of status codes that are returned with LDAP responses sent by the LDAP server (see RFC 2251). In the JNDI, error conditions are indicated as checked exceptions that are subclasses of NamingException. See the Naming Exceptions section for an overview of the JNDI exception classes.

The LDAP service provider translates the LDAP status code it receives from the LDAP server to the appropriate subclass of NamingException. The following table shows the mapping between LDAP status codes and JNDI exceptions.

LDAP Status Code	Meaning	Exception or Action
0	Success	Report success.
1	Operations error	`NamingException`
2	Protocol error	`CommunicationException`
3	Time limit exceeded.	`TimeLimitExceededException`
4	Size limit exceeded.	`SizeLimitExceededException`
5	Compared false.	Used by `DirContext.search()`. Does not generate an exception.
6	Compared true.	Used by `DirContext.search()`. Does not generate an exception.
7	Authentication method not supported.	`AuthenticationNotSupportedException`
8	Strong authentication required.	`AuthenticationNotSupportedException`
9	Partial results being returned.	If the environment property `"java.naming.referral"` is set to `"ignore"` or the contents of the error do not contain a referral, throw a `PartialResultException`. Otherwise, use contents to build a referral.
10	Referral encountered.	If the environment property `"java.naming.referral"` is set to `"ignore"`, then ignore. If the property is set to `"throw"`, throw `ReferralException`. If the property is set to `"follow"`, then the LDAP provider processes the referral. If the `"java.naming.ldap.referral.limit"` property has been exceeded, throw `LimitExceededException`.
11	Administrative limit exceeded.	`LimitExceededException`
12	Unavailable critical extension requested.	`OperationNotSupportedException`
13	Confidentiality required.	`AuthenticationNotSupportedException`
14	SASL bind in progress.	Used internally by the LDAP provider during authentication.
16	No such attribute exists.	`NoSuchAttributeException`
17	An undefined attribute type.	`InvalidAttributeIdentifierException`
18	Inappropriate matching	`InvalidSearchFilterException`
19	A constraint violation.	`InvalidAttributeValueException`
20	An attribute or value already in use.	`AttributeInUseException`
21	An invalid attribute syntax.	`InvalidAttributeValueException`
32	No such object exists.	`NameNotFoundException`
33	Alias problem	`NamingException`
34	An invalid DN syntax.	`InvalidNameException`
35	Is a leaf.	Used by the LDAP provider; usually doesn't generate an exception.
36	Alias dereferencing problem	`NamingException`
48	Inappropriate authentication	`AuthenticationNotSupportedException`
49	Invalid credentials	`AuthenticationException`
50	Insufficient access rights	`NoPermissionException`
51	Busy	`ServiceUnavailableException`
52	Unavailable	`ServiceUnavailableException`
53	Unwilling to perform	`OperationNotSupportedException`
54	Loop detected.	`NamingException`
64	Naming violation	`InvalidNameException`
65	Object class violation	`SchemaViolationException`
66	Not allowed on non-leaf.	`ContextNotEmptyException`
67	Not allowed on RDN.	`SchemaViolationException`
68	Entry already exists.	`NameAlreadyBoundException`
69	Object class modifications prohibited.	`SchemaViolationException`
71	Affects multiple DSAs.	`NamingException`
80	Other	`NamingException`

Distributed databases are going to replace RDMS?

2009-01-19T23:05:00.000-08:00

Richard Jones of Last.fm just posted an overview with a great deal of engineering insight:

http://www.metabrew.com/article/anti-rdbms-a-list-of-distributed-key-value-stores/

Swaroop CH of Yahoo wrote an overview of distributed DBs:

http://www.swaroopch.com/notes/Distributed_Storage_Systems

The End of an Architectural Era(It’s Time for a Complete Rewrite)

http://www.vldb.org/conf/2007/papers/industrial/p1150-stonebraker.pdf

The HBase performance

http://www.mail-archive.com/hadoop-user@lucene.apache.org/msg02540.html

Another HBase clone: Voldermort project

2009-01-15T02:13:00.000-08:00

http://project-voldemort.com/

JBoss tools 3 release candidate

2009-01-15T01:28:00.000-08:00

JBoss tools 3 release candidate was released, could be taken from here

Mount new disk on CentOS linux

2009-01-08T01:16:00.000-08:00

Check what is the new hard disk device name with "fdisk -l", it should be something like /dev/sdb. You can easily identify which is the new drive by running "mount" and finding the drive that exists in "fdisk -l" but is not mounted.
```
mount; fdisk -l;
```
Create a partition on the new drive, (the sample code assumes the disk is /dev/sdb)
```
echo -ne "n\np\n1\n\n\nw\n" | fdisk /dev/sdb1
```
Create a filesystem on the new partition, we use ext3 file system.
```
 mkfs.ext3 /dev/sdb1
```
Create a directory where to mount the partition
```
mkdir /mount/data2
```
Edit /etc/fstab an add a record for the new drive at the end of the file. This will make the server mount the drive automatically after reboot. Mount options (like noatime and nodiratime) can be added as a comma separated list of values after "defaults": "defaults,noatime,nodiratime"
```
echo "/dev/sdb1  /mount/data2  ext3  defaults 0 0" >> /etc/fstab
```

Linux records information about when files were created and last modified as well as when it was last accessed. There is a cost associated with recording the last access time. Linux has a special mount option for file systems called noatime that can be added to each line that addresses one file system in the /etc/fstab file. If a file system has been mounted with this option, reading accesses to the file system will no longer result in an update to the atime information associated with the file. The importance of the noatime setting is that it eliminates the need by the system to make writes to the file system for files which are simply being read. Since writes can be somewhat expensive, this can result in measurable performance gains."

nodiratime does the same thing but for directories. I know the beginners guide says to use both mount options on filesystems, but from others I've talked to and places I've read it seems noatime implies nodiratime because noatime is a superset and nodiratime is a subset used specifically to disable it for directories but leave it on for files, and when you use noatime, it does it for everything, files/dirs

   echo "/dev/sdb1  /mount/data2  ext3  rw,noatime,nodiratime 0 0" >> /etc/fstab

Mount the drive. "mount -a" just mounts everything according to /etc/fstab.

```
mount -a
```
Restart the server to make sure it starts ok with the new drive mounted.
```
shutdown -r now
```

An Introduction to ZooKeeper from Yahoo DN

2009-01-08T00:45:00.000-08:00

An introduction to ZooKeeper from YDN

Tasks management on Linux using crontab

2008-12-28T05:01:00.000-08:00

Crontab

The crontab (cron derives fromchronos, Greek for time; tab stands fortable) command, found in Unix and Unix-like operating systems, is used to schedule commands to be executed periodically. To see what crontabs are currently running on your system, you can open a terminal and run:

crontab -l

To edit the list of cronjobs you can run:

crontab -e

This will open a the default editor ( vi or pico) to manipulate the crontab settings. If you save and exit the editor, all your cronjobs are saved into crontab. Cronjobs are written in the following format:

* * * * * /bin/execute/this/script.sh

Scheduling explained

As you can see there are 5 stars. The stars represent different date parts in the following order:

minute (from 0 to 59)
hour (from 0 to 23)
day of month (from 1 to 31)
month (from 1 to 12)
day of week (from 0 to 6) (0=Sunday)

Execute every minute

If you leave the star, or asterisk, it means every. Maybe that's a bit unclear. Let's use the the previous example again:

* * * * * /bin/execute/this/script.sh

They are all still asterisks! So this means execute /bin/execute/this/script.sh:

every minute
of every hour
of every day of the month
of every month
and every day in the week.

In short: This script is being executed every minute. Without exception.

Execute every Friday 1AM

So if we want to schedule the script to run at 1AM every Friday, we would need the following cronjob:

0 1 * * 5 /bin/execute/this/script.sh

Get it? The script is now being executed when the system clock hits:

minute: 0
of hour: 1
of day of month: * (every day of month)
of month: * (every month)
and weekday: 5 (=Friday)

Execute 10 past after every hour on the 1st of every month

Here's another one, just for practicing

10 * 1 * * /bin/execute/this/script.sh

Fair enough, it takes some getting used to, but it offers great flexibility.

If you want to run something every 10 minutes

0,10,20,30,40,50 * * * * /bin/execute/this/script.sh

Mailing the crontab output of just one cronjob

And change the cronjob like this:*/10 * * * * /bin/execute/this/script.sh 2>&1 | mail -s "Cronjob ouput" yourname@

Running Hadoop with HBase On CentOS Linux (Multi-Node Cluster)

2008-12-25T02:41:00.000-08:00

1.Prerequisites

1.1 Configure single nodes

Use my tutorial "Running Hadoop with HBase On CentOS Linux (Single-Node Cluster)"

1.2 SSH public access

Add mapping to /etc/hosts for each your nodes:

10.1.0.55 master
10.1.0.56 slave

To do that you need to add public key from master to slave ~/.ssh/authorized_keys and via verse, so eventually tou should be able to do the following:


[root@37 /usr/local/bin/hbase]ssh master
The authenticity of host 'master (10.1.0.55)' can't be established.
RSA key fingerprint is 09:e2:73:ac:6f:42:d1:da:13:20:76:10:36:29:c4:62.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'master,10.1.0.55' (RSA) to the list of known hosts.
Last login: Thu Dec 25 07:19:16 2008 from slave
[root@37 ~]exit
logout
Connection to master closed.
[root@37 /usr/local/bin/hbase]ssh slave
The authenticity of host 'slave (10.1.0.56)' can't be established.
RSA key fingerprint is 5c:4f:81:19:ad:f3:78:02:ce:64:f1:67:10:ca:c5:b8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'slave,10.1.0.56' (RSA) to the list of known hosts.
Last login: Thu Dec 25 07:16:47 2008 from 38.d.de.static.xlhost.com
[root@38 ~]exit
logout
Connection to slave closed.

2.Cluster Overview

The master node will also act as a slave because we only have two machines available in our cluster but still want to spread data storage and processing to multiple machines.

The master node will run the "master" daemons for each layer: namenode for the HDFS storage layer, and jobtracker for the MapReduce processing layer. Both machines will run the "slave" daemons: datanode for the HDFS layer, and tasktracker for MapReduce processing layer. Basically, the "master" daemons are responsible for coordination and management of the "slave" daemons while the latter will do the actual data storage and data processing work.

3. Hadoop core Configuration

3.1 conf/masters (master only)

The conf/masters file defines the master nodes of our multi-node cluster.

On master, update $HADOOP_HOME/conf/masters that it looks like this:

  
master

3.2 conf/slaves (master only)

This conf/slaves file lists the hosts, one per line, where the Hadoop slave daemons (datanodes and tasktrackers) will run. We want both the master box and the slave box to act as Hadoop slaves because we want both of them to store and process data.

On master, update $HADOOP_HOME/conf/slaves that it looks like this:


master
slave

If you have additional slave nodes, just add them to the conf/slaves file, one per line (do this on all machines in the cluster).

 master
slave
slave01
slave02
slave03

3.3 conf/hadoop-site.xml (all machines)

Following a sample configuration for all hosts, explanation of all parameters link





hadoop.tmp.dir
/usr/local/bin/hadoop/datastore/hadoop-${user.name}
A base for other temporary directories.



fs.default.name
hdfs://master:54310
The name of the default file system.  A URI whose
scheme and authority determine the FileSystem implementation.  The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class.  The uri's authority is used to
determine the host, port, etc. for a filesystem.



mapred.job.tracker
master:54311
The host and port that the MapReduce job tracker runs
at.  If "local", then jobs are run in-process as a single map
and reduce task.




dfs.replication
1
Default block replication.
The actual number of replications can be specified when the file is created.
The default is used if replication is not specified in create time.



mapred.reduce.tasks
8
The default number of reduce tasks per job.  Typically set
to a prime close to the number of available hosts.  Ignored when
mapred.job.tracker is "local".



mapred.tasktracker.reduce.tasks.maximum
8
The maximum number of reduce tasks that will be run
simultaneously by a task tracker.



mapred.child.java.opts
-Xmx500m
Java opts for the task tracker child processes.
The following symbol, if present, will be interpolated: @taskid@ is replaced
by current TaskID. Any other occurrences of '@' will go unchanged.
For example, to enable verbose gc logging to a file named for the taskid in
/tmp and to set the heap maximum to be a gigabyte, pass a 'value' of:
-Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc

The configuration variable mapred.child.ulimit can be used to control the
maximum virtual memory of the child processes.

Very important: now format each node data storage before proceeding.

3.4 Start/Stop Hadoop cluster

HDFS daemons: start/stop namenode command bin/start-dfs.sh or bin/stop-dfs.sh
MapReduce daemons: run the command $HADOOP_HOME/bin/start-mapred.sh on the machine you want the jobtracker to run on. This will bring up the MapReduce cluster with the jobtracker running on the machine you ran the previous command on, and tasktrackers on the machines listed in the conf/slaves file.

[root@37 /usr/local/bin/hadoop]bin/start-dfs.sh
starting namenode, logging to /usr/local/bin/hadoop/bin/../logs/hadoop-root-namenode-37.c3.33.static.xlhost.com.out
master: starting datanode, logging to /usr/local/bin/hadoop/bin/../logs/hadoop-root-datanode-37.c3.33.static.xlhost.com.out
slave: starting datanode, logging to /usr/local/bin/hadoop/bin/../logs/hadoop-root-datanode-38.c3.33.static.xlhost.com.out
master: starting secondarynamenode, logging to /usr/local/bin/hadoop/bin/../logs/hadoop-root-secondarynamenode-37.c3.33.static.xlhost.com.out
[root@37 /usr/local/bin/hadoop]bin/start-mapred.sh
starting jobtracker, logging to /usr/local/bin/hadoop/bin/../logs/hadoop-root-jobtracker-37.c3.33.static.xlhost.com.out
master: starting tasktracker, logging to /usr/local/bin/hadoop/bin/../logs/hadoop-root-tasktracker-37.c3.33.static.xlhost.com.out
slave: starting tasktracker, logging to /usr/local/bin/hadoop/bin/../logs/hadoop-root-tasktracker-38.c3.33.static.xlhost.com.out
[root@37 /usr/local/bin/hadoop]jps]
-bash: jps]: command not found
[root@37 /usr/local/bin/hadoop]jps
28638 DataNode
29052 Jps
28527 NameNode
28764 SecondaryNameNode
28847 JobTracker
28962 TaskTracker
[root@37 /usr/local/bin/hadoop]bin/hadoop dfs -copyFromLocal LICENSE.txt testWordCount
[root@37 /usr/local/bin/hadoop]bin/hadoop dfs -ls
Found 1 items
-rw-r--r--   1 root supergroup      11358 2008-12-25 08:20 /user/root/testWordCount
[root@37 /usr/local/bin/hadoop]bin/hadoop jar hadoop-0.18.2-examples.jar wordcount testWordCount testWordCount-out
08/12/25 08:21:46 INFO mapred.FileInputFormat: Total input paths to process : 1
08/12/25 08:21:46 INFO mapred.FileInputFormat: Total input paths to process : 1
08/12/25 08:21:47 INFO mapred.JobClient: Running job: job_200812250818_0001
08/12/25 08:21:48 INFO mapred.JobClient:  map 0% reduce 0%
08/12/25 08:21:53 INFO mapred.JobClient:  map 50% reduce 0%
08/12/25 08:21:55 INFO mapred.JobClient:  map 100% reduce 0%
08/12/25 08:22:01 INFO mapred.JobClient:  map 100% reduce 12%
08/12/25 08:22:02 INFO mapred.JobClient:  map 100% reduce 25%
08/12/25 08:22:05 INFO mapred.JobClient:  map 100% reduce 41%
08/12/25 08:22:09 INFO mapred.JobClient:  map 100% reduce 52%
08/12/25 08:22:14 INFO mapred.JobClient:  map 100% reduce 64%
08/12/25 08:22:19 INFO mapred.JobClient:  map 100% reduce 66%
08/12/25 08:22:24 INFO mapred.JobClient:  map 100% reduce 77%
08/12/25 08:22:29 INFO mapred.JobClient:  map 100% reduce 79%
08/12/25 08:25:30 INFO mapred.JobClient:  map 100% reduce 89%
08/12/25 08:26:12 INFO mapred.JobClient: Job complete: job_200812250818_0001
08/12/25 08:26:12 INFO mapred.JobClient: Counters: 17
08/12/25 08:26:12 INFO mapred.JobClient:   Job Counters
08/12/25 08:26:12 INFO mapred.JobClient:     Data-local map tasks=1
08/12/25 08:26:12 INFO mapred.JobClient:     Launched reduce tasks=9
08/12/25 08:26:12 INFO mapred.JobClient:     Launched map tasks=2
08/12/25 08:26:12 INFO mapred.JobClient:     Rack-local map tasks=1
08/12/25 08:26:12 INFO mapred.JobClient:   Map-Reduce Framework
08/12/25 08:26:12 INFO mapred.JobClient:     Map output records=1581
08/12/25 08:26:12 INFO mapred.JobClient:     Reduce input records=593
08/12/25 08:26:12 INFO mapred.JobClient:     Map output bytes=16546
08/12/25 08:26:12 INFO mapred.JobClient:     Map input records=202
08/12/25 08:26:12 INFO mapred.JobClient:     Combine output records=1292
08/12/25 08:26:12 INFO mapred.JobClient:     Map input bytes=11358
08/12/25 08:26:12 INFO mapred.JobClient:     Combine input records=2280
08/12/25 08:26:12 INFO mapred.JobClient:     Reduce input groups=593
08/12/25 08:26:12 INFO mapred.JobClient:     Reduce output records=593
08/12/25 08:26:12 INFO mapred.JobClient:   File Systems
08/12/25 08:26:12 INFO mapred.JobClient:     HDFS bytes written=6117
08/12/25 08:26:12 INFO mapred.JobClient:     Local bytes written=19010
08/12/25 08:26:12 INFO mapred.JobClient:     HDFS bytes read=13872
08/12/25 08:26:12 INFO mapred.JobClient:     Local bytes read=8620

4. HBase configuration

4.1 $HBASE_HOME/hbase-site.sh




hbase.rootdir
hdfs://master:54310/hbase
The directory shared by region servers.



hbase.master
master:60000
The host and port that the HBase master runs at.

4.2 $HBASE_HOME/conf/regionservers


master
slave

4.3 Start HBase on master

Use command $HBASE_HOME/bin/hbase-start.sh


[root@37 /usr/local/bin/hbase]bin/start-hbase.sh
starting master, logging to /usr/local/bin/hbase/bin/../logs/hbase-root-master-37.c3.33.static.xlhost.com.out
slave: starting regionserver, logging to /usr/local/bin/hbase/bin/../logs/hbase-root-regionserver-38.c3.33.static.xlhost.com.out
master: starting regionserver, logging to /usr/local/bin/hbase/bin/../logs/hbase-root-regionserver-37.c3.33.static.xlhost.com.out
[root@37 /usr/local/bin/hbase]jps
30362 SecondaryNameNode
30455 JobTracker
30570 TaskTracker
30231 DataNode
32003 Jps
31772 HMaster
30115 NameNode
31919 HRegionServer

On slave you suppose to see the following output:


[root@38 /usr/local/bin/hbase]jps
513 DataNode
6360 Jps
6214 HRegionServer
628 TaskTracker

Code highlighting in blogspot

2008-12-24T06:41:00.000-08:00

Adding syntax highlight to blogger is very simple, first of all you need to download SyntaxHighlighter.

Now extract the contents of the package and upload the Scripts and Styles folder to any host or website which can be linked from your blog.

To make it work, you will need to edit your blog's template and add the following code after the -- end outer-wrapper -- tag:

To add some java code to your blog use:

The list of supported languages and their aliases can be found at the syntaxhighlighter wiki

Running Hadoop with HBase on CentOS Linux (Single-Node Cluster)

2008-12-24T00:06:00.000-08:00

1.Purpose of this tutorial:

In this tutorial, I will describe the required steps for setting up a single-node Hadoop/Hbase cluster using the Hadoop Distributed File System (HDFS) on CentOS Linux.

Hadoop is a framework written in Java for running applications on large clusters of commodity hardware and incorporates features similar to those of the Google File System and of MapReduce. HDFS is a highly fault-tolerant distributed file system and like Hadoop designed to be deployed on low-cost hardware. It provides high throughput access to application data and is suitable for applications that have large data sets.

HBase is the Hadoop database. Its an open-source, distributed, column-oriented store modeled after the Google paper, Bigtable: A Distributed Storeage System for Structured Data by Chang et al. Just as Bigtable leverages the distributed data storage provided by the Google File System, HBase provides Bigtable-like capabilities on top of Hadoop.

HBase's goal is the hosting of very large tables -- billions of rows X millions of columns -- atop clusters of commodity hardware.

2.Prerequisites

CentOS Linux.
Hadoop 0.18.2


[root@localhost bin]# wget 'http://mirror.mirimar.net/apache/hadoop/core/hadoop-0.18.2/hadoop-0.18.2.tar.gz'
--02:49:53--  http://mirror.mirimar.net/apache/hadoop/core/hadoop-0.18.2/hadoop-0.18.2.tar.gz
Resolving mirror.mirimar.net... 194.90.150.47
Connecting to mirror.mirimar.net|194.90.150.47|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 16836495 (16M) [application/x-gzip]
Saving to: `hadoop-0.18.2.tar.gz'

100%[=========================================================================================================================================>] 16,836,495   471K/s   in 36s

HBase 0.18.1


[root@localhost bin]# wget 'http://mirror.mirimar.net/apache/hadoop/hbase/hbase-0.18.1/hbase-0.18.1.tar.gz'
--02:51:42--  http://mirror.mirimar.net/apache/hadoop/hbase/hbase-0.18.1/hbase-0.18.1.tar.gz
Resolving mirror.mirimar.net... 194.90.150.47
Connecting to mirror.mirimar.net|194.90.150.47|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 16295734 (16M) [application/x-gzip]
Saving to: `hbase-0.18.1.tar.gz'

100%[=========================================================================================================================================>] 16,295,734   496K/s   in 34s

JDK1.5.14 from Sun
SSH
Edit environment sessings


[root@34 ~]# vi ~/.bash_profile
// JDK installation directory
export JAVA_HOME=/usr/local/bin/jdk1.5.0_14
PATH=$JAVA_HOME/bin:$PATH:$HOME/bin
export HADOOP_HOME=/usr/local/bin/hadoop
export HBASE_HOME=/usr/local/bin/hbase
//optional
export PS1="[\u@\h \w]"

Please don't forget to add all environment variables to ~/.bash_profile, otherwise all exports will be deleted after you disconnect your SSH session.

2.1 Java

For now we'll use HBase 0.18.1 which was compiled with JDK1.5 and Hadoop 0.18.2 which supports jdk1.5.x, today Hadoop 0.19.0 is available, but it requires jdk1.6, HBase suppose to support jdk1.6 only in version 0.19

Install JDK1.5.14,


[root@localhost /usr/local/bin]wget 'http://cds.sun.com/is-bin/INTERSHOP.enfinity/WFS/CDS-CDS_Developer-Site/en_US/-/USD/VerifyItem-Start/jdk-1_5_0_14-linux-i586.bin?BundledLineItemUUID=PspIBe.oQkIAAAEeWNw8f4HX&OrderID=na1IBe.ouqIAAAEePNw8f4HX&ProductID=YOzACUFBuXAAAAEYlak5AXuQ&FileName=/jdk-1_5_0_14-linux-i586.bin' -O jdk-1_5_0_14-linux-i586.bin
--03:25:00--  http://cds.sun.com/is-bin/INTERSHOP.enfinity/WFS/CDS-CDS_Developer-Site/en_US/-/USD/VerifyItem-Start/jdk-1_5_0_14-linux-i586.bin?BundledLineItemUUID=PspIBe.oQkIAAAEeWNw8f4HX&OrderID=na1IBe.ouqIAAAEePNw8f4HX&ProductID=YOzACUFBuXAAAAEYlak5AXuQ&FileName=/jdk-1_5_0_14-linux-i586.bin
Resolving cds.sun.com... 72.5.239.134
Connecting to cds.sun.com|72.5.239.134|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://cds-esd.sun.com/ESD37/JSCDL/jdk/1.5.0_14/jdk-1_5_0_14-linux-i586.bin?AuthParam=1230539222_7e7c419133c9fb57c076e8e08293fd8c&TicketId=B%2Fw5lxuBTFhPQB1LOFJTnQTr&GroupName=CDS&FilePath=/ESD37/JSCDL/jdk/1.5.0_14/jdk-1_5_0_14-linux-i586.bin&File=jdk-1_5_0_14-linux-i586.bin [following]
--03:25:01--  http://cds-esd.sun.com/ESD37/JSCDL/jdk/1.5.0_14/jdk-1_5_0_14-linux-i586.bin?AuthParam=1230539222_7e7c419133c9fb57c076e8e08293fd8c&TicketId=B%2Fw5lxuBTFhPQB1LOFJTnQTr&GroupName=CDS&FilePath=/ESD37/JSCDL/jdk/1.5.0_14/jdk-1_5_0_14-linux-i586.bin&File=jdk-1_5_0_14-linux-i586.bin
Resolving cds-esd.sun.com... 98.27.88.9, 98.27.88.39
Connecting to cds-esd.sun.com|98.27.88.9|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 49649265 (47M) [application/x-sdlc]
Saving to: `jdk-1_5_0_14-linux-i586.bin'

100%[=========================================================================================================================================>] 49,649,265  5.36M/s   in 9.0s

03:25:11 (5.25 MB/s) - `jdk-1_5_0_14-linux-i586.bin' saved [49649265/49649265]

define JAVA_HOME and add $JAVA_HOME/bin to $PATH

2.2 SSH public key authentication

Hadoop requires SSH public access to manage its nodes, i.e. remote machines plus your local machine if you want to use Hadoop on it, so

[root@rt ~/.ssh] ssh-keygen -t rsa
//This will create two files in your ~/.ssh directory 
//id_rsa: your private key
//id_rsa.pub: is your public key.
[root@rt ~/.ssh] cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[root@rt ~/.ssh] ssh localhost
The authenticity of host 'localhost (127.0.0.1)' can't be established.
RSA key fingerprint is 55:d7:91:86:ea:86:8f:51:89:9f:68:b0:75:88:52:72.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
[root@rt ~]

As you see the final test is to see if you able to make ssh public authentication connection to the localhost

If the SSH connection fails, these general tips might help:

Enable debugging with ssh -vvv localhost and investigate the error in detail.
Check the SSH server configuration in /etc/ssh/sshd_config, in particular the options PubkeyAuthentication (which should be set to yes) and AllowUsers (if this option is active, add the hadoop user to it). If you made any changes to the SSH server configuration file, you can force a configuration reload with sudo /etc/init.d/ssh reload.

2.3 Disabling IPv6

To disable IPv6 on CentOS Linux, open /etc/modprobe.d/blacklist in the editor of your choice and add the following lines to the end of the file:

 # disable IPv6
 blacklist ipv6

You have to reboot your machine in order to make the changes take effect.

2.4 Edit open files limit.

Edit file /etc/security/limits.conf, add the following lines:

root - nofile 100000

root - locks 100000

Run ulimit -n 1000000 in shell.

3.Hadoop

3.1 Installation

- Unpack hadoop archive to /usr/local/bin (could any directory)

- Move unpacked directory to /usr/local/bin/hadoop: mv hadoop.18.2 hadoop

- Set HADOOP_HOME: export HADOOP_HOME=/usr/local/bin/hadoop

3.2 Configuration

Set up JAVA_HOME in $HADOOP_HOME/conf/hadoop-env.sh to point to your java location:


//The java implementation to use.  Required.
export JAVA_HOME=/usr/local/bin/jdk1.5.0_14

Set up $HADOOP_HOME/conf/hadoop-site.sh

Any site-specific configuration of Hadoop is configured in $HADOOP_HOME/conf/hadoop-site.xml. Here we will configure the directory where Hadoop will store its data files, the ports it listens to, etc. Our setup will use Hadoop's Distributed File System, HDFS, even though our little "cluster" only contains our single local machine.

You can leave the settings below as is with the exception of the hadoop.tmp.dir variable which you have to change to the directory of your choice, for example:

/usr/local/hadoop-datastore/hadoop-${user.name}.

Hadoop will expand ${user.name} to the system user which is running Hadoop, so in our case this will be hadoop and thus the final path will be /usr/local/hadoop-datastore/hadoop-hadoop.






hadoop.tmp.dir
/usr/local/bin/hadoop/datastore/hadoop-${user.name}
A base for other temporary directories.



fs.default.name
hdfs://master:54310
The name of the default file system.  A URI whose
scheme and authority determine the FileSystem implementation.  The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class.  The uri's authority is used to
determine the host, port, etc. for a filesystem.



mapred.job.tracker
master:54311
The host and port that the MapReduce job tracker runs
at.  If "local", then jobs are run in-process as a single map
and reduce task.



mapred.reduce.tasks
4
The default number of reduce tasks per job.  Typically set
to a prime close to the number of available hosts.  Ignored when
mapred.job.tracker is "local".



mapred.tasktracker.reduce.tasks.maximum
4
The maximum number of reduce tasks that will be run
simultaneously by a task tracker.



dfs.replication
2
Default block replication.
The actual number of replications can be specified when the file is created.
The default is used if replication is not specified in create time.

2.2 Formatting the name node

The first step to starting up your Hadoop installation is formatting the Hadoop filesystem which is implemented on top of the local filesystem of your "cluster" (which includes only your local machine if you followed this tutorial). You need to do this the first time you set up a Hadoop cluster. Do not format a running Hadoop filesystem, this will cause all your data to be erased.

To format the filesystem (which simply initializes the directory specified by the dfs.name.dir variable), run the command

[root@cc hadoop]# bin/hadoop namenode -format

The output suppose to be like this:


[root@cc hadoop]# bin/hadoop namenode -format
08/12/24 10:56:34 INFO dfs.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = cc.d.de.static.xlhost.com/206.222.13.204
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 0.18.2
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.18 -r 709042; compiled by 'ndaley' on Thu Oct 30 01:07:18 UTC 2008
************************************************************/
Re-format filesystem in /usr/local/bin/hadoop/datastore/hadoop-root/dfs/name ? (Y or N) Y
08/12/24 10:57:40 INFO fs.FSNamesystem: fsOwner=root,root,bin,daemon,sys,adm,disk,wheel
08/12/24 10:57:40 INFO fs.FSNamesystem: supergroup=supergroup
08/12/24 10:57:40 INFO fs.FSNamesystem: isPermissionEnabled=true
08/12/24 10:57:40 INFO dfs.Storage: Image file of size 78 saved in 0 seconds.
08/12/24 10:57:40 INFO dfs.Storage: Storage directory /usr/local/bin/hadoop/datastore/hadoop-root/dfs/name has been successfully formatted.
08/12/24 10:57:40 INFO dfs.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at cc.d.de.static.xlhost.com/206.222.13.204
************************************************************/

2.3 Starting/Stopping your single-node cluster

Run the command:


[root@cc hadoop]# $HADOOP_HOME/bin/start-all.sh

You suppose to see the following output:


starting namenode, logging to /usr/local/bin/hadoop/bin/../logs/hadoop-root-namenode-cc.com.out
localhost: starting datanode, logging to /usr/local/bin/hadoop/bin/../logs/hadoop-root-datanode-cc.out
localhost: starting secondarynamenode, logging to /usr/local/bin/hadoop/bin/../logs/hadoop-root-secondarynamenode-cc.out
starting jobtracker, logging to /usr/local/bin/hadoop/bin/../logs/hadoop-root-jobtracker-cc.com.out
localhost: starting tasktracker, logging to /usr/local/bin/hadoop/bin/../logs/hadoop-root-tasktracker-cc.com.out

Run example map-reduce job that comes with hadoop installation:


[root@38 /usr/local/bin/hadoop]bin/hadoop dfs -copyFromLocal LICENSE.txt testWordCount
[root@38 /usr/local/bin/hadoop]bin/hadoop dfs -ls
Found 1 items
-rw-r--r--   1 root supergroup      11358 2008-12-25 04:54 /user/root/testWordCount
[root@38 /usr/local/bin/hadoop]bin/hadoop jar hadoop-0.18.2-examples.jar wordcount testWordCount testWordCount-output
08/12/25 04:55:47 INFO mapred.FileInputFormat: Total input paths to process : 1
08/12/25 04:55:47 INFO mapred.FileInputFormat: Total input paths to process : 1
08/12/25 04:55:48 INFO mapred.JobClient: Running job: job_200812250447_0001
08/12/25 04:55:49 INFO mapred.JobClient:  map 0% reduce 0%
08/12/25 04:55:51 INFO mapred.JobClient:  map 100% reduce 0%
08/12/25 04:55:56 INFO mapred.JobClient: Job complete: job_200812250447_0001
08/12/25 04:55:56 INFO mapred.JobClient: Counters: 16
08/12/25 04:55:56 INFO mapred.JobClient:   Job Counters
08/12/25 04:55:56 INFO mapred.JobClient:     Data-local map tasks=2
08/12/25 04:55:56 INFO mapred.JobClient:     Launched reduce tasks=1
08/12/25 04:55:56 INFO mapred.JobClient:     Launched map tasks=2
08/12/25 04:55:56 INFO mapred.JobClient:   Map-Reduce Framework
08/12/25 04:55:56 INFO mapred.JobClient:     Map output records=1581
08/12/25 04:55:56 INFO mapred.JobClient:     Reduce input records=593
08/12/25 04:55:56 INFO mapred.JobClient:     Map output bytes=16546
08/12/25 04:55:56 INFO mapred.JobClient:     Map input records=202
08/12/25 04:55:56 INFO mapred.JobClient:     Combine output records=1292
08/12/25 04:55:56 INFO mapred.JobClient:     Map input bytes=11358
08/12/25 04:55:56 INFO mapred.JobClient:     Combine input records=2280
08/12/25 04:55:56 INFO mapred.JobClient:     Reduce input groups=593
08/12/25 04:55:56 INFO mapred.JobClient:     Reduce output records=593
08/12/25 04:55:56 INFO mapred.JobClient:   File Systems
08/12/25 04:55:56 INFO mapred.JobClient:     HDFS bytes written=6117
08/12/25 04:55:56 INFO mapred.JobClient:     Local bytes written=18568
08/12/25 04:55:56 INFO mapred.JobClient:     HDFS bytes read=13872
08/12/25 04:55:56 INFO mapred.JobClient:     Local bytes read=8542
[root@38 /usr/local/bin/hadoop]bin/hadoop dfs -ls testWordCount-output
Found 2 items
drwxr-xr-x   - root supergroup          0 2008-12-25 04:55 /user/root/testWordCount-output/_logs
-rw-r--r--   1 root supergroup       6117 2008-12-25 04:55 /user/root/testWordCount-output/part-00000
[root@38 /usr/local/bin/hadoop]bin/hadoop dfs -cat testWordCount-output/part-00000
// suppose to see something like this
...
tracking        1
trade   1
trademark,      1
trademarks,     1
transfer        1
transformation  1
translation     1
...

To stop Hadoop cluster run the following:


[root@37 /usr/local/bin/hadoop]bin/stop-all.sh
no jobtracker to stop
localhost: no tasktracker to stop
no namenode to stop
localhost: no datanode to stop
localhost: no secondarynamenode to stop

2.4 Hadoop monitoring and debugging

Please see hadoop tips of how to debug Map-Reduce programs. Worth to mention that hadoop logs are providing the most information
from $HADOOP_HOME/logs or there are links from hadoop web interfaces.

Hadoop comes with several web interfaces which are by default (see conf/hadoop-default.xml) available at these locations:

The job tracker web UI provides information about general job statistics of the Hadoop cluster, running/completed/failed jobs and a job history log file

http://localhost:50030/ - web UI for MapReduce job tracker(s)

The task tracker web UI shows you running and non-running tasks.

http://localhost:50060/ - web UI for task tracker(s)

The name node web UI shows you a cluster summary including information about total/remaining capacity, live and dead nodes. Additionally, it allows you to browse the HDFS namespace and view the contents of its files in the web browser.

http://localhost:50070/ - web UI for HDFS name node(s)

If everything work fine you suppose to see the following output after running jps utility:


[root@cc hadoop]# jps
3060 SecondaryNameNode
3136 JobTracker
2814 NameNode
3270 TaskTracker
3458 Jps

[root@38 /usr/local/bin/hadoop]netstat -plten | grep java
tcp        0      0 :::50020                    :::*                        LISTEN      0          1248347094 5131/java
tcp        0      0 ::ffff:127.0.0.1:54310      :::*                        LISTEN      0          1248346675 5032/java
tcp        0      0 ::ffff:127.0.0.1:54311      :::*                        LISTEN      0          1248347128 5351/java
tcp        0      0 :::50090                    :::*                        LISTEN      0          1248347119 5272/java
tcp        0      0 :::50060                    :::*                        LISTEN      0          1248347327 5467/java
tcp        0      0 :::50030                    :::*                        LISTEN      0          1248347293 5351/java
tcp        0      0 :::48498                    :::*                        LISTEN      0          1248347011 5272/java
tcp        0      0 :::50070                    :::*                        LISTEN      0          1248346888 5032/java
tcp        0      0 :::53210                    :::*                        LISTEN      0          1248347103 5351/java
tcp        0      0 :::50010                    :::*                        LISTEN      0          1248347000 5131/java
tcp        0      0 :::50075                    :::*                        LISTEN      0          1248347020 5131/java
tcp        0      0 :::40315                    :::*                        LISTEN      0          1248346669 5032/java
tcp        0      0 ::ffff:127.0.0.1:47198      :::*                        LISTEN      0          1248347345 5467/java
tcp        0      0 :::56575                    :::*                        LISTEN      0          1248346834 5131/java

3.HBase

3.1 Configuration

- Unpack HBase archive to /usr/local/bin
- Move hbase.18.1 to hbase
- Define HBASE_HOME point to /usr/local/bin/hbase( don't forget to edit ~/.bash_profile )
- Define JAVA_HOME in $HBASE_HOME/conf/hbase-env.sh


export HADOOP_CLASSPATH=/usr/local/bin/hbase/conf:/usr/local/bin/hbase/hbase-0.18.1.jar:/usr/local/bin/hbase/hbase-0.18.1-test.jar

- Edit $HADOOP_HOME/conf/hadoop-env.sh, add this(for instance):


export JAVA_HOME=/usr/local/bin/jdk1.5.0_14/

3.2 Pseudo-Distributed Operation

A pseudo-distributed operation is simply a distributed operation run on a single host. Once you have confirmed your DFS setup, configuring HBase for use on one host requires modification of ${HBASE_HOME}/conf/hbase-site.xml, which needs to be pointed at the running Hadoop DFS instance. Use hbase-site.xml to override the properties defined in ${HBASE_HOME}/conf/hbase-default.xml (hbase-default.xml itself should never be modified). At a minimum the hbase.rootdir property should be redefined in hbase-site.xml to point HBase at the Hadoop filesystem to use. For example, adding the property below to your hbase-site.xml says that HBase should use the /hbase directory in the HDFS whose namenode is at port 54310 on your local machine:




hbase.rootdir
hdfs://localhost:54310/hbase
The directory shared by region servers.

3.3 Example API Usage

Once you have a running HBase, you probably want a way to hook your application up to it.If your application is in Java, then you should use the Java API. The following example takes as input excel formatted file and name of already existed table in HTable, process records and writes them to HBase. You could look at client example here


package org.examples;

import java.io.IOException;
import java.util.Iterator;
import java.util.regex.Pattern;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hbase.io.BatchUpdate;
import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
import org.apache.hadoop.hbase.mapred.TableReduce;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.WritableComparator;
import org.apache.hadoop.io.WritableUtils;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.JobClient;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.Mapper;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;
import org.apache.hadoop.mapred.TextInputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;

/**
* Sample uploader Map-Reduce example class. Takes excel format file as input and write output to HBase
*
*/

public class SampleUploader extends MapReduceBase implements Mapper, Tool
{
static enum Counters { MAP_LINES,REDUCE_LINES }
private static final String NAME = "SampleUploader";
private Configuration conf;
static final String OUTPUT_COLUMN = "value:";
static final String OUTPUT_KEY = "key:";
long numRecords;
private Text idText = new Text();
private Text recordText = new Text();
private String inputFile;

/** A WritableComparator optimized for Text keys. */
public static class Comparator extends WritableComparator
{
public Comparator()
{
super(Text.class);
}

public int compare(byte[] b1, int s1, int l1,
byte[] b2, int s2, int l2)
{
int n1 = WritableUtils.decodeVIntSize(b1[s1]);
int n2 = WritableUtils.decodeVIntSize(b2[s2]);
return compareBytes(b1, s1+n1, l1-n1, b2, s2+n2, l2-n2);
}
}


public JobConf createSubmittableJob(String[] args)
{
JobConf c = new JobConf(getConf(), SampleUploader.class);
c.setJobName(NAME);
c.setInputFormat(TextInputFormat.class);
FileInputFormat.setInputPaths(c, new Path(args[0]));
//c.setInputPaths(new Path(args[0]));
c.setMapperClass(this.getClass());
c.setMapOutputKeyClass(Text.class);
c.setMapOutputValueClass(Text.class);
c.setReducerClass(TableUploader.class);
TableReduce.initJob(args[1], TableUploader.class, c);
return c;
}
public void configure(JobConf job)
{
inputFile = job.get("map.input.file");
}
public void map(LongWritable k, Text v,OutputCollector output, Reporter r) throws IOException
{

String lineWithoutURLs = Pattern.compile("\"[^\"]* "").matcher(v.toString()).replaceAll("");
String userID = lineWithoutURLs.substring(22, 54);

r.incrCounter(Counters.MAP_LINES, 1);
if ((++numRecords % 10000) == 0)
{
System.out.println("Finished mapping of " + numRecords + " records " + "from the input file: " + inputFile);
}
idText.set(userID);
recordText.set(lineWithoutURLs);
output.collect( idText,recordText );
}

public static class TableUploader  extends TableReduce
{

@Override
public void reduce( Text k, Iterator v,
OutputCollector output,
Reporter r) throws IOException
{

BatchUpdate outval = new BatchUpdate(k.toString());
while (v.hasNext())
{
String value = v.next().toString();
String dateStamp = value.substring(0, 20).replaceAll("[-:,]", "");
outval.put( OUTPUT_COLUMN + dateStamp, value.getBytes());
r.incrCounter(Counters.REDUCE_LINES, 1);
}
output.collect( new ImmutableBytesWritable( k.getBytes()), outval);
}
}



static int printUsage()
{
System.out.println(NAME + "  ");
return -1;
}

public int run( String[] args) throws Exception
{
// Make sure there are exactly 2 parameters left.
if (args.length != 2) {
System.out.println("ERROR: Wrong number of parameters: " +
args.length + " instead of 2.");
return printUsage();
}
JobClient.runJob(createSubmittableJob(args));
return 0;
}

public Configuration getConf() {
return this.conf;
}

public void setConf(final Configuration c) {
this.conf = c;
}

public static void main(String[] args) throws Exception
{
int errCode = ToolRunner.run(new Configuration(), new SampleUploader(),
args);
System.exit(errCode);

}



}

3.4 Running HBase and using HBase shell

Start HBase with the following command:


${HBASE_HOME}/bin/start-hbase.sh

If HBase is started succesfully you suppose to see a following output after running jps:


[root@37 /usr/local/bin/hadoop]jps
10379 DataNode
18303 Jps
10637 TaskTracker
10536 JobTracker
10286 NameNode
17512 HMaster

Check if HBase is running with web interface: http://localhost:60030

Once HBase has started, run HBase Shell with

${HBASE_HOME}/bin/hbase shell

Create a sample table for our tests:


create 'table_keyInMemory', {NAME => 'key',IN_MEMORY => true, VERSIONS => 1,BLOCKCACHE => true},{NAME => 'value',VERSIONS => 1}

To stop HBase, exit the HBase shell and enter:


${HBASE_HOME}/bin/stop-hbase.sh

3.5 Finally: Rinning Map-Reduce jobs:

First copy an input file from local filesystem:


[root@37 /usr/local/bin/hadoop]bin/hadoop dfs -copyFromLocal /localFile fileNameInHadoopDFS
[root@37 /usr/local/bin/hadoop] bin/hadoop dfs -ls
Found 1 items -rw-r--r-- 1 root supergroup 27325 2008-12-25 03:26 /user/root fileNameInHadoopDFS
// Run Map-Reduce script
[root@37 /usr/local/bin/hadoop]bin/hadoop jar Test.jar org.exelate.Uploader 100 sampleTable
08/12/25 03:55:40 INFO mapred.FileInputFormat: Total input paths to process : 1
08/12/25 03:55:40 INFO mapred.FileInputFormat: Total input paths to process : 1
08/12/25 03:55:40 INFO mapred.JobClient: Running job: job_200812250352_0002
08/12/25 03:55:41 INFO mapred.JobClient:  map 0% reduce 0%
08/12/25 03:55:44 INFO mapred.JobClient:  map 100% reduce 0%
08/12/25 03:55:50 INFO mapred.JobClient: Job complete: job_200812250352_0002
08/12/25 03:55:50 INFO mapred.JobClient: Counters: 17
08/12/25 03:55:50 INFO mapred.JobClient:   Job Counters
08/12/25 03:55:50 INFO mapred.JobClient:     Data-local map tasks=2
08/12/25 03:55:50 INFO mapred.JobClient:     Launched reduce tasks=1
08/12/25 03:55:50 INFO mapred.JobClient:     Launched map tasks=2
08/12/25 03:55:50 INFO mapred.JobClient:   Map-Reduce Framework
08/12/25 03:55:50 INFO mapred.JobClient:     Map output records=100
08/12/25 03:55:50 INFO mapred.JobClient:     Reduce input records=100
08/12/25 03:55:50 INFO mapred.JobClient:     Map output bytes=16516
08/12/25 03:55:50 INFO mapred.JobClient:     Map input records=100
08/12/25 03:55:50 INFO mapred.JobClient:     Combine output records=0
08/12/25 03:55:50 INFO mapred.JobClient:     Map input bytes=27325
08/12/25 03:55:50 INFO mapred.JobClient:     Combine input records=0
08/12/25 03:55:50 INFO mapred.JobClient:     Reduce input groups=44
08/12/25 03:55:50 INFO mapred.JobClient:     Reduce output records=44
08/12/25 03:55:50 INFO mapred.JobClient:   File Systems
08/12/25 03:55:50 INFO mapred.JobClient:     Local bytes written=33928
08/12/25 03:55:50 INFO mapred.JobClient:     HDFS bytes read=30048
08/12/25 03:55:50 INFO mapred.JobClient:     Local bytes read=16921
08/12/25 03:55:50 INFO mapred.JobClient:   org.exelate.SampleUploader$Counters
08/12/25 03:55:50 INFO mapred.JobClient:     MAP_LINES=100
08/12/25 03:55:50 INFO mapred.JobClient:     REDUCE_LINES=100

HDFS Command Reference

There are many more commands in bin/hadoop dfs than were demonstrated here, although these basic operations will get you started. Running bin/hadoop dfs with no additional arguments will list all commands which can be run with the FsShell system. Furthermore, bin/hadoop dfs -help commandName will display a short usage summary for the operation in question, if you are stuck.

A table of all operations is reproduced below. The following conventions are used for parameters:

italics denote variables to be filled out by the user.

"path" means any file or directory name.

"path..." means one or more file or directory names.

"file" means any filename.
"src" and "dest" are path names in a directed operation.

"localSrc" and "localDest" are paths as above, but on the local file system. All other file and path names refer to objects inside HDFS.

Parameters in [brackets] are optional.

Command	Operation
-ls path	Lists the contents of the directory specified by path, showing the names, permissions, owner, size and modification date for each entry.
-lsr path	Behaves like `-ls`, but recursively displays entries in all subdirectories of path.
-du path	Shows disk usage, in bytes, for all files which match path; filenames are reported with the full HDFS protocol prefix.
-dus path	Like `-du`, but prints a summary of disk usage of all files/directories in the path.
-mv src dest	Moves the file or directory indicated by src to dest, within HDFS.
-cp src dest	Copies the file or directory identified by src to dest, within HDFS.
-rm path	Removes the file or empty directory identified by path.
-rmr path	Removes the file or directory identified by path. Recursively deletes any child entries (i.e., files or subdirectories of path).
-put localSrc dest	Copies the file or directory from the local file system identified by localSrc to dest within the DFS.
-copyFromLocal localSrc dest	Identical to `-put`
-moveFromLocal localSrc dest	Copies the file or directory from the local file system identified by localSrc to dest within HDFS, then deletes the local copy on success.
-get [-crc] src localDest	Copies the file or directory in HDFS identified by src to the local file system path identified by localDest.
-getmerge src localDest [addnl]	Retrieves all files that match the path src in HDFS, and copies them to a single, merged file in the local file system identified by localDest.
-cat filename	Displays the contents of filename on stdout.
-copyToLocal [-crc] src localDest	Identical to `-get`
-moveToLocal [-crc] src localDest	Works like `-get`, but deletes the HDFS copy on success.
-mkdir path	Creates a directory named path in HDFS. Creates any parent directories in path that are missing (e.g., like `mkdir -p` in Linux).
-setrep [-R] [-w] rep path	Sets the target replication factor for files identified by path to rep. (The actual replication factor will move toward the target over time)
-touchz path	Creates a file at path containing the current time as a timestamp. Fails if a file already exists at path, unless the file is already size 0.
-test -[ezd] path	Returns 1 if path exists; has zero length; or is a directory, or 0 otherwise.
-stat [format] path	Prints information about path. format is a string which accepts file size in blocks (%b), filename (%n), block size (%o), replication (%r), and modification date (%y, %Y).
-tail [-f] file	Shows the lats 1KB of file on stdout.
-chmod [-R] mode,mode,... path...	Changes the file permissions associated with one or more objects identified by path.... Performs changes recursively with `-R`. mode is a 3-digit octal mode, or `{augo}+/-{rwxX}`. Assumes `a` if no scope is specified and does not apply a umask.
-chown [-R] [owner][:[group]] path...	Sets the owning user and/or group for files or directories identified by path.... Sets owner recursively if `-R` is specified.
-chgrp [-R] group path...	Sets the owning group for files or directories identified by path.... Sets group recursively if `-R` is specified.
-help cmd	Returns usage information for one of the commands listed above. You must omit the leading '-' character in cmd

Change hostname in Linux

2008-12-23T03:27:00.000-08:00

Change hostname in Linux

First you need to find out your hostname, you can do this with

$ hostname
localhost.localdomain
$

Edit /etc/hosts

You need to edit /etc/hosts and add a line for your host name

$ cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1               localhost.localdomain localhost
$

My new server IP is 72.232.196.90, i need to assign it hostname server12.hosthat.com, to do this, i have edited /etc/hosts as follows.

# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1               localhost.localdomain localhost
72.232.196.90           server12.hosthat.com server12

Edit /etc/sysconfig/network

First lets see what is in the file

$ cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=localhost.localdomain
$

To change servers hostname to server12.hosthat.com, change the file as follows.

$ cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=server12.hosthat.com
$

Now you need to reboot the server to change the hostname.

vmstat

2008-12-23T03:21:00.000-08:00

vmstat

Report virtual memory statistics

SYNOPSIS

vmstat [-n] [delay [ count]]
vmstat[-V]

vmstat reports information about processes, memory, paging, block IO, traps, and cpu activity.

The first report produced gives averages since the last reboot. Additional reports give information on a sampling period of length delay. The process and memory reports are instantaneous in either case.


[root@]# vmstat 1
   procs                      memory      swap          io     system      cpu
 r  b  w   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id
 0  0  0 6085936  91884 1707572 1333604   48   50    70    73   46     7  1  1 23
 0  0  0 6085936  88464 1707608 1335120    0    0  1374    49 2750  1784  0  0 100
 0  0  0 6085936  88048 1707660 1335952    0    0   753   913 2532  1883  0  0 100
 0  0  0 6085936  87632 1707824 1337460    0    0  1282   908 2452  2054  0  0 100

Use Public/Private Keys for Authentication

2008-12-15T23:34:00.000-08:00

First, create a public/private key pair on the client that you will use to connect to the server (you will need to do this from each client machine from which you connect):

$ ssh-keygen -t rsa

This will create two files in your ~/.ssh directory 
id_rsa: your private key
id_rsa.pub: is your public key.

If you don't want to still be asked for a password each time you connect,
just press enter when asked for a password when creating the key pair.
It is up to you to decide whether or not you should password encrypt
your key when you create it. If you don't password encrypt your key,
then anyone gaining access to your local machine will automatically
have ssh access to the remote server. Also, root on the local machine
has access to your keys although one assumes that if you can't trust
root (or root is compromised) then you're in real trouble. Encrypting
the key adds additional security at the expense of eliminating the need
for entering a password for the ssh server only to be replaced with
entering a password for the use of the key. Now set permissions on your private key:

$ chmod 700 ~/.ssh
$ chmod 600 ~/.ssh/id_rsa

Copy the public key (id_rsa.pub) to the server and install it to the authorized_keys list:

$ cat id_rsa.pub >> ~/.ssh/authorized_keys
$ chmod 600 ~/.ssh/authorized_keys

Once you've checked you can successfully login to the server using your public/private key pair,
you can disable password authentication completely by adding the following setting to your
/etc/ssh/sshd_config

# Disable password authentication forcing use of keys
PasswordAuthentication no