Docker + HELM + Kubernetes

Kafka benchmarking

bin/kafka-run-class.sh org.apache.kafka.tools.ProducerPerformance --topic=kafka_benchmark --num-records=10000 --throughput=10 --record-size=200 --producer-props bootstrap.servers=localhost:9092 buffer.memory=67108864 batch.size=6

bin/kafka-consumer-perf-test.sh --zookeeper localhost:2181 --messages 50000000 --topic kafka_benchmark --threads 1

bin/kafka-consumer-perf-test.sh --messages 100000000 -topic network_perf --threads 3 --broker-list dl6-l2-kafka-01:9092,dl6-l2-kafka-02:9092,dl6-l2-kafka-03:9092 -show-detailed-stats

https://gist.github.com/jkreps/c7ddb4041ef62a900e6c

GIT statistics

Show number of commits by developer

git shortlog | grep -E '^[^ ]' | sort -u | wc -l

Show number of commits for a period

git log --pretty=oneline --after=12/31/2016 | wc -l

Useful links

Awesome GO

https://github.com/fleveque/awesome-awesomes#go

Awesome Hadoop

https://github.com/youngwookim/awesome-hadoop

Awesome Machine Learning

https://github.com/josephmisiti/awesome-machine-learning

Awesome Node.js

https://github.com/sindresorhus/awesome-nodejs

Awesome awesomes

https://github.com/fleveque/awesome-awesomes

Numbers Everyone Should Know

From Google Pro Tips: Numbers Everyone Should Know

To evaluate design alternatives you first need a good sense of how long typical operations will take. Dr. Dean gives this list:

• L1 cache reference 0.5 ns
• Branch mispredict 5 ns
• L2 cache reference 7 ns
• Mutex lock/unlock 100 ns
• Main memory reference 100 ns
• Compress 1K bytes with Zippy 10,000 ns
• Send 2K bytes over 1 Gbps network 20,000 ns
• Read 1 MB sequentially from memory 250,000 ns
• Round trip within same datacenter 500,000 ns
• Disk seek 10,000,000 ns
• Read 1 MB sequentially from network 10,000,000 ns
• Read 1 MB sequentially from disk 30,000,000 ns
• Send packet CA->Netherlands->CA 150,000,000 ns 

Some things to notice:

• Notice the magnitude differences in the performance of different options.
• Datacenters are far away so it takes a long time to send anything between them.
• Memory is fast and disks are slow.
• By using a cheap compression algorithm a lot (by a factor of 2) of network bandwidth can be saved.
• Writes are 40 times more expensive than reads.
• Global shared data is expensive. This is a fundamental limitation of distributed systems. The lock contention in shared heavily written objects kills performance as transactions become serialized and slow.
• Architect for scaling writes.
• Optimize for low write contention.
• Optimize wide. Make writes as parallel as you can.

Example: Generate Image Results Page Of 30 Thumbnails

The is the example given in the video. Two design alternatives are used as design thought experiments.

Design 1 - Serial 

• Read images serially. Do a disk seek. Read a 256K image and then go on to the next image.
• Performance: 30 seeks * 10 ms/seek + 30 * 256K / 30 MB /s = 560ms

Design 2 - Parallel 

• Issue reads in parallel.
• Performance: 10 ms/seek + 256K read / 30 MB/s = 18ms
• There will be variance from the disk reads, so the more likely time is 30-60ms

Instagram - architecture that worth now 1B

• Amazon shop. They use many of Amazon's services. With only 3 engineers so don’t have the time to look at self hosting.
• 100+ EC2 instances total for various purposes.
• Ubuntu Linux 11.04 (“Natty Narwhal”). Solid, other Ubuntu versions froze on them.
• Amazon’s Elastic Load Balancer routes requests and 3 nginx instances sit behind the ELB.
• SSL terminates at the ELB, which lessens the CPU load on nginx.
• Amazon’s Route53 for the DNS.
• 25+ Django application servers on High-CPU Extra-Large machines.
• Traffic is CPU-bound rather than memory-bound, so High-CPU Extra-Large machines are a good balance of memory and CPU.
• Gunicorn as their WSGI server. Apache harder to configure and more CPU intensive.
• Fabric is used to execute commands in parallel on all machines. A deploy takes only seconds.
• PostgreSQL (users, photo metadata, tags, etc) runs on 12 Quadruple Extra-Large memory instances.
• Twelve PostgreSQL replicas run in a different availability zone.
• PostgreSQL instances run in a master-replica setup using Streaming Replication. EBS is used for snapshotting, to take frequent backups.
• EBS is deployed in a software RAID configuration. Uses mdadm to get decent IO.
• All of their working set is stored memory. EBS doesn’t support enough disk seeks per second.
• Vmtouch (portable file system cache diagnostics) is used to manage what data is in memory, especially when failing over from one machine to another, where there is no active memory profile already.
• XFS as the file system. Used to get consistent snapshots by freezing and unfreezing the RAID arrays when snapshotting.
• Pgbouncer is used pool connections to PostgreSQL.
• Several terabytes of photos are stored on Amazon S3.
• Amazon CloudFront as the CDN.
• Redis powers their main feed, activity feed, sessions system, and other services.
• Redis runs on several Quadruple Extra-Large Memory instances. Occasionally shard across instances.
• Redis runs in a master-replica setup. Replicas constantly save to disk. EBS snapshots backup the DB dumps. Dumping on the DB on the master was too taxing.
• Apache Solr powers the geo-search API. Like the simple JSON interface.
• 6 memcached instances for caching. Connect using pylibmc & libmemcached. Amazon Elastic Cache service isn't any cheaper.
• Gearman is used to: asynchronously share photos to Twitter, Facebook, etc; notifying real-time subscribers of a new photo posted; feed fan-out.
• 200 Python workers consume tasks off the Gearman task queue.
• Pyapns (Apple Push Notification Service) handles over a billion push notifications. Rock solid.
• Munin to graph metrics across the system and alert on problems. Write many custom plugins using Python-Munin to graph, signups per minute, photos posted per second, etc.
• Pingdom for external monitoring of the service.
• PagerDuty for handling notifications and incidents.
• Sentry for Python error reporting.

RAMFS vs TMPFS on Linux

Using ramfs or tmpfs you can allocate part of the physical memory to be used as a partition. You can mount this partition and start writing and reading files like a hard disk partition. Since you’ll be reading and writing to the RAM, it will be faster.

When a vital process becomes drastically slow because of disk writes, you can choose either ramfs or tmpfs file systems for writing files to the RAM.


Both tmpfs and ramfs mount will give you the power of fast reading and writing files from and to the primary memory. When you test this on a small file, you may not see a huge difference. You’ll notice the difference only when you write large amount of data to a file with some other processing overhead such as network.

1. How to mount Tmpfs

# mkdir -p /mnt/tmp  # mount -t tmpfs -o size=20m tmpfs /mnt/tmp

The last line in the following df -k shows the above mounted /mnt/tmp tmpfs file system.

# df -k Filesystem      1K-blocks  Used     Available Use%  Mounted on /dev/sda2       32705400   5002488  26041576  17%   / /dev/sda1       194442     18567    165836    11%   /boot tmpfs           517320     0        517320    0%    /dev/shm tmpfs           20480      0        20480     0%    /mnt/tmp

2. How to mount Ramfs

# mkdir -p /mnt/ram  # mount -t ramfs -o size=20m ramfs /mnt/ram

The last line in the following mount command shows the above mounted /mnt/ram ramfs file system.

# mount /dev/sda2 on / type ext3 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) /dev/sda1 on /boot type ext3 (rw) tmpfs on /dev/shm type tmpfs (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) fusectl on /sys/fs/fuse/connections type fusectl (rw) tmpfs on /mnt/tmp type tmpfs (rw,size=20m) ramfs on /mnt/ram type ramfs (rw,size=20m)

You can mount ramfs and tmpfs during boot time by adding an entry to the /etc/fstab.

3. Ramfs vs Tmpfs

Primarily both ramfs and tmpfs does the same thing with few minor differences.

• Ramfs will grow dynamically. So, you need control the process that writes the data to make sure ramfs doesn’t go above the available RAM size in the system. Let us say you have 2GB of RAM on your system and created a 1 GB ramfs and mounted as /tmp/ram. When the total size of the /tmp/ram crosses 1GB, you can still write data to it. System will not stop you from writing data more than 1GB. However, when it goes above total RAM size of 2GB, the system may hang, as there is no place in the RAM to keep the data.
• Tmpfs will not grow dynamically. It would not allow you to write more than the size you’ve specified while mounting the tmpfs. So, you don’t need to worry about controlling the process that writes the data to make sure tmpfs doesn’t go above the specified limit. It may give errors similar to “No space left on device”.
• Tmpfs uses swap.
• Ramfs does not use swap.

4. Disadvantages of Ramfs and Tmpfs

Since both ramfs and tmpfs is writing to the system RAM, it would get deleted once the system gets rebooted, or crashed. So, you should write a process to pick up the data from ramfs/tmpfs to disk in periodic intervals. You can also write a process to write down the data from ramfs/tmpfs to disk while the system is shutting down. But, this will not help you in the time of system crash.

Table: Comparison of ramfs and tmpfs

Experimentation	Tmpfs	Ramfs
Fill maximum space and continue writing	Will display error	Will continue writing
Fixed Size	Yes	No
Uses Swap	Yes	No
Volatile Storage	Yes	Yes

If you want your process to write faster, opting for tmpfs is a better choice with precautions about the system crash.

Realtime Apache Hadoop at Facebook

1. Facebook use Hadoop/Hbase for a following products Titan( Facebook messages), Puma( real-time publisher insights), ODS( Facebook internal metrics).


2. Currently in use 4 data centers, about 2000 nodes.


3. Data center replication is not supported as part of HBase, so they wrote some internal product to do that.


4. All data LZO compressed.


5. Each data center has 2PB of data replicated x3, total 6PB.


6. Each node have 12 discs each 1Tb.


7. Load on each node 55% read, 45% write.


8. To collect data from front end servers they're using Scribe, which is open sourced.

Slides here.

WHY HADOOP AND HBASE( from facebook paper)

The requirements for the storage system for our workloads can be summarized as follows:

1. Elasticity: We need to be able to add incremental capacity to our storage systems with minimal overhead and no downtime. In some cases we may want to add capacity rapidly and the system should automatically balance load and utilization across new hardware.

2. High write throughput: Most of the applications store (and optionally index) tremendous amounts of data and require high aggregate write throughput.

3. Efficient and low-latency strong consistency semantics within a data center: There are important applications like Messages that require strong consistency within a data center. This requirement often arises directly from user expectations. For example ‘‘unread’’ message counts displayed on the home page and the messages shown in the inbox page view should be consistent with respect to each other. While a globally distributed strongly consistent system is practically impossible, a system that could at least provide strong consistency within a data center would make it possible to provide a good user experience. We also knew that (unlike other Facebook applications), Messages was easy to federate so that a particular user could be served entirely out of a single data center making strong consistency within a single data center a critical requirement for the Messages project. Similarly, other projects, like realtime log aggregation, may be deployed entirely within one data center and are much easier to program if the system provides strong consistency guarantees.

4. Efficient random reads from disk: In spite of the widespread use of application level caches (whether embedded or via memcached), at Facebook scale, a lot of accesses miss the cache and hit the back-end storage system. MySQL is very efficient at performing random reads from disk and any new system would have to be comparable.

5. High Availability and Disaster Recovery: We need to provide a service with very high uptime to users that covers both planned and unplanned events (examples of the former being events like software upgrades and addition of hardware/capacity and the latter exemplified by failures of hardware components). We also need to be able to tolerate the loss of a data center with minimal data loss and be able to serve data out of another data center in a reasonable time frame.

6. Fault Isolation: Our long experience running large farms of MySQL databases has shown us that fault isolation is critical. Individual databases can and do go down, but only a small fraction of users are affected by any such event. Similarly, in our warehouse usage of Hadoop, individual disk failures affect only a small part of the data and the system quickly recovers from such faults.

7. Atomic read-modify-write primitives: Atomic increments and compare-and-swap APIs have been very useful in building lockless concurrent applications and are a must have from the underlying storage system.

8. Range Scans: Several applications require efficient retrieval of a set of rows in a particular range. For example all the last 100 messages for a given user or the hourly impression counts over the last 24 hours for a given advertiser.

It is also worth pointing out non-requirements:

1. Tolerance of network partitions within a single data center: Different system components are often inherently centralized. For example, MySQL servers may all be located within a few racks, and network partitions within a data center would cause major loss in serving capabilities therein. Hence every effort is made to eliminate the possibility of such events at the hardware level by having a highly redundant network design.

2. Zero Downtime in case of individual data center failure: In our experience such failures are very rare, though not impossible. In a less than ideal world where the choice of system design boils down to the choice of compromises that are acceptable, this is one compromise that we are willing to make given the low occurrence rate of such events. We might revise this non-requirement at a later time.

3. Active-active serving capability across different data centers: As mentioned before, we were comfortable making the assumption that user data could be federated across different data centers (based ideally on user locality). Latency (when user and data locality did not match up) could be masked by using an application cache close to the user.

Some less tangible factors were also at work. Systems with existing production experience for Facebook and in-house expertise were greatly preferred. When considering open-source projects, the strength of the community was an important factor. Given the level of engineering investment in building and maintaining systems like these –– it also made sense to choose a solution that was broadly applicable (rather than adopt point solutions based on differing architecture and codebases for each workload).

After considerable research and experimentation, we chose Hadoop and HBase as the foundational storage technology for these next generation applications. The decision was based on the state of HBase at the point of evaluation as well as our confidence in addressing the features that were lacking at that point via in- house engineering. HBase already provided a highly consistent, high write-throughput key-value store. The HDFS NameNode stood out as a central point of failure, but we were confident that our HDFS team could build a highly-available NameNode (AvatarNode) in a reasonable time-frame, and this would be useful for our warehouse operations as well. Good disk read-efficiency seemed to be within striking reach (pending adding Bloom filters to HBase’’s version of LSM Trees, making local DataNode reads efficient and caching NameNode metadata). Based on our experience operating the Hive/Hadoop warehouse, we knew HDFS was stellar in tolerating and isolating faults in the disk subsystem. The failure of entire large HBase/HDFS clusters was a scenario that ran against the goal of fault-isolation, but could be considerably mitigated by storing data in smaller HBase clusters. Wide area replication projects, both in-house and within the HBase community, seemed to provide a promising path to achieving disaster recovery.

HBase is massively scalable and delivers fast random writes as well as random and streaming reads. It also provides row-level atomicity guarantees, but no native cross-row transactional support. From a data model perspective, column-orientation gives extreme flexibility in storing data and wide rows allow the creation of billions of indexed values within a single table. HBase is ideal for workloads that are write-intensive, need to maintain a large amount of data, large indices, and maintain the flexibility to scale out quickly.

A Day Made of Glass... Made possible by Corning.

An Introduction to RDF and the Jena RDF API

http://jena.sourceforge.net/tutorial/RDF_API/

XAuth - an introduction from Chris Messina

XAuth - an introduction from Chris Messina on Vimeo.

CentOS: find out what network ports are in use

To get a quick idea of what ports you have open on your local box, you can use nmap.

[root@api1dev /usr/local/bin]nmap localhost
Starting Nmap 5.00 ( http://nmap.org ) at 2010-05-10 03:02 EDT
Interesting ports on vmlinux-testing1 (127.0.0.1):
Not shown: 991 closed ports
PORT     STATE SERVICE
21/tcp   open  ftp
22/tcp   open  ssh
80/tcp   open  http
111/tcp  open  rpcbind
389/tcp  open  ldap
631/tcp  open  ipp
726/tcp  open  unknown
8009/tcp open  ajp13
8080/tcp open  http-proxy

For more detailed information, try netstat:

netstat -an

You’ll get a breakdown of every socket open on your machine – useful for figuring out who’s connected and from where.

Interesting articles about Ad-Exchange and RTB

The Ad-Exchange Model (Part I)
RTB Part I Followup
RTB Part II: Supply supply supply!
RTB Part III: Cookies & User Data!

Alternative to HBase + Map-Reduce

mongoDB

Combining the best features of document databases, key-value stores, and RDBMSes.

MongoDB (from "humongous") is a scalable, high-performance, open source, schema-free, document-oriented database. Written in C++, MongoDB features:

• Document-oriented storage (the simplicity and power of JSON-like data schemas)
• Dynamic queries
• Full index support, including secondary indexes, inner-objects, embedded arrays, geospatial
• Query profiling
• Fast, in-place updates
• Efficient storage of binary data large objects (e.g. photos and videos)
• Replication and fail-over support
• Auto-sharding for cloud-level scalability
• MapReduce for complex aggregation
• Commercial Support, Training, and Consulting

MongoDB bridges the gap between key-value stores (which are fast and highly scalable) and traditional RDBMS systems. Many companies are using MongoDB in production today.

Quick Links

• Downloads
• Follow @mongodb on Twitter
• Use Cases | Philosophy
• Hosting Center
• Drivers
• Source Code
• Blog | Articles
• Tutorial | Try MongoDB in the Browser
• Licensing | Example Snippets | BugDB (Jira)

Support

• Support forums: mongodb-user | more...
• IRC: irc.freenode.net/#mongodb
• Commercial support: 10gen
• Training: 10gen

LAMP on EC2 : Setting up Amazon Web Services

http://goo.gl/z1Zb

http://goo.gl/xhqB

Web performance in seven steps: Summary and Conclusions

Web performance in seven steps: Summary and Conclusions

Previous time I blogged about the last step of the seven steps, step 7: Share the responsibility for the whole chain, a non-technical but rather a communication and behavior thing which I found crucial for success. We now have reached the end of this series and I'll sum up the topics we've dealt with and draw some conclusions.

In this growing on line world with demanding customers it has become essential that services provided on the web are always available and always fast enough. This is often challenging to developers and operators: performance problems manifest themselves in various ways, like in frustration, loss of revenue and disruption of development; and just adding hardware is a doubtful solution.

The question is: how can we as developers and operators assure that our web site is always available and always fast? My answer is: you need the right approach. I present that approach: measure, don’t guess; seven steps to performance success. These seven steps are as follows:

• Step 1: Define performance requirements;
• Step 2: Execute a proof of concept;
• Step 3: Test representatively;
• Step 4: Test continuously;
• Step 5: Monitor and diagnose;
• Step 6: Tune based on evidence;
• Step 7: Share the responsibility for the whole chain.

This approach provides a pro-active way of working which my customers appreciate as valuable. It can actually be leveraged to assure high performance all the time, for virtually any on- and off-line application.

This blog series has been an interesting journey for me. Some time ago we presented our EJAPP Top 10 of performance problems. Now we have added this approach of seven steps to help assure your applications performance.

It has worked for us and our customers. How does this all work for you in practice? We’d like to hear your feedback.

Location based DB schema in mySQL

http://dev.mysql.com/tech-resources/articles/4.1/gis-with-mysql.html

http://dev.mysql.com/doc/refman/4.1/en/spatial-extensions.html

http://howto-use-mysql-spatial-ext.blogspot.com/

http://forge.mysql.com/wiki/GIS_Functions#Introduction

http://forums.mysql.com/read.php?23,40909,43079#msg-43079

OpenInviter

• Easy access to your visitors address book in all major email providers and social networks around the world.


• Completely painless and easy way of integrating in your website. It takes virtually not more than 5 minutes to have your own OpenInviter^TM up and running on your site.


• Constant updates so that you can sit back and relax and always have access to the latest ways to get your visitor's address book.


• WGET-ready! Yes, you read right! OpenInviter^TM is the only contacts importer supporting both WGET and cURL as methods of handling requests (since version 1.2) so now you can use it on ANY server you want without the hassle of installing libcurl!


• Real time access to the service statuses so you can know if there is an email provider that is not working right with OpenInviter^TM.

http://openinviter.com/faq.php

GO: Google Launches Its Own Programming Language

GO: Google Launches Its Own Programming Language

Posted using ShareThis

Upgrading mySQL from 5.0 to 5.1.on CentOS 5

My first attempt was to use yum:

[root@linux /] yum info mysql
Loading "fastestmirror" plugin
Loading mirror speeds from cached hostfile
* base: mirror.sanctuaryhost.com
* updates: mirror.fdcservers.net
* addons: mirror.steadfast.net
* extras: mirror.trouble-free.net
Installed Packages
Name   : mysql
Arch   : i386
Version: 5.0.45
Release: 7.el5
Size   : 7.3 M
Repo   : installed
Summary: MySQL client programs and shared libraries.
Description:
MySQL is a multi-user, multi-threaded SQL database server. MySQL is a
client/server implementation consisting of a server daemon (mysqld)
and many different client programs and libraries. The base package
contains the MySQL client programs, the client shared libraries, and
generic MySQL files.


Available Packages
Name   : mysql
Arch   : i386
Version: 5.0.77
Release: 3.el5
Size   : 4.8 M
Repo   : base
Summary: MySQL client programs and shared libraries
Description:
MySQL is a multi-user, multi-threaded SQL database server. MySQL is a
client/server implementation consisting of a server daemon (mysqld)
and many different client programs and libraries. The base package
contains the MySQL client programs, the client shared libraries, and
generic MySQL files.

The version 5.0.77 is good, but it not last updated release, which is 5.1.40 up today, which contains an important mysql feature named
Pationing: Enables distributing portions of individual tables across a file system, according to rules which can be set when the table is created. In effect, different portions of a table are stored as separate tables in different locations, but from the user point of view, the partitioned table is still a single table.

Basically it won’t do an upgrade as the vendor has changed from being MySQL to Sun Microsystems, and as a result I have to do a complete uninstall and re-install manually, first backup your data, stop mysql service and uninstall the current installation( 5.0.45 in my case):

[root@16 /]service mysqld stop
[root@16 /]rpm -qa | grep -i '^mysql-'
mysql-server-5.0.45-7.el5
mysql-5.0.45-7.el5
mysql-devel-5.0.45-7.el5
[root@16 /]rpm -e mysql-server-5.0.45-7.el5
[root@16 /]rpm -e mysql-5.0.45-7.el5
[root@16 /]rpm -e mysql-devel-5.0.45-7.el5

Now download all the current MySQL packages you need and install all with rpm -i( links from mySQL site):

[root@linux /usr/local/bin] mkdir mysql_5.1.40
[root@linux /usr/local/bin] cd mysql_5.1.40
wget http://dev.mysql.com/get/Downloads/MySQL-5.1/MySQL-server-community-5.1.40-0.rhel5.i386.rpm/from/http://mirror.mirimar.net/mysql/
wget http://dev.mysql.com/get/Downloads/MySQL-5.1/MySQL-client-community-5.1.40-0.rhel5.i386.rpm/from/http://mirror.mirimar.net/mysql/
wget http://dev.mysql.com/get/Downloads/MySQL-5.1/MySQL-shared-community-5.1.40-0.rhel5.i386.rpm/from/http://mirror.mirimar.net/mysql/
wget http://dev.mysql.com/get/Downloads/MySQL-5.1/MySQL-shared-compat-5.1.40-0.rhel5.i386.rpm/from/http://mirror.mirimar.net/mysql/
wget http://dev.mysql.com/get/Downloads/MySQL-5.1/MySQL-devel-community-5.1.40-0.rhel5.i386.rpm/from/http://mirror.mirimar.net/mysql/
wget http://dev.mysql.com/get/Downloads/MySQL-5.1/MySQL-embedded-community-5.1.40-0.rhel5.i386.rpm/from/http://mirror.mirimar.net/mysql/
wget http://dev.mysql.com/get/Downloads/MySQL-5.1/MySQL-test-community-5.1.40-0.rhel5.i386.rpm/from/http://mirror.mirimar.net/mysql/
[root@linux /mysql_5.1.40] rpm -i MySQL-shared-community-5.1.40-0.rhel5
[root@linux /mysql_5.1.40] rpm -i MySQL-embedded-community-5.1.40-0.rhel5
[root@linux /mysql_5.1.40] rpm -i MySQL-server-community-5.1.40-0.rhel5
[root@linux /mysql_5.1.40] rpm -i MySQL-client-community-5.1.40-0.rhel5
[root@linux /mysql_5.1.40] rpm -i MySQL-test-community-5.1.40-0.rhel5
[root@linux /mysql_5.1.40] rpm -i MySQL-devel-community-5.1.40-0.rhel5
[root@linux /mysql_5.1.40] mysql_upgrade
[root@linux /mysql_5.1.40] service mysql start
[root@linux /mysql_5.1.40] mysqladmin -V
 mysqladmin  Ver 8.42 Distrib 5.1.40, for pc-linux-gnu on i686