php版のfluentdモドキを作ってみた

アクセスログなどの収集とか色々便利なソフトとして有名なfluentdっぽいのがPHPで無いようなので作ってみました。
PHPでデーモン作るにはどうやっているのかを勉強するために、tailコマンドのFオプションを作ることから始まりました。

php5-tail

ファイル構成

/etc/
    php5-tail/
        config.php
        cache/
            access.pos
        method/
            format.php
            parse.php
            stdout.php
/usr/
    local/
       bin/
           php5-tail

Inotifyモジュールのインストール

sudo pecl install inotify

動作確認

php5-tail -c /etc/php5-tail/config.php

HTTPサーバへアクセス

curl http://localhost/

出力結果

Array
(
    [host] => 192.168.196.1
    [time] => 14/Apr/2013:10:41:54 -0700
    [path] => /
    [referer] => -
    [agent] => Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.43 Safari/537.31
)

デーモン化

循環参照コレクタを有効にする

gc_enable();

起動した親プロセスを終了する

$pid = pcntl_fork();
if ($pid === -1) {
    throw new \Exception('Process could not be forked');
} else if ($pid) {
    // 親プロセス
    exit();
} else {
    // 子プロセス
}

シグナルをキャッチする

pcntl_signal(SIGHUP,  'handler');
pcntl_signal(SIGINT,  'handler');
pcntl_signal(SIGQUIT, 'handler');
pcntl_signal(SIGTERM, 'handler');
pcntl_signal(SIGTSTP, 'handler');

function handler($signo) {
    switch ($signo) {
    case SIGHUP:   // 再読み込み
        echo $signo."\n";
        break;
    case SIGINT:   // 割り込み
    case SIGQUIT:  // 終了
    case SIGTERM:  // 正常終了
    case SIGTSTP:  // サスペンド
        echo $signo."\n";
        posix_kill(posix_getpid(), SIGUSR1);
        exit();
    }
}

declare(ticks = 1);
wile (true) {
    echo '.';
}

多分この3つをいれれば、とりあえずデーモン化出来ると思います。
System_Daemon – Pearを参考にしました。

Posted in php.

コンパイルからのインストール

漢(オトコ)のコンピュータ道: spider

必要モジュールなどをインストール

sudo aptitude install gcc g++ make cmake
sudo aptitude install bison libncurses5-dev libncursesw5-dev

MySQLユーザー/グループの追加

sudo groupadd mysql
sudo useradd -r -g mysql mysql

ソースのダウンロード

sudo wget https://launchpadlibrarian.net/131491052/mysql-5.5.14-spider-3.0-vp-0.18-hs-1.2-q4m-0.95.tgz
sudo tar xzvf mysql-5.5.14-spider-3.0-vp-0.18-hs-1.2-q4m-0.95.tgz
cd mysql-5.5.14-spider-3.0-vp-0.18-hs-1.2-q4m-0.95

Spiderホームページ

CMAKEを利用したコンパイル

sudo cmake . \
 -DCMAKE_INSTALL_PREFIX=/usr/local/mysql \
 -DDEFAULT_CHARSET=utf8 \
 -DDEFAULT_COLLATION=utf8_general_ci \
 -DENABLED_LOCAL_INFILE=true \
 -DWITH_INNOBASE_STORAGE_ENGINE=1 \
 -DWITH_EXTRA_CHARSETS=all \
 -DWITH_READLINE=ON

sudo make
sudo make install

ディレクトリの作成と権限変更

sudo chown -R mysql:mysql /usr/local/mysql/

sudo cp ./scripts/install_*.sql /usr/local/mysql/scripts/

sudo mkdir /var/log/mysql/
sudo touch /var/log/mysql/error.log
sudo touch /var/log/mysql/query.log
sudo touch /var/log/mysql/slow.log
sudo chown -R mysql:mysql /var/log/mysql/

MySQLの設定

sudo vi /etc/my.cnf
[client]
port            = 3306
socket          = /var/run/mysqld/mysqld.sock

[mysqld]
port                = 3306
socket              = /var/run/mysqld/mysqld.sock
basedir             = /usr/local/mysql
datadir             = /var/lib/mysql
tmpdir              = /tmp
log_error           = /var/log/mysql/error.log
slow-query-log      = ON
slow-query-log-file = /var/log/mysql/slow.log
skip-external-locking
key_buffer_size     = 256M
max_allowed_packet  = 1M
table_open_cache    = 256
sort_buffer_size    = 1M
read_buffer_size    = 1M
read_rnd_buffer_size = 4M
myisam_sort_buffer_size = 64M
thread_cache_size = 8
query_cache_size= 16M
thread_concurrency = 8

log-bin=mysql-bin
binlog_format=mixed
server-id       = 1

log_bin_trust_function_creators = 1

[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash

[myisamchk]
key_buffer_size = 128M
sort_buffer_size = 128M
read_buffer = 2M
write_buffer = 2M

[mysqlhotcopy]
interactive-timeout

[mysqld_safe]
basedir=/usr/local/mysql

データベースの初期化

sudo mkdir /var/lib/mysql
sudo chown mysql:mysql /var/lib/mysql

cd /usr/local/mysql
sudo ./scripts/mysql_install_db --user=mysql --basedir=/usr/local/mysql --datadir=/var/lib/mysql

起動スクリプトの設定

sudo cp /usr/local/mysql/support-files/mysql.server /etc/init.d/

シンボリックリンクの作成

cd /usr/bin
sudo ln -s /usr/local/mysql/bin/* ./

cd /usr/share
sudo ln -s /usr/local/mysql/share ./mysql

MySQLサービスの起動

sudo service mysql.server start

全権限ユーザーの作成

mysql> CREATE USER 'sheeps'@'%' IDENTIFIED BY  '***';
mysql> GRANT ALL PRIVILEGES ON * . * TO  'sheeps'@'%' IDENTIFIED BY  '***' WITH GRANT OPTION MAX_QUERIES_PER_HOUR 0 MAX_CONNECTIONS_PER_HOUR 0 MAX_UPDATES_PER_HOUR 0 MAX_USER_CONNECTIONS 0 ;

ユーザーと同一のDBに全権限を与える

mysql> CREATE USER 'wordpress'@'localhost' IDENTIFIED BY  '***';
mysql> GRANT USAGE ON * . * TO  'wordpress'@'localhost' IDENTIFIED BY  '***' WITH MAX_QUERIES_PER_HOUR 0 MAX_CONNECTIONS_PER_HOUR 0 MAX_UPDATES_PER_HOUR 0 MAX_USER_CONNECTIONS 0 ;
mysql> CREATE DATABASE IF NOT EXISTS  `wordpress` ;
mysql> GRANT ALL PRIVILEGES ON  `wordpress` . * TO  'wordpress'@'localhost';

Spider

Spiderストレージエンジンのインストール

cd /usr/local/mysql
./bin/mysql -u root < ./scripts/install_spider.sql

インストール確認
[sourcecode language="plain"]
mysql> SELECT * FROM INFORMATION_SCHEMA.plugins WHERE PLUGIN_NAME = 'SPIDER';
+-------------+----------------+---------------+----------------+---------------------+----------------+------------------------+---------------+-----------------------+----------------+-------------+
| PLUGIN_NAME | PLUGIN_VERSION | PLUGIN_STATUS | PLUGIN_TYPE | PLUGIN_TYPE_VERSION | PLUGIN_LIBRARY | PLUGIN_LIBRARY_VERSION | PLUGIN_AUTHOR | PLUGIN_DESCRIPTION | PLUGIN_LICENSE | LOAD_OPTION |
+-------------+----------------+---------------+----------------+---------------------+----------------+------------------------+---------------+-----------------------+----------------+-------------+
| SPIDER | 3.0 | ACTIVE | STORAGE ENGINE | 50514.0 | ha_spider.so | 1.2 | Kentoku Shiba | Spider storage engine | GPL | ON |
+-------------+----------------+---------------+----------------+---------------------+----------------+------------------------+---------------+-----------------------+----------------+-------------+

mysql> SELECT * FROM mysql.func WHERE dl = 'ha_spider.so';
+------------------------------+-----+--------------+-----------+
| name | ret | dl | type |
+------------------------------+-----+--------------+-----------+
| spider_direct_sql | 2 | ha_spider.so | function |
| spider_bg_direct_sql | 2 | ha_spider.so | aggregate |
| spider_ping_table | 2 | ha_spider.so | function |
| spider_copy_tables | 2 | ha_spider.so | function |
| spider_flush_table_mon_cache | 2 | ha_spider.so | function |
+------------------------------+-----+--------------+-----------+

mysql> SELECT * FROM INFORMATION_SCHEMA.plugins WHERE PLUGIN_LIBRARY = 'ha_spider.so';
+------------------+----------------+---------------+--------------------+---------------------+----------------+------------------------+---------------+---------------------------------+----------------+-------------+
| PLUGIN_NAME | PLUGIN_VERSION | PLUGIN_STATUS | PLUGIN_TYPE | PLUGIN_TYPE_VERSION | PLUGIN_LIBRARY | PLUGIN_LIBRARY_VERSION | PLUGIN_AUTHOR | PLUGIN_DESCRIPTION | PLUGIN_LICENSE | LOAD_OPTION |
+------------------+----------------+---------------+--------------------+---------------------+----------------+------------------------+---------------+---------------------------------+----------------+-------------+
| SPIDER | 3.0 | ACTIVE | STORAGE ENGINE | 50514.0 | ha_spider.so | 1.2 | Kentoku Shiba | Spider storage engine | GPL | ON |
| SPIDER_ALLOC_MEM | 0.1 | ACTIVE | INFORMATION SCHEMA | 50514.0 | ha_spider.so | 1.2 | Kentoku Shiba | Spider memory allocating viewer | GPL | ON |
+------------------+----------------+---------------+--------------------+---------------------+----------------+------------------------+---------------+---------------------------------+----------------+-------------+
[/sourcecode]

VPストレージエンジンのインストール

cd /usr/local/mysql
./bin/mysql -u root < ./scripts/install_vp.sql

インストール確認
[sourcecode language="plain"]
mysql> SELECT * FROM INFORMATION_SCHEMA.plugins WHERE PLUGIN_NAME = 'VP';
+-------------+----------------+---------------+----------------+---------------------+----------------+------------------------+---------------+--------------------------------------+----------------+-------------+
| PLUGIN_NAME | PLUGIN_VERSION | PLUGIN_STATUS | PLUGIN_TYPE | PLUGIN_TYPE_VERSION | PLUGIN_LIBRARY | PLUGIN_LIBRARY_VERSION | PLUGIN_AUTHOR | PLUGIN_DESCRIPTION | PLUGIN_LICENSE | LOAD_OPTION |
+-------------+----------------+---------------+----------------+---------------------+----------------+------------------------+---------------+--------------------------------------+----------------+-------------+
| VP | 0.18 | ACTIVE | STORAGE ENGINE | 50514.0 | ha_vp.so | 1.2 | Kentoku Shiba | Vertical Partitioning Storage Engine | GPL | ON |
+-------------+----------------+---------------+----------------+---------------------+----------------+------------------------+---------------+--------------------------------------+----------------+-------------+

mysql> SELECT * FROM mysql.func WHERE dl = 'ha_vp.so';
+----------------+-----+----------+----------+
| name | ret | dl | type |
+----------------+-----+----------+----------+
| vp_copy_tables | 2 | ha_vp.so | function |
+----------------+-----+----------+----------+
[/sourcecode]

Handler Socketのインストール

cd /usr/local/mysql
./bin/mysql -u root < ./scripts/install_handlersocket.sql

インストール確認
[sourcecode language="plain"]
mysql> SELECT * FROM INFORMATION_SCHEMA.plugins WHERE PLUGIN_NAME = 'handlersocket';
+---------------+----------------+---------------+-------------+---------------------+------------------+------------------------+----------------------------------+--------------------+----------------+-------------+
| PLUGIN_NAME | PLUGIN_VERSION | PLUGIN_STATUS | PLUGIN_TYPE | PLUGIN_TYPE_VERSION | PLUGIN_LIBRARY | PLUGIN_LIBRARY_VERSION | PLUGIN_AUTHOR | PLUGIN_DESCRIPTION | PLUGIN_LICENSE | LOAD_OPTION |
+---------------+----------------+---------------+-------------+---------------------+------------------+------------------------+----------------------------------+--------------------+----------------+-------------+
| handlersocket | 1.2 | ACTIVE | DAEMON | 50514.0 | handlersocket.so | 1.2 | higuchi dot akira at dena dot jp | | BSD | ON |
+---------------+----------------+---------------+-------------+---------------------+------------------+------------------------+----------------------------------+--------------------+----------------+-------------+
[/sourcecode]

CREATE SERVERによりデータノードとなるMySQLサーバを登録

CREATE SERVER `db000` FOREIGN DATA WRAPPER mysql OPTIONS (USER 'mysql', PASSWORD '***', HOST '192.168.22.6', PORT 3306);
CREATE SERVER `db001` FOREIGN DATA WRAPPER mysql OPTIONS (USER 'mysql', PASSWORD '***', HOST '192.168.22.7', PORT 3306);
CREATE SERVER `db002` FOREIGN DATA WRAPPER mysql OPTIONS (USER 'mysql', PASSWORD '***', HOST '192.168.22.8', PORT 3306);

実データを保存するMySQKサーバ側

CREATE TABLE `spider-sample`.`sample` (  
  `id` int(11) NOT NULL,  
  `store` int(11) NOT NULL, 
  `test` varchar(32) DEFAULT NULL, 
  PRIMARY KEY (id,store)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

Spider

CREATE TABLE `spider-sample`.`sample` (  
  `id` int(11) NOT NULL,  
  `store` int(11) NOT NULL, 
  `test` varchar(32) DEFAULT NULL, 
  PRIMARY KEY (id,store)
) ENGINE=Spider DEFAULT CHARSET=utf8
PARTITION BY HASH(store) (
  PARTITION p0 COMMENT 'server "db000", table "sample"',
  PARTITION p1 COMMENT 'server "db001", table "sample"',
  PARTITION p2 COMMENT 'server "db002", table "sample"'
);  

Spiderでincrement設定

SHOW VARIABLES LIKE 'auto_inc%';
+--------------------------+-------+
| Variable_name            | Value |
+--------------------------+-------+
| auto_increment_increment | 1     |
| auto_increment_offset    | 1     |
+--------------------------+-------+

Spiderでインサートする場合に複数台使用する場合に
上記数値を調整する

spider000

SET auto_increment_increment=3
SET auto_increment_offset=1

spider001

SET auto_increment_increment=3
SET auto_increment_offset=2

spider002

SET auto_increment_increment=3
SET auto_increment_offset=3

spiderが3台構成の例


権限エラー

Access denied; you need (at least one of) the SUPER privilege(s) for this operation

ログ設定を統一する

show variables like 'spider_internal_sql_log_off';
+-----------------------------+-------+
| Variable_name               | Value |
+-----------------------------+-------+
| spider_internal_sql_log_off | ON    |
+-----------------------------+-------+

show variables like 'spider_remote_sql_log_off';
+---------------------------+-------+
| Variable_name             | Value |
+---------------------------+-------+
| spider_remote_sql_log_off | -1    |
+---------------------------+-------+

SET GLOBAL spider_remote_sql_log_off = 1;

show variables like 'spider_remote_sql_log_off';
+---------------------------+-------+
| Variable_name             | Value |
+---------------------------+-------+
| spider_remote_sql_log_off | 1     |
+---------------------------+-------+

RDSをデータ保存用のMySQLにする場合は上記設定が必要になるようです。

hadoop設定の注意点

ネーム解決エラー

INFO org.apache.hadoop.metrics.MetricsUtil: Unable to obtain hostName
java.net.UnknownHostException: ip-192-168-11-16 : ip-192-168-11-16
        at java.net.InetAddress.getLocalHost(InetAddress.java:1374)
        at org.apache.hadoop.metrics.MetricsUtil.getHostName(MetricsUtil.java:91)
        at org.apache.hadoop.metrics.MetricsUtil.createRecord(MetricsUtil.java:80)
        at org.apache.hadoop.security.UserGroupInformation$UgiMetrics.(UserGroupInformation.java:102)
        at org.apache.hadoop.security.UserGroupInformation.(UserGroupInformation.java:208)
        at org.apache.hadoop.fs.FileSystem$Cache$Key.(FileSystem.java:1765)
        at org.apache.hadoop.fs.FileSystem$Cache$Key.(FileSystem.java:1758)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1626)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
        at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:225)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1668)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1623)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1641)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1767)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1784)

ネーム解決されないエラーがでました。
hostsだけ、設定して問題ない場合もありましたが
hostnameを設定するとうまく動作しました。

hostnameの設定

sudo vi /etc/hostname
slaves001.sheeps.me

hostnameの反映

sudo hostname -b -F /etc/hostname

hostname -b -F /etc/hostname

hostsの設定

sudo vi /etc/hosts
192.168.11.14   masters000.sheeps.me    masters000
192.168.11.15   slaves000.sheeps.me     slaves000
192.168.11.16   slaves001.sheeps.me     slaves001
192.168.11.17   slaves002.sheeps.me     slaves002

サービスの起動

sudo service hadoop-0.20-datanode restart
sudo service hadoop-0.20-tasktracker restart

tcpdumpで確認

sudo tcpdump -s 1600 -X -i eth0 src port 8020
sudo tcpdump -s 1600 -X -i eth0 dst port 8020
sudo tcpdump -s 1600 -X -i eth0 src port 8021
sudo tcpdump -s 1600 -X -i eth0 dst port 8021

masters000.sheeps.meから実行

sudo tcpdump -s 1600 -X -i eth0 src host slaves000
sudo tcpdump -s 1600 -X -i eth0 src host slaves001
sudo tcpdump -s 1600 -X -i eth0 src host slaves002

自分用リンク

JavaScript

lodash Underscore.js
Underscoreなどのパフォーマンス強化版など

dropzonejs.js - ファイルアップロード・ドロップ&ドロップサポート

Backbone
Backbone.js 日本語リファレンス

havelog:Backboneタグリスト
BackboneのTipsなど

Backbone.jsでUIバインディング

Backbone.jsの使い方
BackboneのTipsなど日本語で解説

Backbone.jsやUnderscore.jsなどの翻訳

RequireJS と Backbone.js を組み合わせてモデルとビューをモジュール化してみる

Backboneの小ネタなパターン3つ (Backbone Advent Calendar 2012 21th day)

プラグイン
backbone.viewkitプラグイン

backbone.validationプラグイン

グラフ系
Sparkline - ミニグラフ描画
JustGage - 計器グラフ描画
xCharts - 棒グラフやライングラフ
easy pie chart - 円グラフ
morris.js - SVG(Raphaeljsを使用)によるグラフ描画
Flotr2 - 強力なグラフ描画
DOMTree - DOMツリーなグラフ描画
canviz - Graphviz のDot言語を利用できる描画
Data-Driven Documents - D3.js 超強力なグラフ描画

JointJS - チャート・フロー描画

カレンダー
fullcalendar - googleカレンダー風

変換ツール
emscripten - C/C++をjavascriptへ変換
  ウシロード - emscriptenの使い方
  graphviz.js - Graphvizのjs移植
asm.js - low-level subset of JavaScript

スマートフォン

javascript開発
phonegap
trigger.io

node.js

通信プラグイン・開発支援
socket.io

PHP

PHPのBDDフレームワーク Behat について発表してきました
Behat、BDDテストフレームワーク

Slim is a PHP micro framework
Route callback PHPフレームワーク

スタイルシート

cssarrowplease - 吹き出し(矢印)付きメッセージ
Font Awesome - WEBフォントを利用したアイコン集

ショッピングサイト

Stores.jp - 最新技術満載ショッピングサイト
ベイス - ショップ作成が30秒で出来るサイト

まとめ・記事閲覧

Antenna - スタイリッシュな記事閲覧

SEO

Schema Japan - 検索エンジン等にコンテンツの情報を伝える仕組み

PHPのプロファイリング

Gitからソースを取得

git clone https://github.com/facebook/xhprof.git

コンパイル

cd xhprof/extension/
phpize
./configure
make
sudo make install

モジュールの追加

sudo vi /etc/php5/conf.d/xhprof.ini
; configuration for php XHprof module
extension=xhprof.so
xhprof.output_dir="/var/www/xhprof.sheeps.me/log"

コールグラフ生成用ツールのインストール

sudo aptitude -y install graphviz

Webインターフェースの設定

cp -f xhprof/xhprof_html/ /var/www/xhprof

xhprof_htmlの中身を閲覧できるように設定

XHProfの実行サンプル
[php]
<?php
function bar($x) {
if ($x > 0) {
bar($x – 1);
}
}

function foo() {
for ($idx = 0; $idx < 5; $idx++) {
bar($idx);
$x = strlen("abc");
}
}

// start profiling
xhprof_enable();

// run program
foo();

// stop profiler
$xhprof_data = xhprof_disable();

// display raw xhprof data for the profiler run
print_r($xhprof_data);

$XHPROF_ROOT = realpath(dirname(__FILE__) .’/..’);
include_once $XHPROF_ROOT . "/xhprof_lib/utils/xhprof_lib.php";
include_once $XHPROF_ROOT . "/xhprof_lib/utils/xhprof_runs.php";

// save raw data for this profiler run using default
// implementation of iXHProfRuns.
$xhprof_runs = new XHProfRuns_Default();

// save the run under a namespace "xhprof_foo"
$run_id = $xhprof_runs->save_run($xhprof_data, "xhprof_foo");

echo "—————\n".
"Assuming you have set up the http based UI for \n".
"XHProf at some address, you can view run at \n".
"http://<xhprof-ui-address>/index.php?run=$run_id&source=xhprof_foo\n".
"—————\n";
?>
[/php]

Posted in php.

MySQLのインポート・エクスポート

テーブルごとにダンプ

mysqldump --quick \
          --single-transaction \
          --add-locks \
          --no-autocommit \
          --user=mysql \
          --password=passwd \
          --host=localhost \
          --default-character-set=utf8 \
          smpldb \
          smpltbl > ./tbldump.sql

CSVファイル出力

SELECT * FROM `smpltbl` INTO OUTFILE "/var/tmp/smpltbl.csv" FIELDS TERMINATED BY ',';

CSVファイル入力

LOAD DATA LOCAL INFILE "/var/tmp/smpltbl.csv" INTO TABLE `smpltbl` FIELDS TERMINATED BY ',';

バイナリログの変換

mysqlbinlog /var/lib/mysql/groonga.log > /tmp/groonga.sql

JAVA開発支援ツール

Mavenのインストール

sudo aptitude -y install maven2

環境変数の設定

export JAVA_HOME=/usr/lib/jvm/java-6-sun
export CLASSPATH=".:/usr/lib/jvm/java-6-sun/lib" 

Mavenへライブラリのインストール

Hadoop関連

export HADOOP_HOME=/usr/lib/hadoop-0.20
export HBASE_HOME=/usr/lib/hbase


mvn install:install-file -DgroupId=org.apache.hadoop -DartifactId=hadoop-core -Dversion=1.2.1 -Dpackaging=jar -Dfile=${HADOOP_HOME}/hadoop-core.jar
mvn install:install-file -DgroupId=org.apache.zookeeper -DartifactId=zookeeper -Dversion=3.4.2 -Dpackaging=jar -Dfile=${HBASE_HOME}/lib/zookeeper.jar
mvn install:install-file -DgroupId=org.apache.hadoop -DartifactId=hbase -Dversion=0.90.6 -Dpackaging=jar -Dfile=${HBASE_HOME}/hbase.jar

Sunライブラリ関連
Download the version 1.1 API Documentation, Jar and Source
Java Management Extension (JMX) 1.2.1

mvn install:install-file -DgroupId=javax.jms -DartifactId=jms -Dversion=1.1 -Dpackaging=jar -Dfile=/usr/lib/jvm/java-6-sun/lib/jms.jar
mvn install:install-file -DgroupId=com.sun.jmx -DartifactId=jmxri -Dversion=1.2.1 -Dpackaging=jar -Dfile=/usr/lib/jvm/java-6-sun/lib/jmxri.jar
mvn install:install-file -DgroupId=com.sun.jdmk -DartifactId=jmxtools -Dversion=1.2.1 -Dpackaging=jar -Dfile=/usr/lib/jvm/java-6-sun/lib/jmxtools.jar

Mavenリポジトリ

Mavenでの開発

プロジェクトの作成

mkdir projects
cd projects

mvn archetype:create -DgroupId=me.sheeps.hdfs -DartifactId=sample
cd sample

生成されるファイル

./sample
    src/
        main/
            java/
                me/
                    sheeps/
                        hdfs/
                            App.java
        test/
            java/
                me/
                    sheeps/
                        hdfs/
                            AppTest.java
    pom.xml

pom.xmlに利用するライブラリを追加

vi pom.xml

[xml]
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>me.sheeps.hdfs</groupId>
<artifactId>sample</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>

<name>sample</name>
<url>http://maven.apache.org</url>

<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.6</source>
<target>1.6</target>
<encoding>UTF-8</encoding>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<archive>
<manifest>
<mainClass>me.sheeps.hdfs.App</mainClass>
<packageName>me.sheeps.hdfs</packageName>
<addClasspath>true</addClasspath>
<addExtensions>true</addExtensions>
<classpathPrefix>lib</classpathPrefix>
</manifest>
</archive>
</configuration>
</plugin>
</plugins>
</build>

<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>com.sun.jmx</groupId>
<artifactId>jmxri</artifactId>
<version>1.2.1</version>
</dependency>

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>0.20</version>
</dependency>

<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
<version>3.4.2</version>
</dependency>

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hbase</artifactId>
<version>0.90.6</version>
</dependency>

<dependency>
<groupId>commons-cli</groupId>
<artifactId>commons-cli</artifactId>
<version>1.1</version>
</dependency>

</dependencies>
</project>
[/xml]

hBaseのサンプル

[java]
package me.sheeps.hdfs;

import java.io.IOException;
import java.lang.System;
import java.util.NavigableMap;
import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.util.GenericOptionsParser;

import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.ResultScanner;
import org.apache.hadoop.hbase.filter.Filter;
import org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter;
import org.apache.hadoop.hbase.filter.BinaryPrefixComparator;
import org.apache.hadoop.hbase.filter.CompareFilter.CompareOp;
import org.apache.hadoop.hbase.filter.DependentColumnFilter;
import org.apache.hadoop.hbase.util.Bytes;

/**
* Appクラス
*
* @package Sample
* @author Yujiro Takahashi <yujiro3@gmail.com>
*/
public class App {
/**
* メイン
*
* @access public
* @param String[] args
* @return void
*/
public static void main(String[] args) throws Exception {
// 設定情報の読み込み
Configuration conf = HBaseConfiguration.create();
conf.addResource("/etc/hbase/conf/hbase-site.xml");
conf.set("hbase.client.scanner.caching", "3");

// 引数のパース
new GenericOptionsParser(conf, args);

HTable table = new HTable(conf, Bytes.toBytes("accesslog")); // テーブルを指定

// Scan条件の指定
Scan scan = new Scan();
Filter filter = new DependentColumnFilter(
Bytes.toBytes("user"), // カラムファミリを指定
Bytes.toBytes("id"), // 修飾子を指定
false,
CompareOp.EQUAL,
new BinaryPrefixComparator(Bytes.toBytes(args[0]))
);
scan.setFilter(filter);
ResultScanner scanner = table.getScanner(scan);

System.out.println("/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/");

for (Result rr = scanner.next(); rr != null; rr = scanner.next()) {
String row = Bytes.toString(rr.getValue(Bytes.toBytes(args[1]), Bytes.toBytes(args[2])));
System.out.println(row);
}
}
}
[/java]

コンパイル

mvn compile

パッケージ化

mvn clean package

実行

hadoop jar ./target/sample-1.0-SNAPSHOT.jar 1258878 log timestamp

PEARパッケージ管理のPyrusをインストール

sqlite3が必要になります

sudo aptitude -y install php5-sqlite

PyrusのPHARファイルをダウンロード

mkdir ~/.pear
cd ~/.pear/
wget http://pear2.php.net/pyrus.phar

Pyrusの初期設定

php pyrus.phar install

php pyrus.phar mypear ~/.pear/vendor
php pyrus.phar set bin_dir ~/.pear/vendor/bin

pyrusコマンド登録

vi ~/.pear/pyrus
#!/usr/bin/env bash

/usr/bin/env php -q ~/.pyrus/pyrus.phar $@
chmod +x ~/.pear/pyrus

Aliasでpyrusコマンド登録

alias pyrus='php -q ~/.pear/pyrus.phar'

ヘルプの表示

pyrus --help
Pyrus, the PHP manager

Usage:
  php pyrus.phar [/path/to/pyrus] [options]
  php pyrus.phar [/path/to/pyrus] [options]  [options] [args]

Options:
  -v, --verbose   increase verbosity
  -p, --paranoid  set or increase paranoia level
  -h, --help      show this help message and exit
  --version       show the program version and exit

Commands:
  install             Install a package.  Use install --plugin to install
                      plugins
  upgrade             Upgrade a package.  Use upgrade --plugin to upgrade
                      plugins
  uninstall           Uninstall a package.  Use uninstall --plugin to
                      uninstall plugins
  info                Display information about a package
  build               Build a PHP extension package from source and install
                      the compiled extension
  list-upgrades       List packages with upgrades available
  remote-list         List all remote packages in a channel, organized by
                      category
  download            Download a remote package to the current directory
  list-packages       List all installed packages in all channels
  list-channels       List all discovered channels
  channel-discover    Discover a new channel
  channel-del         Remove a channel from the registry
  upgrade-registry    Upgrade an old PEAR installation to the new registry
                      format
  run-scripts         Run all post-install scripts for a package
  set                 Set a configuration value
  get                 Get configuration value(s). Leave blank for all
                      values
  mypear              Set a configuration value
  help                Get help on a particular command, or all commands
  search              Search a registry of PEAR channels for packages
  make                Create or update a package.xml from a standard PEAR2
                      directory layout
  pickle              Create or update a package.xml and then package a
                      PECL extension release
  package             Create a release from an existing package.xml
  run-phpt            Run PHPT tests
  generate-pear2      Generate the source layout for a new
                      Pyrus-installable package
  generate-ext        Generate the source layout for a new PHP extension
                      that is PECL-ready
  scs-update          Simple channel server: Update all releases of a
                      within the get/ directory.
  scs-create          Simple channel server: Create a channel.xml, get/ and
                      rest/ directory for a channel
  scs-add-maintainer  Simple Channel Server: Add a new maintaing developer
                      to the channel
  scs-add-category    Simple Channel Server: Add a new category to the
                      channel
  scs-categorize      Simple Channel Server: Categorize a package
  scs-release         Simple Channel Server: Release a package

Unknown command: --help

Pyrus, the PHP manager

Usage:
  php pyrus.phar [/path/to/pyrus] [options]
  php pyrus.phar [/path/to/pyrus] [options]  [options] [args]

Options:
  -v, --verbose   increase verbosity
  -p, --paranoid  set or increase paranoia level
  -h, --help      show this help message and exit
  --version       show the program version and exit

Commands:
  install             Install a package.  Use install --plugin to install
                      plugins
  upgrade             Upgrade a package.  Use upgrade --plugin to upgrade
                      plugins
  uninstall           Uninstall a package.  Use uninstall --plugin to
                      uninstall plugins
  info                Display information about a package
  build               Build a PHP extension package from source and install
                      the compiled extension
  list-upgrades       List packages with upgrades available
  remote-list         List all remote packages in a channel, organized by
                      category
  download            Download a remote package to the current directory
  list-packages       List all installed packages in all channels
  list-channels       List all discovered channels
  channel-discover    Discover a new channel
  channel-del         Remove a channel from the registry
  upgrade-registry    Upgrade an old PEAR installation to the new registry
                      format
  run-scripts         Run all post-install scripts for a package
  set                 Set a configuration value
  get                 Get configuration value(s). Leave blank for all
                      values
  mypear              Set a configuration value
  help                Get help on a particular command, or all commands
  search              Search a registry of PEAR channels for packages
  make                Create or update a package.xml from a standard PEAR2
                      directory layout
  pickle              Create or update a package.xml and then package a
                      PECL extension release
  package             Create a release from an existing package.xml
  run-phpt            Run PHPT tests
  generate-pear2      Generate the source layout for a new
                      Pyrus-installable package
  generate-ext        Generate the source layout for a new PHP extension
                      that is PECL-ready
  scs-update          Simple channel server: Update all releases of a
                      within the get/ directory.
  scs-create          Simple channel server: Create a channel.xml, get/ and
                      rest/ directory for a channel
  scs-add-maintainer  Simple Channel Server: Add a new maintaing developer
                      to the channel
  scs-add-category    Simple Channel Server: Add a new category to the
                      channel
  scs-categorize      Simple Channel Server: Categorize a package
  scs-release         Simple Channel Server: Release a package
Posted in php.

日本語フォントのインストールがメイン

Phantomjsのインストール

cd /usr/local/share/
sudo wget http://phantomjs.googlecode.com/files/phantomjs-1.7.0-linux-x86_64.tar.bz2
sudo tar xjvf phantomjs-1.7.0-linux-x86_64.tar.bz2
sudo rm -rf phantomjs-1.7.0-linux-x86_64.tar.bz2
sudo mv phantomjs-1.7.0-linux-x86_64 phantomjs
sudo ln -s /usr/local/share/phantomjs/bin/phantomjs /usr/bin/phantomjs

日本語フォントのインストール

フォント設定ツールのインストール

sudo aptitude -y install fontconfig

フォント一覧を確認

fc-list
/usr/share/fonts/truetype/ttf-dejavu/DejaVuSerif.ttf: DejaVu Serif:style=Book
/usr/share/fonts/truetype/ttf-dejavu/DejaVuSans.ttf: DejaVu Sans:style=Book
/usr/share/fonts/truetype/ttf-dejavu/DejaVuSansMono-Bold.ttf: DejaVu Sans Mono:style=Bold
/usr/share/fonts/truetype/ttf-dejavu/DejaVuSerif-Bold.ttf: DejaVu Serif:style=Bold
/usr/share/fonts/truetype/ttf-dejavu/DejaVuSans-Bold.ttf: DejaVu Sans:style=Bold
/usr/share/fonts/truetype/ttf-dejavu/DejaVuSansMono.ttf: DejaVu Sans Mono:style=Book

好きなフォントをインストールします。

Takaoフォント

sudo aptitude -y install fonts-takao

IPAフォント

sudo aptitude -y install fonts-ipafont

IPAexフォント

sudo aptitude -y install otf-ipaexfont-gothic otf-ipaexfont-mincho

梅フォント

sudo aptitude -y install fonts-horai-umefont

UmePlusフォント

sudo aptitude -y install fonts-umeplus

さざなみフォント

sudo aptitude -y install ttf-sazanami-gothic ttf-sazanami-mincho

東風フォント

sudo aptitude -y install ttf-kochi-gothic ttf-kochi-mincho

VLフォント

sudo aptitude -y install ttf-vlgothic

IPAフォントを入れた場合

sudo aptitude -y install fonts-ipafont
fc-list
/usr/share/fonts/opentype/ipafont-mincho/ipam.ttf: IPAMincho,IPA明朝:style=Regular
/usr/share/fonts/truetype/ttf-dejavu/DejaVuSerif.ttf: DejaVu Serif:style=Book
/usr/share/fonts/truetype/ttf-dejavu/DejaVuSans.ttf: DejaVu Sans:style=Book
/usr/share/fonts/opentype/ipafont-gothic/ipagp.ttf: IPAPGothic,IPA Pゴシック:style=Regular
/usr/share/fonts/truetype/ttf-dejavu/DejaVuSansMono-Bold.ttf: DejaVu Sans Mono:style=Bold
/usr/share/fonts/truetype/ttf-dejavu/DejaVuSerif-Bold.ttf: DejaVu Serif:style=Bold
/usr/share/fonts/opentype/ipafont-mincho/ipamp.ttf: IPAPMincho,IPA P明朝:style=Regular
/usr/share/fonts/opentype/ipafont-gothic/ipag.ttf: IPAGothic,IPAゴシック:style=Regular
/usr/share/fonts/truetype/fonts-japanese-mincho.ttf: IPAMincho,IPA明朝:style=Regular
/usr/share/fonts/truetype/fonts-japanese-gothic.ttf: IPAGothic,IPAゴシック:style=Regular
/usr/share/fonts/truetype/ttf-dejavu/DejaVuSans-Bold.ttf: DejaVu Sans:style=Bold
/usr/share/fonts/truetype/ttf-dejavu/DejaVuSansMono.ttf: DejaVu Sans Mono:style=Book

Yahoo!ジャパンのスクリーンショット

phantomjs /usr/share/phantomjs/examples/rasterize.js http://www.yahoo.co.jp/ yahoo.png

文字化けせずにスクリーンショットが撃てるはすです。

Hadoop streaming用のスクリプトファイルを配布する時など

lsyncd

lsyncdインストール

sudo aptitude -y install lsyncd

lsyncd設定ファイル

sudo vi /etc/lsyncd/lsyncd.conf.lua
----
-- Streaming configuration file for lsyncd.
--
settings = {
    statusFile = "/var/run/lsyncd.stat",
    statusInterval = 30,
}

sync { 
    default.rsync, 
    source="/home/mapred/",
    target="slaves000:/home/mapred/",
    rsyncOps={"-aruz", "--delete"}, 
    delay=10 
}
sync { 
    default.rsync, 
    source="/home/mapred/",
    target="slaves001:/home/mapred/",
    rsyncOps={"-aruz", "--delete"}, 
    delay=10 
}
sync { 
    default.rsync, 
    source="/home/mapred/",
    target="slaves002:/home/mapred/",
    rsyncOps={"-aruz", "--delete"}, 
    delay=10 
}

lsyncdデーモン起動

sudo /etc/init.d/lsyncd start

rsync

rsyncd設定ファイル

sudo vi /etc/rsyncd.conf
# GLOBAL OPTIONS

# pid file = /var/run/rsync.pid
# log file = /var/log/rsync.log

timeout = 600
hosts allow *.sheeps.me
read only = yes

max connections = 2
dont compress = *.gz *.tgz *.zip *.z *.rpm *.deb *.iso *.bz2 *.tbz

[MapReduce]
comment = PHP for Hadoop streaming
path = /home/mapred
uid = mapred
gid = mapred

rsyncdデフォルトファイル

sudo vi /etc/default/rsync
# start rsync in daemon mode from init.d script?
#  only allowed values are "true", "false", and "inetd"
#  Use "inetd" if you want to start the rsyncd from inetd,
#  all this does is prevent the init.d script from printing a message
#  about not starting rsyncd (you still need to modify inetd's config yourself).
RSYNC_ENABLE=true

rsyncデーモン起動

sudo /etc/init.d/rsync start

SSHを利用したlsyncd設定

sudo vi /etc/lsyncd/lsyncd.conf.lua
----
-- Streaming configuration file for lsyncd.
--
settings = {
    statusFile = "/var/run/lsyncd.stat",
    statusInterval = 30,
}

sync {
    default.rsyncssh,
    source="/home/mapred/",
    host="hdfs@slaves000",
    targetdir="/home/mapred/",
    rsyncOps={"-aruz", "--delete"}, 
    delay=10
}

sync {
    default.rsyncssh,
    source="/home/mapred/",
    host="hdfs@slaves001",
    targetdir="/home/mapred/",
    rsyncOps={"-aruz", "--delete"}, 
    delay=10
}

sync {
    default.default.rsyncssh,
    source="/home/mapred/",
    host="hdfs@slaves002",
    targetdir="/home/mapred/",
    rsyncOps={"-aruz", "--delete"}, 
    delay=10
}

rootのSSH設定

sudo vi /root/.ssh/config
Host slaves000
    HostName            slaves000.sheeps.me
    IdentityFile        /root/.ssh/id_rsa
    User                hdfs

Host slaves001
    HostName            slaves001.sheeps.me
    IdentityFile        /root/.ssh/id_rsa
    User                hdfs

Host slaves002
    HostName            slaves002.sheeps.me
    IdentityFile        /root/.ssh/id_rsa
    User                hdfs
sudo cp $HADOOP_HOME/.ssh/id_rsa /root/.ssh/id_rsa
sudo chmod 0600 /root/.ssh/id_rsa

Lsyncdからrsyncを実行するユーザーを変更する方法がわからないので
デフォルトのrootで実行させています。

接続先が/root/.ssh/known_hostsに登録されていないとエラーとなりました。
予め接続して登録しておく必要があるようです。