ポートの開閉

ufwの利用

ファイアーウォールを無効にする

sudo ufw disable

すべてのアクセスを拒否に設定

sudo ufw default deny

SSHを許可

sudo ufw allow ssh

HTTPを許可

sudo ufw allow http

HTTPSを許可

sudo ufw allow https

MySQLをローカルネットワークだけ許可

sudo ufw allow from 192.168.11.0/24 to any port mysql

ファイアーウォールを有効にする

sudo ufw enable

プロキシサーバを無効にする

sudo ufw deny 3128/tcp

ルールを削除する

sudo ufw delete DENY 3128/tcp

設定確認

sudo ufw status

Status: active

To                         Action      From
--                         ------      ----
22                         ALLOW       Anywhere
80                         ALLOW       Anywhere
443                        ALLOW       Anywhere
3306                       ALLOW       192.168.11.0/24

ufwヘルプ

Commands:
 enable                          enables the firewall
 disable                         disables the firewall
 default ARG                     set default policy
 logging LEVEL                   set logging to LEVEL
 allow ARGS                      add allow rule
 deny ARGS                       add deny rule
 reject ARGS                     add reject rule
 limit ARGS                      add limit rule
 delete RULE|NUM                 delete RULE
 insert NUM RULE                 insert RULE at NUM
 reset                           reset firewall
 status                          show firewall status
 status numbered                 show firewall status as numbered list of RULES
 status verbose                  show verbose firewall status
 show ARG                        show firewall report
 version                         display version information

Application profile commands:
 app list                        list application profiles
 app info PROFILE                show information on PROFILE
 app update PROFILE              update PROFILE
 app default ARG                 set default application policy

確認

現在有効な接続状況確認

netstat -antu

Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN

MySQLとMongoDBの設定ファイル内でローカル接続のみを有効に設定しているため
127.0.0.1の制限がある。

番号付き一覧表示

iptables -L --line-numbers

TCPダンプで確認

sudo tcpdump -s 1600 -X -i eth0 src port 80
sudo tcpdump -s 1600 -X -i eth0 dst port 80

phpを利用したMapReduce実行

サンプルデータの配置

サンプルデータの作成

echo Hello World Bye World > file01
echo Hello Hadoop Goodbye Hadoop > file02

ls
file01  file02

HDFS上にinputディレクトリを作成

sudo -u hdfs hadoop fs -mkdir /user/hdfs/input

HDFS上にサンプルデータを配置

sudo -u hdfs hadoop fs -put file01 /user/hdfs/input/file01
sudo -u hdfs hadoop fs -put file02 /user/hdfs/input/file02

sudo -u hdfs hadoop fs -cat /user/hdfs/input/file01
Hello World Bye World

sudo -u hdfs hadoop fs -cat /user/hdfs/input/file02
echo Hello Hadoop Goodbye Hadoop

map処理の作成

vi map.php

[php]
<?php
while (($row = fgetcsv(STDIN, 1024, " ")) !== FALSE) {
foreach ($row as $word) {
if ($word !== ”) {
echo "${word}\t1\n";
}
}
}
?>
[/php]

map.phpローカルテスト

cat file01 file02 | php ./map.php

Hello   1
World   1
Bye     1
World   1
Hello   1
Hadoop  1
Goodbye 1
Hadoop  1

キーと値のペアが出力される。
値は文字の出現回数とし1を固定。

map処理と同じ状態で出力

cat file01 file02 | php ./map.php | sort

Bye     1
Goodbye 1
Hadoop  1
Hadoop  1
Hello   1
Hello   1
World   1
World   1

キーを元にソートされるためsortコマンドへ送る。

reduce処理の作成

vi reduce.php

[php]
<?php
$count = array();
while ((list($key, $value) = fgetcsv(STDIN, 1024, "\t")) !== FALSE) {
$count[$key] = empty($count[$key]) ? 1: $count[$key] + 1;
}

foreach ( $count as $key => $value ) {
echo "${key}\t${value}\n";
}
?>
[/php]

reduce.phpローカルテスト

cat file01 file02 | php ./map.php | sort | php ./reduce.php

Bye     1
Goodbye 1
Hadoop  2
Hello   2
World   2

キーと値のペアを配列にマップしてカウント

Hadoop Streamingの実行

ファイルの配信

scp -r /home/mapred hdfs@slaves000:/home/
scp -r /home/mapred hdfs@slaves001:/home/
scp -r /home/mapred hdfs@slaves002:/home/

ストリーミングモジュールを利用してmapreduceの実行

sudo su hdfs

/usr/lib/hadoop-0.20/bin/hadoop \
  jar /usr/lib/hadoop-0.20/contrib/streaming/hadoop-streaming-0.20.2-cdh3u5.jar \
  -input /user/hdfs/input \
  -output /user/hdfs/output \
  -mapper '/usr/bin/php /home/mapred/map.php' \
  -reducer '/usr/bin/php /home/mapred/reduce.php'

/user/hdfs/outputが既に存在しているとエラーになります。

結果の確認

sudo -s hdfs hadoop fs -ls /user/hdfs/output
Found 3 items
-rw-r--r--   1 hdfs supergroup          0 2012-12-02 04:23 /user/hdfs/output/_SUCCESS
drwxr-xr-x   - hdfs supergroup          0 2012-12-02 04:24 /user/hdfs/output/_logs
-rw-r--r--   1 hdfs supergroup         41 2012-12-02 04:25 /user/hdfs/output/part-00000

sudo -u hdfs hadoop fs -cat /user/hdfs/output/part-00000

Bye     1
Goodbye 1
Hadoop  2
Hello   2
World   2

HDFS上の/user/hdfs/output/に結果が保存される。

HDFSマスター1つ HDFSスレーブ3つの構成

DataNodeとTaskTrackerのインストール

sudo aptitude -y install hadoop-0.20 hadoop-0.20-datanode hadoop-0.20-tasktracker

設定ファイルの同期

全スレーブにSSH公開鍵の登録

ssh root@slaves000 mkdir /usr/lib/hadoop-0.20/.ssh
scp /usr/lib/hadoop-0.20/.ssh/authorized_keys root@slaves000:/usr/lib/hadoop-0.20/.ssh/
ssh root@slaves000 chown -R hdfs:hdfs /usr/lib/hadoop-0.20/.ssh/
ssh root@slaves000 mod 0600 /usr/lib/hadoop-0.20/.ssh/authorized_keys 

設定ファイルの配信

rsync -av /etc/hadoop-0.20/conf hdfs@slaves000:/etc/hadoop-0.20/conf
rsync -av /etc/hadoop-0.20/conf hdfs@slaves001:/etc/hadoop-0.20/conf
rsync -av /etc/hadoop-0.20/conf hdfs@slaves002:/etc/hadoop-0.20/conf

予めhdfs権限で上書きできるように設定する必要があります。

/usr/lib/hadoop-0.20/conf/以下はマスターと同じ設定にします。

設定ファイルの編集

hostsの設定

sudo vi /etc/hosts
192.168.196.125   masters000.sheeps.me    masters000
192.168.196.126   slaves000.sheeps.me     slaves000
192.168.196.127   slaves001.sheeps.me     slaves001
192.168.196.128   slaves002.sheeps.me     slaves002

初期化

cacheディレクトリの設定

sudo mkdir -p /var/lib/hadoop-0.20/cache
sudo chown -R hdfs:hadoop /var/lib/hadoop-0.20

sudo chmod 0777 /var/lib/hadoop-0.20/cache

公開鍵の登録

sudo su hdfs
cd
mkdir ./.ssh
echo ssh-rsa ************** >> ./.ssh/authorized_keys
chmod 0600 ./.ssh/authorized_keys

サービスの起動

DataNodeとTaskTrackerの起動

sudo service hadoop-0.20-datanode start
sudo service hadoop-0.20-tasktracker start

HDFSマスターのインストールへ

HDFSマスター1つ HDFSスレーブ3つの構成

NamenodeとJobtrackerのインストール

sudo aptitude -y install hadoop-0.20 hadoop-0.20-namenode hadoop-0.20-jobtracker

設定ファイルの編集

core-site.xmlの設定

sudo vi /etc/hadoop-0.20/conf/core-site.xml

[sourcecode language=”plain”]
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!– Put site-specific property overrides in this file. –>

<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://masters000:8020</value>
</property>

<property>
<name>hadoop.tmp.dir</name>
<value>/var/lib/hadoop-0.20/cache/${user.name}</value>
</property>

<!– OOZIE proxy user setting –>
<property>
<name>hadoop.proxyuser.oozie.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.oozie.groups</name>
<value>*</value>
</property>
</configuration>
[/sourcecode]

hdfs-site.xmlの設定

sudo vi /etc/hadoop-0.20/conf/hdfs-site.xml

[sourcecode language=”plain”]
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!– Put site-specific property overrides in this file. –>

<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<!– Immediately exit safemode as soon as one DataNode checks in.
On a multi-node cluster, these configurations must be removed. –>
<property>
<name>dfs.safemode.extension</name>
<value>0</value>
</property>
<property>
<name>dfs.safemode.min.datanodes</name>
<value>1</value>
</property>
<property>
<!– specify this so that running ‘hadoop namenode -format’ formats the right dir –>
<name>dfs.name.dir</name>
<value>/var/lib/hadoop-0.20/cache/hadoop/dfs/name</value>
</property>

<!– Enable Hue Plugins –>
<property>
<name>dfs.namenode.plugins</name>
<value>org.apache.hadoop.thriftfs.NamenodePlugin</value>
<description>Comma-separated list of namenode plug-ins to be activated.
</description>
</property>
<property>
<name>dfs.datanode.plugins</name>
<value>org.apache.hadoop.thriftfs.DatanodePlugin</value>
<description>Comma-separated list of datanode plug-ins to be activated.
</description>
</property>
<property>
<name>dfs.thrift.address</name>
<value>0.0.0.0:10090</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.support.append</name>
<value>true</value>
</property>
<property>
<name>dfs.support.broken.append</name>
<value>true</value>
</property>
</configuration>
[/sourcecode]

WebHDFSを有効にappendをサポートするように追加しています。
fluentdなどでは必要になるプラグインになります。

mapred-site.xmlの設定

sudo vi /etc/hadoop-0.20/conf/mapred-site.xml

[sourcecode language=”plain”]
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!– Put site-specific property overrides in this file. –>

<configuration>
<property>
<name>mapred.job.tracker</name>
<value>masters000:8021</value>
</property>

<!– Enable Hue plugins –>
<property>
<name>mapred.jobtracker.plugins</name>
<value>org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin</value>
<description>Comma-separated list of jobtracker plug-ins to be activated.
</description>
</property>
<property>
<name>jobtracker.thrift.address</name>
<value>0.0.0.0:9290</value>
</property>
</configuration>
[/sourcecode]

hadoop-env.shの設定

sudo vi /etc/hadoop-0.20/conf/hadoop-env.sh

[sourcecode language=”plain”]
# Set Hadoop-specific environment variables here.

# The only required environment variable is JAVA_HOME. All others are
# optional. When running a distributed configuration it is best to
# set JAVA_HOME in this file, so that it is correctly defined on
# remote nodes.

# The java implementation to use. Required.
export JAVA_HOME=/usr/lib/jvm/java-6-sun

# Extra Java CLASSPATH elements. Optional.
# export HADOOP_CLASSPATH="<extra_entries>:$HADOOP_CLASSPATH"

# The maximum amount of heap to use, in MB. Default is 1000.
# export HADOOP_HEAPSIZE=2000

# Extra Java runtime options. Empty by default.
# if [ "$HADOOP_OPTS" == "" ]; then export HADOOP_OPTS=-server; else HADOOP_OPTS+=" -server"; fi

# Command specific options appended to HADOOP_OPTS when specified
export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_NAMENODE_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_SECONDARYNAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_DATANODE_OPTS"
export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_BALANCER_OPTS"
export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_JOBTRACKER_OPTS"
# export HADOOP_TASKTRACKER_OPTS=
# The following applies to multiple commands (fs, dfs, fsck, distcp etc)
# export HADOOP_CLIENT_OPTS

# Extra ssh options. Empty by default.
# export HADOOP_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=HADOOP_CONF_DIR"

# Where log files are stored. $HADOOP_HOME/logs by default.
# export HADOOP_LOG_DIR=${HADOOP_HOME}/logs

# File naming remote slave hosts. $HADOOP_HOME/conf/slaves by default.
# export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves

# host:path where hadoop code should be rsync’d from. Unset by default.
# export HADOOP_MASTER=master:/home/$USER/src/hadoop

# Seconds to sleep between slave commands. Unset by default. This
# can be useful in large clusters, where, e.g., slave rsyncs can
# otherwise arrive faster than the master can service them.
# export HADOOP_SLAVE_SLEEP=0.1

# The directory where pid files are stored. /tmp by default.
# NOTE: this should be set to a directory that can only be written to by
# the users that are going to run the hadoop daemons. Otherwise there is
# the potential for a symlink attack.
# export HADOOP_PID_DIR=/var/hadoop/pids

# A string representing this instance of hadoop. $USER by default.
# export HADOOP_IDENT_STRING=$USER

# The scheduling priority for daemon processes. See ‘man nice’.
# export HADOOP_NICENESS=10
[/sourcecode]
JAVA_HOME=/usr/lib/jvm/java-6-sunだけ変更すれば基本動くようです。

mastersの設定

sudo vi /etc/hadoop-0.20/conf/masters
masters000

slavesの設定

sudo vi /etc/hadoop-0.20/conf/slaves
slaves000
slaves001
slaves002

初期化

cacheディレクトリの設定

sudo mkdir -p /var/lib/hadoop-0.20/cache
sudo chown -R hdfs:hadoop /var/lib/hadoop-0.20

sudo chmod 0777 /var/lib/hadoop-0.20/cache

SSH公開鍵の登録

sudo su hdfs
ssh-keygen -t rsa -P "" 
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys 

各サーバへパスフレーズなしでsshログイン出来るように設定します。

NameNodeのフォーマット

sudo -u hdfs hadoop namenode -format

設定ファイルの編集

hostsの設定

sudo vi /etc/hosts
127.0.0.1         localhost
127.0.0.1         masters000.sheeps.me    masters000
192.168.196.125   masters000.sheeps.me    masters000
192.168.196.126   slaves000.sheeps.me     slaves000
192.168.196.127   slaves001.sheeps.me     slaves001
192.168.196.128   slaves002.sheeps.me     slaves002

ドメイン設定をしっかりやらないと上手くいかない場合があるようです。

サービスの起動

NameNodeとJobTrackerの起動

sudo service hadoop-0.20-namenode start
sudo service hadoop-0.20-jobtracker start

起動時のclasspath

/usr/lib/hadoop-0.20/conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/lib/hadoop-0.20:/usr/lib/hadoop-0.20/hadoop-core-0.20.2-cdh3u5.jar:/usr/lib/hadoop-0.20/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop-0.20/lib/asm-3.2.jar:/usr/lib/hadoop-0.20/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop-0.20/lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar:/usr/lib/hadoop-0.20/lib/commons-codec-1.4.jar:/usr/lib/hadoop-0.20/lib/commons-daemon-1.0.1.jar:/usr/lib/hadoop-0.20/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop-0.20/lib/commons-io-2.1.jar:/usr/lib/hadoop-0.20/lib/commons-lang-2.4.jar:/usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-net-3.1.jar:/usr/lib/hadoop-0.20/lib/core-3.1.1.jar:/usr/lib/hadoop-0.20/lib/guava-r09-jarjar.jar:/usr/lib/hadoop-0.20/lib/hadoop-fairscheduler-0.20.2-cdh3u5.jar:/usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20/lib/jackson-core-asl-1.5.2.jar:/usr/lib/hadoop-0.20/lib/jackson-mapper-asl-1.5.2.jar:/usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jersey-core-1.8.jar:/usr/lib/hadoop-0.20/lib/jersey-json-1.8.jar:/usr/lib/hadoop-0.20/lib/jersey-server-1.8.jar:/usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20/lib/jetty-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jetty-servlet-tester-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jetty-util-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jsch-0.1.42.jar:/usr/lib/hadoop-0.20/lib/junit-4.5.jar:/usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar:/usr/lib/hadoop-0.20/lib/mockito-all-1.8.2.jar:/usr/lib/hadoop-0.20/lib/mysql-connector-java-5.1.22-bin.jar:/usr/lib/hadoop-0.20/lib/oro-2.0.8.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-20081211.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar:/usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar:/usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar

HDFSスレーブのインストールへ

コンパイルからのインストール

必要モジュールなどをインストール

sudo aptitude install gcc g++ make cmake
sudo aptitude install bison libncurses5-dev libncursesw5-dev

MySQLユーザー/グループの追加

sudo groupadd mysql
sudo useradd -r -g mysql mysql

ソースのダウンロード

sudo wget http://ftp.iij.ad.jp/pub/db/mysql/Downloads/MySQL-5.5/mysql-5.5.28.tar.gz
sudo tar xzvf mysql-5.5.28.tar.gz
cd mysql-5.5.28

CMAKEを利用したコンパイル

sudo cmake . \
 -DCMAKE_INSTALL_PREFIX=/usr/local/mysql \
 -DDEFAULT_CHARSET=utf8 \
 -DDEFAULT_COLLATION=utf8_general_ci \
 -DENABLED_LOCAL_INFILE=true \
 -DWITH_INNOBASE_STORAGE_ENGINE=1 \
 -DWITH_EXTRA_CHARSETS=all \
 -DWITH_READLINE=ON

sudo make
sudo make install

ディレクトリの作成と権限変更

sudo chown -R mysql:mysql /usr/local/mysql/

sudo mkdir /var/log/mysql/
sudo touch /var/log/mysql/error.log
sudo touch /var/log/mysql/query.log
sudo touch /var/log/mysql/slow.log
sudo chown -R mysql:mysql /var/log/mysql/

MySQLの設定

sudo vi /etc/my.cnf
[client]
port            = 3306
socket          = /var/run/mysqld/mysqld.sock
[mysqld]
port                = 3306
socket              = /var/run/mysqld/mysqld.sock
basedir             = /usr/local/mysql
datadir             = /var/lib/mysql
tmpdir              = /tmp
log_error           = /var/log/mysql/error.log
slow-query-log      = ON
slow-query-log-file = /var/log/mysql/slow.log
skip-external-locking
key_buffer_size     = 256M
max_allowed_packet  = 1M
table_open_cache    = 256
sort_buffer_size    = 1M
read_buffer_size    = 1M
read_rnd_buffer_size = 4M
myisam_sort_buffer_size = 64M
thread_cache_size = 8
query_cache_size= 16M
thread_concurrency = 8

log-bin=mysql-bin
binlog_format=mixed
server-id       = 1

[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash

[myisamchk]
key_buffer_size = 128M
sort_buffer_size = 128M
read_buffer = 2M
write_buffer = 2M

[mysqlhotcopy]
interactive-timeout

[mysqld_safe]
basedir=/usr/local/mysql

データベースの初期化

sudo mkdir /var/lib/mysql
sudo chown mysql:mysql /var/lib/mysql

cd /usr/local/mysql
sudo ./scripts/mysql_install_db --user=mysql --basedir=/usr/local/mysql --datadir=/var/lib/mysql

全権限ユーザーの作成

mysql> CREATE USER 'sheeps'@'%' IDENTIFIED BY  '***';
mysql> GRANT ALL PRIVILEGES ON * . * TO  'sheeps'@'%' IDENTIFIED BY  '***' WITH GRANT OPTION MAX_QUERIES_PER_HOUR 0 MAX_CONNECTIONS_PER_HOUR 0 MAX_UPDATES_PER_HOUR 0 MAX_USER_CONNECTIONS 0 ;

ユーザーと同一のDBに全権限を与える

mysql> CREATE USER 'wordpress'@'localhost' IDENTIFIED BY  '***';
mysql> GRANT USAGE ON * . * TO  'wordpress'@'localhost' IDENTIFIED BY  '***' WITH MAX_QUERIES_PER_HOUR 0 MAX_CONNECTIONS_PER_HOUR 0 MAX_UPDATES_PER_HOUR 0 MAX_USER_CONNECTIONS 0 ;
mysql> CREATE DATABASE IF NOT EXISTS  `wordpress` ;
mysql> GRANT ALL PRIVILEGES ON  `wordpress` . * TO  'wordpress'@'localhost';

SpiderやMroongaなどコンパイルが必要になる時いつも忘れるので

nginxとphp5の設定など

nginxのインストール

sudo aptitude install -y nginx

PHPインストール

sudo aptitude -y install php5 php5-common php5-curl php5-dev php5-fpm
sudo aptitude -y install php5-gd php5-imagick php5-intl php5-json php5-mcrypt
sudo aptitude -y install php5-memcache php5-mongo php5-mysql php5-xdebug

nginxのサイト設定

sudo vi /etc/nginx/sites-available/sheeps.me
server {
    server_name *.sheeps.me;

    root /var/www/$host/www;
    index index.html index.php;

    location / {
        try_files $uri $uri/ /index.php;
    }

    # API
    location /api/ {
        try_files $uri /api.php?$args;
    }

    location ~ \.php$ {
        include /etc/nginx/fastcgi_params;
        fastcgi_pass    unix:/var/run/php5-fpm.sock;
        fastcgi_index   index.php;
        fastcgi_param   SCRIPT_FILENAME /var/www/$host/www$fastcgi_script_name;
    }

    # serve static files directly
    location ~* ^.+.(jpg|jpeg|gif|css|png|js|ico|html|xml|bmp|swf|wmv)$ {
        root /var/www/$host/www;
        access_log off;
        expires 30d;
    }

    location ~ /\.ht {
        deny  all;
    }
}

DNS Aレコードの設定

a   *.sheeps.me     163.43.161.128

php-fpmの設定

sudo vi /etc/php5/fpm/pool.d/www.conf
[www]
listen = /var/run/php5-fpm.sock
user = www-data
group = www-data
pm = dynamic
pm.max_children = 10
pm.start_servers = 4
pm.min_spare_servers = 2
pm.max_spare_servers = 6
request_slowlog_timeout = 1s
slowlog = /var/log/php5/$pool.log.slow
chdir = /

5.4.6-1ubuntu1.1ではaptでインストールした場合にデフォルトで
unixドメインで動作するように設定されるようです。

ファイルオープン上限の確認

ulimit -n

1024

アクセス数の多いサイトなどでは、/etc/security/limits.confなどで設定

XHProfのインストール

git clone https://github.com/facebook/xhprof.git
cd ./xhprof/extension/
phpize
./configure
make
sudo make install
sudo vi /etc/php5/mods-available/xhprof.ini


; configuration for php XHProf module
; priority=20
extension=xhprof.so

Ketamaのインストール

git clone https://github.com/RJ/ketama.git
cd ./ketama/libketama/
make
sudo make install

cd ../php_ketama/
phpize
./configure
make
sudo make install
sudo vi /etc/php5/mods-available/ketama.ini


; configuration for php Ketama module
; priority=20
extension=ketama.so

Ubuntu-serverに最初に入れたソフト

コンパイルとかで使うもの

sudo aptitude install -y gcc g++ make cmake curl pkg-config zlibc git ssh
sudo aptitude install -y bison libncurses5-dev libncursesw5-dev sysv-rc-conf

パッケージのインストール先

dpkg -L [パッケージ名]
dpkg -L mongo

ユーザー追加

sudo useradd -d /home/adminuser -m -s /bin/bash adminuser
sudo locale-gen ja_JP.UTF-8
sudo dpkg-reconfigure locales
Generating locales...
  en_US.UTF-8... done
  ja_JP.UTF-8... up-to-date
Generation complete.

言語追加

sudo aptitude -y install locales

sudo設定

visudo
adminuser  ALL=(ALL:ALL) NOPASSWD:ALL

sun-javaのインストール

Ubuntuにsun-javaをインストールするためにflexion.orgのスクリプトを実行します。

flexion.org

wget https://raw.github.com/flexiondotorg/oab-java6/master/oab-java.sh -O oab-java.sh
chmod +x oab-java.sh
sudo ./oab-java.sh

インストール

sudo aptitude -y install sun-java6-jdk ant

環境変数設定

export JAVA_HOME=/usr/lib/jvm/java-6-sun

aptでhadoopをインストールする準備

HadoopをaptからインストールするためにClouderaのリポジトリを追加します。

Clouderaのリスト作成

sudo vi /etc/apt/sources.list.d/cloudera.list

—- cloudera.list —-

deb http://archive.cloudera.com/debian lucid-cdh3 contrib

公開鍵の登録

curl -s http://archive.cloudera.com/debian/archive.key | sudo apt-key add -

パッケージ情報の更新

sudo aptitude update

Swap領域の確保

1GBを確保

sudo mkdir /var/swap
sudo dd if=/dev/zero of=/var/swap/swapfile bs=1024 count=1024000
sudo mkswap /var/swap/swapfile
sudo swapon /var/swap/swapfile

free

             total       used       free     shared    buffers     cached
Mem:        604376     536116      68260          0      31992     411644
-/+ buffers/cache:      92480     511896
Swap:      1023996         84    1023912

fstabへ登録

sudo vi /etc/fstab

/var/swap/swapfile      swap     swap   defaults        0 0

再起動時にSwap領域を有効にするために設定します。