Warning: Redis::connect(): connect() failed: Connection refused in /var/www/admclub.com/wp-content/object-cache.php on line 418
hadoop分布式云服务器安装 | _站长俱乐部站长俱乐部

hadoop分布式云服务器安装

/etc/sysconfig/i18n
en_US.UTF-8

以下是原始笔记,有时间再做整理。

==================================
设置MASTER到SLAVE无密码登录

master:
cd ~
ssh-keygen -t  rsa

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

vi /etc/ssh/sshd_config
RSAAuthentication yes # 启用 RSA 认证

PubkeyAuthentication yes # 启用公钥私钥配对认证方式

AuthorizedKeysFile .ssh/authorized_keys # 公钥文件路径(和上面生成的文件同)

设置完之后记得重启SSH服务,才能使刚才设置有效。

ssh localhost[机器名]
输入yes,如果可以登录就说明成功
然后退出exit

scp ~/.ssh/id_rsa.pub root@slave1:~/
scp ~/.ssh/id_rsa.pub root@slave2:~/

切换到slave1:
chmod 700 ~/.ssh

cat ~/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
rm ~/id_rsa.pub

=================================

=====================================
SLAVE无密码登录MASTER
slave1
vi /etc/ssh/sshd_config
RSAAuthentication yes # 启用 RSA 认证

PubkeyAuthentication yes # 启用公钥私钥配对认证方式

AuthorizedKeysFile .ssh/authorized_keys # 公钥文件路径(和上面生成的文件同)

设置完之后记得重启SSH服务,才能使刚才设置有效。

ssh localhost[机器名]
输入yes,如果可以登录就说明成功
然后退出exit

ssh-keygen -t  rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
scp ~/.ssh/id_rsa.pub root@master:~/
切换到MASTER
cat ~/id_rsa.pub >> ~/.ssh/authorized_keys
rm ~/id_rsa.pub
=========================================
yum install java-1.6.0-openjdk-devel -y
vi /etc/profile
加入:

#set java environment
export JAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64
export JRE_HOME=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export PATH=/usr/local/hadoop/bin:$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH

chmod +x /etc/profile
source /etc/profile

mkdir /usr/local/hadoop -p
wget http://mirrors.tuna.tsinghua.edu.cn/Apache/hadoop/common/stable/hadoop-1.0.4.tar.gz

wget http://www.us.apache.org/dist/hadoop/common/hadoop-1.1.2/hadoop-1.1.2.tar.gz
tar xzvf hadoop-1.1.2.tar.gz
mv hadoop-1.1.2 /usr/local/hadoop

cd /usr/local/hadoop
mv hadoop-1.1.2/* .
yum install rsync -y

cd src/hdfs
cp hdfs-default.xml /usr/local/hadoop/conf/hdfs-site.xml

cd ../mapred
cp mapred-default.xml /usr/local/hadoop/conf/mapred-site.xml

cd ../core
cp core-default.xml ../../conf/core-site.xml

cd /usr/local/hadoop/conf
vi core-site.xml
内容:

<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop/tmp/hadoop_${user.name}</value>
<description>A base for other temporary directories.</description>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://hadoop-master:54310</value>
<description>The name of the default file system.  A URI whose
scheme and authority determine the FileSystem implementation.  The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class.  The uri's authority is used to
determine the host, port, etc. for a filesystem.</description>
<final>true</final>
</property>

vi hdfs-site.xml
修改Hadoop中HDFS的配置,配置的备份方式默认为3。

<configuration>

<property>

<name>dfs.replication</name>

<value>1</value>

(备注:replication 是数据副本数量,默认为3,salve少于3台就会报错)

</property>

<configuration>

vi mapred-site.xml

修改Hadoop中MapReduce的配置文件,配置的是JobTracker的地址和端口。

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
<name>mapred.job.tracker</name>
<value>hadoop-master:54311</value>
</property>
</configuration>

#cd ~
#cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

如果是在从节点上,则还要配置masters
vi masters
把localhost改成master

如果是主服务器,则要配置slaves

vi slaves
slave1
slave2

cd /usr/local/hadoop
bin/hadoop namenode -format

以下是输出:
13/05/07 11:20:05 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = t1/67.222.158.78
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 1.0.4
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1393290; compiled by 'hortonfo' on Wed Oct  3 05:13:58 UTC 2012
************************************************************/
13/05/07 11:20:06 INFO util.GSet: VM type       = 64-bit
13/05/07 11:20:06 INFO util.GSet: 2% max memory = 19.33375 MB
13/05/07 11:20:06 INFO util.GSet: capacity      = 2^21 = 2097152 entries
13/05/07 11:20:06 INFO util.GSet: recommended=2097152, actual=2097152
13/05/07 11:20:06 INFO namenode.FSNamesystem: fsOwner=root
13/05/07 11:20:06 INFO namenode.FSNamesystem: supergroup=supergroup
13/05/07 11:20:06 INFO namenode.FSNamesystem: isPermissionEnabled=true
13/05/07 11:20:06 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
13/05/07 11:20:06 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
13/05/07 11:20:06 INFO namenode.NameNode: Caching file names occuring more than 10 times
13/05/07 11:20:06 INFO common.Storage: Image file of size 110 saved in 0 seconds.
13/05/07 11:20:06 INFO common.Storage: Storage directory /usr/local/hadoop/hdfs/name has been successfully formatted.
13/05/07 11:20:06 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at t1/67.222.158.78
************************************************************/

vi /usr/local/hadoop/conf/hadoop-env.sh
# The java implementation to use.  Required.
export JAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64

bin/start-all.sh

namenode running as process 1302. Stop it first.
localhost: starting datanode, logging to /usr/local/hadoop/libexec/../logs/hadoop-root-datanode-t1.out
localhost: starting secondarynamenode, logging to /usr/local/hadoop/libexec/../logs/hadoop-root-secondarynamenode-t1.out
jobtracker running as process 1570. Stop it first.
localhost: starting tasktracker, logging to /usr/local/hadoop/libexec/../logs/hadoop-root-tasktracker-t1.out

vi /etc/hosts
+
67.222.158.78 hadoop-master
192.69.213.39   hadoop-2

2.2.3  查看集群状态
在 bin目录下执行:hadoop dfsadmin  -report

在WEB页面下查看Hadoop工作情况

http://localhost:50070
http://localhost:50030

可选
安装FUSE
yum install fuse

vi /etc/sysconfig/modules/my.modules
#!/bin/sh
modprobe fuse >/dev/null 2>&1
chmod +x /etc/sysconfig/modules/my.modules

其它

netstat -tuplen

$ $HADOOP_HOME/bin/hadoop fs -mkdir       /tmp
$ $HADOOP_HOME/bin/hadoop fs -mkdir       /user/hive/warehouse
$ $HADOOP_HOME/bin/hadoop fs -chmod g+w   /tmp
$ $HADOOP_HOME/bin/hadoop fs -chmod g+w   /user/hive/warehouse

发表评论?

0 条评论。

发表评论


注意 - 你可以用以下 HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>