- 首页
- 数据库
- CentOS7安装配置hadoop2.7.2
- 阅读:349
- 日期:2016-07-23 18:07:08
- 来源:csdn
CentOS7安装配置hadoop2.7.2
linux 虚拟机配置
系统配置:
- 虚拟机:一个master(Master.Hadoop),两个slave(Slave1.Hadoop, Slave2.Hadoop)
- 网络设置:我使用的是桥接的方式
- 内存:每个虚拟机配置4G内存
- 分区:手动配置
- 软件选择:最小安装,注意选择开发工具
- 用户设置:密码都设置为:hadoophadoop, 不创建任何用户,操作时使用root直接进行
安装完后各个虚拟机的ip配置(参考用)
主机 |
ip地址 |
master.hadoop |
192.168.200.30 |
slave1.hadoop |
192.168.200.31 |
slave2.hadoop |
192.168.200.32 |
配置本地hosts
<code class="hljs avrasm has-numbering"Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">vi /etc/hosts
// 将以下数据复制进入各个主机中
<span class="hljs-number">192.168</span><span class="hljs-number">.200.30</span> Master<span class="hljs-preprocessor">.Hadoop</span>
<span class="hljs-number">192.168</span><span class="hljs-number">.200.31</span> Slave1<span class="hljs-preprocessor">.Hadoop</span>
<span class="hljs-number">192.168</span><span class="hljs-number">.200.32</span> Slave2<span class="hljs-preprocessor">.Hadoop</span></code><ul class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li></ul>
- 使用以下指令对master主机中进行测试,可使用类似指令在slave主机测试
<code class="hljs avrasm has-numbering"Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-built_in">ping</span> Slave1<span class="hljs-preprocessor">.Hadoop</span>
<span class="hljs-built_in">ping</span> Slave2<span class="hljs-preprocessor">.Hadoop</span></code>
配置Master无密码登录所有Salve
以下在Master主机上配置
<code class="hljs ruby has-numbering"Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">ssh-keygen
/<span class="hljs-regexp">/ 会生成两个文件,放到默认的/root</span><span class="hljs-regexp">/.ssh/</span>文件夹中</code><ul class="pre-numbering"><li>1</li><li>2</li><li>3</li></ul><div class="save_code tracking-ad" data-mod="popu_249"><a target=_blank><img src="http://static.blog.csdn.net/images/save_snippets_01.png" alt="" /></a></div>
<code class="hljs avrasm has-numbering"Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">cat ~/<span class="hljs-preprocessor">.ssh</span>/id_rsa<span class="hljs-preprocessor">.pub</span> >> ~/<span class="hljs-preprocessor">.ssh</span>/authorized_keys</code><ul class="pre-numbering"><li>1</li></ul>
<code class="hljs ruby has-numbering"Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">chmod <span class="hljs-number">600</span> ~<span class="hljs-regexp">/.ssh/authorized</span>_keys</code>
<code class="hljs coffeescript has-numbering"Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">vi /etc/ssh/sshd_config
<span class="hljs-regexp">//</span> 以下三项修改成以下配置
RSAAuthentication <span class="hljs-literal">yes</span> <span class="hljs-comment"># 启用 RSA 认证</span>
PubkeyAuthentication <span class="hljs-literal">yes</span> <span class="hljs-comment"># 启用公钥私钥配对认证方式</span>
AuthorizedKeysFile .ssh/authorized_keys <span class="hljs-comment"># 公钥文件路径(和上面生成的文件同)</span></code><ul class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li></ul><div class="save_code tracking-ad" data-mod="popu_249"><a target=_blank><img src="http://static.blog.csdn.net/images/save_snippets.png" alt="" /></a></div>
<code class="hljs has-numbering"Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">service sshd restart</code>
<code class="hljs ruby has-numbering"Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">/<span class="hljs-regexp">/ scp ~/</span>.ssh/id_rsa.pub 远程用户名@远程服务器<span class="hljs-constant">IP</span><span class="hljs-symbol">:~/</span>
scp ~<span class="hljs-regexp">/.ssh/id</span>_rsa.pub root<span class="hljs-variable">@192</span>.<span class="hljs-number">168.200.31</span><span class="hljs-symbol">:~/</span>
scp ~<span class="hljs-regexp">/.ssh/id</span>_rsa.pub root<span class="hljs-variable">@192</span>.<span class="hljs-number">168.200.32</span><span class="hljs-symbol">:~/</span></code>
以下在Slave主机上配置
<code class="hljs avrasm has-numbering"Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">mkdir ~/<span class="hljs-preprocessor">.ssh</span>
// 修改权限
chmod <span class="hljs-number">700</span> ~/<span class="hljs-preprocessor">.ssh</span></code><ul class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li></ul>
<code class="hljs javascript has-numbering"Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">cat ~<span class="hljs-regexp">/id_rsa.pub >> ~/</span>.ssh/authorized_keys
<span class="hljs-comment">// 修改权限</span>
chmod <span class="hljs-number">600</span> ~<span class="hljs-regexp">/.ssh/</span>authorized_keys</code><ul class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li></ul>
<code class="hljs rust has-numbering"Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">rm –r ~/id_rsa.<span class="hljs-keyword">pub</span></code><ul class="pre-numbering"><li>1</li></ul>
在master主机下进行测试
<code class="hljs nginx has-numbering"Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-title">ssh</span> <span class="hljs-number">192.168.200.31</span>
ssh <span class="hljs-number">192.168.200.32</span>
// 如果能够分别无密码登陆slave1, slave2主机,则成功配置</code>
进行jdk, hadoop软件安装
jdk安装:
- 在/usr下创建java文件夹
- 使用以下指令从真机传入jdk-8u101-linux-x64.tar.gz文件到虚拟主机中
<code class="hljs avrasm has-numbering"Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">// 需要在真机中进入文件目录,地址为虚拟主机的ip地址
scp jdk-8u101-linux-x64.tar.gz<span class="hljs-preprocessor"></span> root<span class="hljs-localvars">@192</span><span class="hljs-number">.168.200.30</span>:/usr/java
scp jdk-8u101-linux-x64.tar.gz<span class="hljs-preprocessor"></span> root<span class="hljs-localvars">@192</span><span class="hljs-number">.168</span><span class="hljs-number">.200.31</span>:/usr/java
scp jdk-8u101-linux-x64.tar.gz<span class="hljs-preprocessor"></span> root<span class="hljs-localvars">@192</span><span class="hljs-number">.168</span><span class="hljs-number">.200.32</span>:/usr/java</code><ul class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li></ul>
<code class="hljs lasso has-numbering"Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">tar zxvf jdk-8u101-linux-x64.tar.gz
<span class="hljs-comment">// 解压后可以删除掉gz文件</span>
rm jdk-8u101-linux-x64.tar.gz</code><ul class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li></ul>
<code class="hljs has-numbering"Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">vi /etc/profile
</code><ul class="pre-numbering"><li>1</li><li>2</li></ul>
<code class="hljs bash has-numbering"Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">
// 将以下数据复制到文件底部
<span class="hljs-keyword">export</span> JAVA_HOME=/usr/java/jdk1.<span class="hljs-number">8.0</span>_101
<span class="hljs-keyword">export</span> JRE_HOME=/usr/java/jdk1.<span class="hljs-number">8.0</span>_101/jre
<span class="hljs-keyword">export</span> CLASSPATH=.:<span class="hljs-variable">$CLASSPATH</span>:<span class="hljs-variable">$JAVA_HOME</span>/lib:<span class="hljs-variable">$JRE_HOME</span>/lib
<span class="hljs-keyword">export</span> PATH=<span class="hljs-variable">$PATH</span>:<span class="hljs-variable">$JAVA_HOME</span>/bin:<span class="hljs-variable">$JRE_HOME</span>/bin</code><ul class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li></ul><div class="save_code tracking-ad" data-mod="popu_249"><a target=_blank><img src="http://static.blog.csdn.net/images/save_snippets.png" alt="" /></a></div>
<code class="hljs bash has-numbering"Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-built_in">source</span> /etc/profile</code><ul class="pre-numbering"><li>1</li></ul>
<code class="hljs lasso has-numbering"Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">java <span class="hljs-attribute">-version</span>
<span class="hljs-comment">// 如果出现以下信息,则配置成功</span>
java version <span class="hljs-string">"1.8.0_101"</span></code>
hadoop安装
- 使用以下指令将hadoop-2.7.2.tar.gz文件复制到/usr目录下
<code class="hljs ruby has-numbering"Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background-image: initial; background-attachment: initial; background-color: transparent; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;"><span>scp hadoop-<span class="hljs-number">2.7</span>.</span><span>2</span><span>.tar.gz root</span><span class="hljs-variable">@192</span><span>.</span><span class="hljs-number">168.200.30</span><span class="hljs-symbol">:/usr/</span></code><ul class="pre-numbering"><li>1</li></ul>
- 解压hadoop-2.7.2.tar.gz文件,并重命名
<code class="hljs avrasm has-numbering"Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">cd /usr
tar zxvf hadoop-<span class="hljs-number">2.7</span><span class="hljs-number">.2</span><span class="hljs-preprocessor">.tar</span><span class="hljs-preprocessor">.gz</span>
mv hadoop-<span class="hljs-number">2.7</span><span class="hljs-number">.2</span> hadoop
// 删除hadoop-<span class="hljs-number">2.7</span><span class="hljs-number">.2</span><span class="hljs-preprocessor">.tar</span><span class="hljs-preprocessor">.gz</span>文件
rm –rf hadoop-<span class="hljs-number">2.7</span><span class="hljs-number">.2</span><span class="hljs-preprocessor">.tar</span><span class="hljs-preprocessor">.gz</span></code><ul class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li></ul>
<code class="hljs bash has-numbering"Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-built_in">cd</span> /usr/hadoop
mkdir tmp</code><ul class="pre-numbering"><li>1</li><li>2</li></ul>
- 把Hadoop的安装路径添加到”/etc/profile”中
<code class="hljs bash has-numbering"Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">vi /etc/profile
// 将以下数据加入到文件末尾
<span class="hljs-keyword">export</span> HADOOP_INSTALL=/usr/hadoop
<span class="hljs-keyword">export</span> PATH=<span class="hljs-variable">${HADOOP_INSTALL}</span>/bin:<span class="hljs-variable">${HADOOP_INSTALL}</span>/sbin<span class="hljs-variable">${PATH}</span>
<span class="hljs-keyword">export</span> HADOOP_MAPRED_HOME=<span class="hljs-variable">${HADOOP_INSTALL}</span>
<span class="hljs-keyword">export</span> HADOOP_COMMON_HOME=<span class="hljs-variable">${HADOOP_INSTALL}</span>
<span class="hljs-keyword">export</span> HADOOP_HDFS_HOME=<span class="hljs-variable">${HADOOP_INSTALL}</span>
<span class="hljs-keyword">export</span> YARN_HOME=<span class="hljs-variable">${HADOOP_INSTALLL}</span>
<span class="hljs-keyword">export</span> HADOOP_COMMON_LIB_NATIVE_DIR=<span class="hljs-variable">${HADOOP_INSTALL}</span>/lib/natvie
<span class="hljs-keyword">export</span> HADOOP_OPTS=<span class="hljs-string">"-Djava.library.path=<span class="hljs-variable">${HADOOP_INSTALL}</span>/lib:<span class="hljs-variable">${HADOOP_INSTALL}</span>/lib/native"</span> </code><ul class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li></ul>
<code class="hljs bash has-numbering"Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-built_in">source</span> /etc/profile</code>
配置hadoop(先只在Master主机配置,配置完后传入两个Slave主机)
- 设置hadoop-env.sh和yarn-env.sh中的java环境变量
<code class="hljs bash has-numbering"Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-built_in">cd</span> /usr/hadoop/etc/hadoop/
vi hadoop-env.sh
// 修改JAVA_HOME
<span class="hljs-keyword">export</span> JAVA_HOME=/usr/java/jdk1.<span class="hljs-number">8.0</span>_101</code><ul class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li></ul>
<code class="hljs xml has-numbering"Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">vi core-site.xml
// 修改文件内容为以下
<span class="hljs-tag"><<span class="hljs-title">configuration</span>></span>
<span class="hljs-tag"><<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">name</span>></span>hadoop.tmp.dir<span class="hljs-tag"></<span class="hljs-title">name</span>></span>
<span class="hljs-tag"><<span class="hljs-title">value</span>></span>/usr/hadoop/tmp<span class="hljs-tag"></<span class="hljs-title">value</span>></span>
<span class="hljs-tag"><<span class="hljs-title">description</span>></span>A base for other temporary directories.<span class="hljs-tag"></<span class="hljs-title">description</span>></span>
<span class="hljs-tag"></<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">name</span>></span>fs.default.name<span class="hljs-tag"></<span class="hljs-title">name</span>></span>
<span class="hljs-tag"><<span class="hljs-title">value</span>></span>hdfs://Master.Hadoop:9000<span class="hljs-tag"></<span class="hljs-title">value</span>></span>
<span class="hljs-tag"></<span class="hljs-title">property</span>></span>
<span class="hljs-tag"></<span class="hljs-title">configuration</span>></span></code><ul class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li></ul>
<code class="hljs xml has-numbering"Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">vi hdfs-site.xml
// 修改文件内容为以下
<span class="hljs-tag"><<span class="hljs-title">configuration</span>></span>
<span class="hljs-tag"><<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">name</span>></span>dfs.namenode.name.dir<span class="hljs-tag"></<span class="hljs-title">name</span>></span>
<span class="hljs-tag"><<span class="hljs-title">value</span>></span>file:///usr/hadoop/dfs/name<span class="hljs-tag"></<span class="hljs-title">value</span>></span>
<span class="hljs-tag"></<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">name</span>></span>dfs.datanode.data.dir<span class="hljs-tag"></<span class="hljs-title">name</span>></span>
<span class="hljs-tag"><<span class="hljs-title">value</span>></span>file:///usr/hadoop/dfs/data<span class="hljs-tag"></<span class="hljs-title">value</span>></span>
<span class="hljs-tag"></<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">name</span>></span>dfs.replication<span class="hljs-tag"></<span class="hljs-title">name</span>></span>
<span class="hljs-tag"><<span class="hljs-title">value</span>></span>1<span class="hljs-tag"></<span class="hljs-title">value</span>></span>
<span class="hljs-tag"></<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">name</span>></span>dfs.nameservices<span class="hljs-tag"></<span class="hljs-title">name</span>></span>
<span class="hljs-tag"><<span class="hljs-title">value</span>></span>hadoop-cluster1<span class="hljs-tag"></<span class="hljs-title">value</span>></span>
<span class="hljs-tag"></<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">name</span>></span>dfs.namenode.secondary.http-address<span class="hljs-tag"></<span class="hljs-title">name</span>></span>
<span class="hljs-tag"><<span class="hljs-title">value</span>></span>Master.Hadoop:50090<span class="hljs-tag"></<span class="hljs-title">value</span>></span>
<span class="hljs-tag"></<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">name</span>></span>dfs.webhdfs.enabled<span class="hljs-tag"></<span class="hljs-title">name</span>></span>
<span class="hljs-tag"><<span class="hljs-title">value</span>></span>true<span class="hljs-tag"></<span class="hljs-title">value</span>></span>
<span class="hljs-tag"></<span class="hljs-title">property</span>></span>
<span class="hljs-tag"></<span class="hljs-title">configuration</span>></span>
</code><ul class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li></ul>
<code class="hljs xml has-numbering"Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"> vi mapred-site.xml
// 修改文件为以下
<span class="hljs-tag"><<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">name</span>></span>mapreduce.framework.name<span class="hljs-tag"></<span class="hljs-title">name</span>></span>
<span class="hljs-tag"><<span class="hljs-title">value</span>></span>yarn<span class="hljs-tag"></<span class="hljs-title">value</span>></span>
<span class="hljs-tag"><<span class="hljs-title">final</span>></span>true<span class="hljs-tag"></<span class="hljs-title">final</span>></span>
<span class="hljs-tag"></<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">name</span>></span>mapreduce.jobtracker.http.address<span class="hljs-tag"></<span class="hljs-title">name</span>></span>
<span class="hljs-tag"><<span class="hljs-title">value</span>></span>Master.Hadoop:50030<span class="hljs-tag"></<span class="hljs-title">value</span>></span>
<span class="hljs-tag"></<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">name</span>></span>mapreduce.jobhistory.address<span class="hljs-tag"></<span class="hljs-title">name</span>></span>
<span class="hljs-tag"><<span class="hljs-title">value</span>></span>Master.Hadoop:10020<span class="hljs-tag"></<span class="hljs-title">value</span>></span>
<span class="hljs-tag"></<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">name</span>></span>mapreduce.jobhistory.webapp.address<span class="hljs-tag"></<span class="hljs-title">name</span>></span>
<span class="hljs-tag"><<span class="hljs-title">value</span>></span>Master.Hadoop:19888<span class="hljs-tag"></<span class="hljs-title">value</span>></span>
<span class="hljs-tag"></<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">name</span>></span>mapred.job.tracker<span class="hljs-tag"></<span class="hljs-title">name</span>></span>
<span class="hljs-tag"><<span class="hljs-title">value</span>></span>http://Master.Hadoop:9001<span class="hljs-tag"></<span class="hljs-title">value</span>></span>
<span class="hljs-tag"></<span class="hljs-title">property</span>></span></code><ul class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li></ul>
<code class="hljs xml has-numbering"Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">vi yarn-site.xml
// 修改文件内容为以下
<span class="hljs-tag"><<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">name</span>></span>yarn.resourcemanager.hostname<span class="hljs-tag"></<span class="hljs-title">name</span>></span>
<span class="hljs-tag"><<span class="hljs-title">value</span>></span>Master.Hadoop<span class="hljs-tag"></<span class="hljs-title">value</span>></span>
<span class="hljs-tag"></<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">name</span>></span>yarn.nodemanager.aux-services<span class="hljs-tag"></<span class="hljs-title">name</span>></span>
<span class="hljs-tag"><<span class="hljs-title">value</span>></span>mapreduce_shuffle<span class="hljs-tag"></<span class="hljs-title">value</span>></span>
<span class="hljs-tag"></<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">name</span>></span>yarn.resourcemanager.address<span class="hljs-tag"></<span class="hljs-title">name</span>></span>
<span class="hljs-tag"><<span class="hljs-title">value</span>></span>Master.Hadoop:8032<span class="hljs-tag"></<span class="hljs-title">value</span>></span>
<span class="hljs-tag"></<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">name</span>></span>yarn.resourcemanager.scheduler.address<span class="hljs-tag"></<span class="hljs-title">name</span>></span>
<span class="hljs-tag"><<span class="hljs-title">value</span>></span>Master.Hadoop:8030<span class="hljs-tag"></<span class="hljs-title">value</span>></span>
<span class="hljs-tag"></<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">name</span>></span>yarn.resourcemanager.resource-tracker.address<span class="hljs-tag"></<span class="hljs-title">name</span>></span>
<span class="hljs-tag"><<span class="hljs-title">value</span>></span>Master.Hadoop:8031<span class="hljs-tag"></<span class="hljs-title">value</span>></span>
<span class="hljs-tag"></<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">name</span>></span>yarn.resourcemanager.admin.address<span class="hljs-tag"></<span class="hljs-title">name</span>></span>
<span class="hljs-tag"><<span class="hljs-title">value</span>></span>Master.Hadoop:8033<span class="hljs-tag"></<span class="hljs-title">value</span>></span>
<span class="hljs-tag"></<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">property</span>></span>
<span class="hljs-tag"><<span class="hljs-title">name</span>></span>yarn.resourcemanager.webapp.address<span class="hljs-tag"></<span class="hljs-title">name</span>></span>
<span class="hljs-tag"><<span class="hljs-title">value</span>></span>Master.Hadoop:8088<span class="hljs-tag"></<span class="hljs-title">value</span>></span>
<span class="hljs-tag"></<span class="hljs-title">property</span>></span>
</code><ul class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li></ul>
配置Hadoop的集群
- 将Master中配置好的hadoop传入两个Slave中
<code class="hljs ruby has-numbering"Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">scp -r /usr/hadoop root<span class="hljs-variable">@192</span>.<span class="hljs-number">168.200.31</span><span class="hljs-symbol">:/usr/</span>
scp -r /usr/hadoop root<span class="hljs-variable">@192</span>.<span class="hljs-number">168.1</span>.<span class="hljs-number">1</span><span class="hljs-symbol">:/usr/</span></code><ul class="pre-numbering"><li>1</li><li>2</li></ul>
<code class="hljs avrasm has-numbering"Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">cd /usr/hadoop/etc/hadoop
vi slaves
// 将文件内容修改为
Slave1<span class="hljs-preprocessor">.Hadoop</span>
Slave2<span class="hljs-preprocessor">.Hadoop</span>
</code><ul class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li></ul>
<code class="hljs rsl has-numbering"Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-comment">// 在Master主机上输入以下指令</span>
hadoop namenode -<span class="hljs-built_in">format</span></code><ul class="pre-numbering"><li>1</li><li>2</li></ul>
<code class="hljs http has-numbering"Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">
<span class="avrasm">// 关闭机器防火墙
<spanSource Code Pro", monospace; font-size: 14px; line-height: 20.3px; white-space: pre; background-color: rgba(128, 128, 128, 0.0470588);">systemctl stop firewalld</span>
cd /usr/hadoop/sbin
./start-all<span class="hljs-preprocessor">.sh</span>
// 更推荐的运行方式:
cd /usr/hadoop/sbin
./start-dfs<span class="hljs-preprocessor">.sh</span>
./start-yarn<span class="hljs-preprocessor">.sh</span>
应该输出以下信息:
Starting namenodes on [Master<span class="hljs-preprocessor">.Hadoop</span>]
<span class="hljs-label">Master.Hadoop:</span> starting namenode, logging to /usr/hadoop/logs/hadoop-root-namenode-localhost<span class="hljs-preprocessor">.localdomain</span><span class="hljs-preprocessor">.out</span>
<span class="hljs-label">Slave2.Hadoop:</span> starting datanode, logging to /usr/hadoop/logs/hadoop-root-datanode-Slave2<span class="hljs-preprocessor">.Hadoop</span><span class="hljs-preprocessor">.out</span>
<span class="hljs-label">Slave1.Hadoop:</span> starting datanode, logging to /usr/hadoop/logs/hadoop-root-datanode-Slave1<span class="hljs-preprocessor">.Hadoop</span><span class="hljs-preprocessor">.out</span>
starting yarn daemons
starting resourcemanager, logging to /usr/hadoop/logs/yarn-root-resourcemanager-localhost<span class="hljs-preprocessor">.localdomain</span><span class="hljs-preprocessor">.out</span>
<span class="hljs-label">Slave1.Hadoop:</span> starting nodemanager, logging to /usr/hadoop/logs/yarn-root-nodemanager-Slave1<span class="hljs-preprocessor">.Hadoop</span><span class="hljs-prepr
本页内容版权归属为原作者,如有侵犯您的权益,请通知我们删除。
最近写的项目中有用到数据库,写了不少蛋疼的sql语句,每次都是好几行代码,而且每次都是重复的没有一点技术含量的代码,虽然也有不少基于sqlite的封装,不过用起来还是感觉不够面向对象! 为了不再写重复的代码,花了几天时间,基于SQLite3简单封装了下,实现了一行代码解决增删改查等常用的功能!并没有太过高深的知识,主要用了runtime和KVC: 首先我们创建个大家都熟悉的Person类,并声明两个属性,下面将以类此展开分析 @interface Person : NSObject @property (
以前只知道MySQL可以做分库分表,支持多种数据库引擎。这次听了来自台湾的MySQL专家Ivan Tu的讲座,有两点印象深刻,一是MySQL具有丰富的高可用方案,二是MySQL也有内存数据库,即MySQL Cluster CGE。Ivan形象的将内存数据库比喻成浅浅的盘子,而传统的MySQL磁盘存储方式比喻成深深的瓶子。 另外,也探讨了MySQL作为开源数据库,和Oracle数据库的关系。MySQL并行发展企业版和社区版,企业版增加了很多企业级的特性,如安全,备份和恢复,监控,以及CGE(电信运营商版)。
CleverCode研究了一下,怎么kill掉慢的sql语句。 1 单个kill mysql show processlist; mysql kill 251; #批量kill 1)查找Lockd语句 mysql -uroot -p123456 -h 192.1.20.101 -e "show processlist" | grep -i 'Locked' locked_log.txt 2)组合kill for line in `cat locked_log.txt | awk '{print $1}'`
前段时间把项目中的JPA从EclipseLink换成Hibernate。这其中遇到了很多问题,也记录了一部分在其他的文章中。这里介绍一个UT中遇到的问题。 当时的测试是忘H2数据库中插入多项数据,然后用Query的方式删除,assert读出来的数据是null。这个测试用例在EclipseLink下面是可以通过的,但换成Hibernate之后发现数据仍然可以读出这个数据。 测试用例如下: @Test public void shouldRemovesAllFmReportFromDb () { FmRepo
实施逻辑复制软件时对在目的端数据库的字符集(排序规则)的要求 1.当目的端数据库是Oracle数据库时,务必保证目的端Oracle数据库的字符集与源头Oracle数据库的字符集保持一致. 2.当目的端数据库是MSSQLServer数据库时,务必保证目的端MSSQLServer 用户数据库的排序规则与源头MSSQLServer 用户数据库的排序规则一致. 3.当目的端数据库是MSSQLServer数据库时,务必保证目的端MSSQLServer master数据库的排序规则与源头MSSQLServer mas
前言 最近在学习使用Hive(版本0.13.1)的过程中,发现了一些坑,它们或许是Hive提倡的比关系数据库更加自由的体现(同时引来一些问题),或许是一些bug。总而言之,这些都需要使用Hive的开发人员额外注意。本文旨在列举我发现的2个通过查询语句向表中插入数据过程中的问题,希望大家注意。 数据准备 为了验证接下来出现的问题,需要先准备两张表 employees和 staged_employees ,并准备好测试数据。首先使用以下语句创建表 employees: create table employe
五、快照 前面实验说明了处理维度的扩展。本节讨论两种事实表的扩展技术。 有些用户,尤其是管理者,经常要看某个特定时间点的数据。也就是说,他们需要数据的快照。周期快照和累积快照是两种常用的事实表扩展技术。 周期快照是在一个给定的时间对事实表进行一段时期的总计。例如,一个月销售订单周期快照汇总每个月底时总的销售订单金额。 累积快照用于跟踪事实表的变化。例如,数据仓库可能需要累积(存储)销售订单从下订单的时间开始,到订单中的商品被打包、运输和到达的
1、说明:创建数据库 CREATE DATABASE database - name 2、说明:删除数据库 drop database dbname 3、说明:备份sql server --- 创建 备份数据的 device USE master EXEC sp_addumpdevice 'disk' , 'testBack' , 'c:\mssql7backup\MyNwind_1.dat' --- 开始 备份 BACKUP DATABASE pubs TO testBack 4、说明:创建新表 cre
最近在使用数据库,综合各方面原因,选择了SQL Server2008.为了对使用的工具有更加深入的了解,所以仔细地阅读了经典翻译书籍《SQL Server 2008高级程序设计》,在此将读书过程中的感想和理解记录下来,以备以后查询温习,也希望对其他使用SQL Sercer的小伙伴们有启发帮助。 对数据库系统而言,并发是最主要的问题。它用于表示两个或多个用户同时对同一个对象进行交互的情况。用户越多——更特定的说,事物越多,即在同一时刻可以合理成功完成的事情越多,并发性也就越高。 在联
oracle 11g卸载有两种方式,一种是使用 Oracle Universal Installer管理工具; 1、停用oracle服务:进入计算机管理,在服务中,找到oracle开头的所有服务,右击选择停止 2、在开始菜单中,找到Universal Installer,运行 Oracle Universal Installer,单击卸载产品 3、 在产品清单窗口中,单击全部展开,除了OraDb11g_home1外,勾选其他项目,单击删除 卸载完后,再删除:OraDb11g_home1 3、按Wind