Skip to content

Latest commit

 

History

History
265 lines (233 loc) · 8.3 KB

Hive安装部署.md

File metadata and controls

265 lines (233 loc) · 8.3 KB
Raw

一、下载

http://archive.apache.org/dist/hive/

二、解压

[root@hadoop1 hive-3.1.2]# tar -zxvf apache-hive-3.1.2-bin.tar.gz -C /usr/bigdata/
[root@hadoop1 hive-3.1.2]# cd /usr/bigdata/
[root@hadoop1 hive-3.1.2]# mv apache-hive-3.1.2-bin/ hive-3.1.2
[root@hadoop1 hive-3.1.2]# cd hive-3.1.2/

三、修改配置文件名

修改 /usr/bigdata/hive-3.1.2/conf 目录下的 hive-env.sh.template 名称为 hive-env.sh

[root@hadoop1 conf]# mv hive-env.sh.template hive-env.sh

四、配置 hive-env.sh文件

① 配置HADOOP_HOME路径

[root@hadoop1 conf]# vim hive-env.sh
export HADOOP_HOME=/usr/bigdata/hadoop-3.1.1

② 配置HIVE_CONF_DIR路径

[root@hadoop1 conf]# vim hive-env.sh
export HIVE_CONF_DIR=/usr/bigdata/hive-3.1.2

五、配置 Hadoop 集群

5.1 修改 core-site.xml 文件

<property>
    <name>hadoop.proxyuser.xxx.hosts</name>
    <value>*</value>
</property>
<property>
    <name>hadoop.proxyuser.xxx.groups</name>
    <value>*</value>
</property>

其中"xxx"是连接beeline的用户, 将"xxx"替换成自己的用户名即可

"*"表示可通过超级代理"xxx"操作hadoop的用户、用户组和主机

5.2 必须启动 hdfs 和 yarn

[root@hadoop1 conf]# start-all.sh

5.3 在 HDFS 上创建文件夹并赋予权限

[root@hadoop1 conf]# hadoop fs -mkdir /tmp
[root@hadoop1 conf]# hadoop fs -mkdir -p /user/hive/warehouse
[root@hadoop1 conf]# hadoop fs -chmod g+w /tmp
[root@hadoop1 conf]# hadoop fs -chmod g+w /user/hive/warehouse

六、安装 MariaDB, 配置权限

6.1 安装

CentOS7安装MariaDB

6.2 配置

① 连接到mysql

[root@hadoop1 conf]# mysql -uroot -p
输入密码

② 查询user表

MariaDB [(none)]> use mysql;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [mysql]> select User, Host, Password from user;
+------+-----------+-------------------------------------------+
| User | Host      | Password                                  |
+------+-----------+-------------------------------------------+
| root | localhost | *23AE809DDACAF96AF0FD78ED04B6A265E05AA257 |
| root | hadoop1   | *23AE809DDACAF96AF0FD78ED04B6A265E05AA257 |
| root | 127.0.0.1 | *23AE809DDACAF96AF0FD78ED04B6A265E05AA257 |
| root | ::1       | *23AE809DDACAF96AF0FD78ED04B6A265E05AA257 |
+------+-----------+-------------------------------------------+
4 rows in set (0.00 sec)

③ 修改user表，把Host表内容修改为%

MariaDB [mysql]> update user set host='%' where host='localhost';
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

④ 删除root用户的其他host

MariaDB [mysql]> delete from user where Host='hadoop1';
Query OK, 1 row affected (0.00 sec)

MariaDB [mysql]> delete from user where Host='127.0.0.1';
Query OK, 1 row affected (0.00 sec)

MariaDB [mysql]> delete from user where Host='::1';
Query OK, 1 row affected (0.00 sec)

⑤ 刷新

MariaDB [mysql]> flush privileges;
Query OK, 0 rows affected (0.00 sec)

MariaDB [mysql]> quit
Bye

七、Hive 元数据配置到 MariaDB

7.1 下载 MariaDB 驱动包

下载地址: https://downloads.mariadb.com/Connectors/java/connector-java-2.1.2/mariadb-java-client-2.1.2.jar

7.2 解压后 将 驱动包 拷贝到 $HIVE_HOME/lib 目录下

[root@hadoop1 opt]# mv mariadb-java-client-2.1.2.jar /usr/bigdata/hive-3.1.2/lib/

7.3 配置 Metastore 到 MariaDB

① 在 $HIVE_HOME/conf 目录下创建 hive-site.xml

[root@hadoop1 conf]# vim hive-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
        <property>
          <name>javax.jdo.option.ConnectionURL</name>
          <value>jdbc:mysql://hadoop1:3306/metastore?createDatabaseIfNotExist=true</value>
          <description>JDBC connect string for a JDBC metastore</description>
        </property>

        <property>
          <name>javax.jdo.option.ConnectionDriverName</name>
          <value>org.mariadb.jdbc.Driver</value>
          <description>Driver class name for a JDBC metastore</description>
        </property>

        <property>
          <name>javax.jdo.option.ConnectionUserName</name>
          <value>root</value>
          <description>username to use against metastore database</description>
        </property>

        <property>
          <name>javax.jdo.option.ConnectionPassword</name>
          <value>123</value>
          <description>password to use against metastore database</description>
        </property>
</configuration>

② 初始化hive元数据库

[root@hadoop1 bin]# schematool -dbType mysql -initSchema

③ 查看 MariaDB, metastore 数据已经生成了

[root@hadoop1 opt]# mysql -uroot -p
Enter password: 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 7
Server version: 5.5.65-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| metastore          |
| mysql              |
| performance_schema |
+--------------------+
4 rows in set (0.00 sec)

八、连接到hive

[root@hadoop1 bin]# hive
hive> show databases;
OK
default
Time taken: 0.591 seconds, Fetched: 1 row(s)

hive> set hive.exec.mode.local.auto=true;
hive> set mapreduce.map.memory.mb=1025;
hive> set mapreduce.reduce.memory.mb=1025; 
hive> set mapreduce.job.reduces=30;
hive> create table test(id int, name string);
hive> show tables;
OK
test
Time taken: 0.027 seconds, Fetched: 1 row(s)

hive> insert into test values(1, 'kino');
hive> select * from test;
OK
1	kino
Time taken: 0.099 seconds, Fetched: 1 row(s)

九、HiveHDBC访问

9.1 启动 hiveserver2

[root@hadoop1 lib]# hiveserver2

9.2 连接

[root@hadoop1 lib]# beeline -u "jdbc:hive2://hadop1:10000/default" -n hdfs

十、Hive On Spark

10.1 upload spark jar

下载 spark 原生 安装包

$ tar -zxvf spark-3.0.0-bin-without-hadoop.tgz 
$ cd spark-3.0.0-bin-without-hadoo
$ hdfs dfs -mkdir /spark-jars
$ hdfs dfs -put jars/* /spark-jars

10.2 修改 hive-site.xml 文件

<!--Spark依赖位置（注意：端口号8020必须和namenode的端口号一致）-->
<property>
    <name>spark.yarn.jars</name>
    <value>hdfs://hadoop1:8020/spark-jars/*</value>
</property>
<!--Hive执行引擎-->
<property>
    <name>hive.execution.engine</name>
    <value>spark</value>
</property>

10.3 测试

[root@hadoop1 lib]# beeline -u "jdbc:hive2://hadop1:10000/default" -n hdfs