本文以遠程模式安裝Hive2.1.1將hive的元數據放置在MySQL數據庫中。
1 安裝mysql數據庫
sudo apt-get install mysql-server
重啟mysql服務使得配置文件生效
sudo service mysql restart
創建hive專用賬戶
CREATE USER 'hive'@'%' IDENTIFIED BY '123456';
給hive賬戶授予所有權限
grant all privileges on *.* to 'hive'@'%' identified by '123456' with grant option;
刷新系統權限表,使配置生效
flush privileges;
2 解壓安裝hive
cd /usr/local
sudo tar -xvzf apache-hive-2.1.1-bin.tar.gz
sudo mv apache-hive-2.1.1-bin/ hive-2.1.1
配置系統環境變量
sudo gedit .bashrc
export HIVE_HOME=/usr/local/hive-2.1.1
exportPATH=$HIVE_HOME/bin:$HIVE_HOME/lib:$PATH
使得環境變量配置生效
source .bashrc
3 配置hive?
3.1 修改conf/hive-env.sh文件
cd /usr/local/hive-2.1.1/conf/
sudo cp hive-env.sh.template hive-env.sh
sudo chown hadoop:hadoop hive-env.sh
sudo vi hive-env.sh
HADOOP_HOME=/usr/local/hadoop-2.7.3
export HIVE_CONF_DIR=/usr/local/hive-2.1.1/conf
export HIVE_AUX_JARS_PATH=/usr/local/hive-2.1.1/lib
3.2 修改日志屬性文件配置日志存儲目錄?
修改hive-log4j2.properties
sudo cp hive-log4j2.properties.template hive-log4j2.properties
sudo chown hadoop:hadoop hive-log4j2.properties
sudo vi hive-log4j2.properties
property.hive.log.dir = /usr/local/hive-2.1.1/logs
修改llap-cli-log4j2.properties
property.hive.log.dir = /usr/local/hive-2.1.1/logs
property.hive.log.file = llap-cli.log
3.3 修改hive-site.xml配置文件,主要修改如下配置項目
<property><name>hive.exec.local.scratchdir</name><value>/usr/local/hive-2.1.1/tmp</value><description>Local scratch space for Hive jobs</description></property><property><name>hive.downloaded.resources.dir</name><value>/usr/local/hive-2.1.1/tmp/${hive.session.id}_resources</value><description>Temporary local directory for added resources in the remote file system.</description></property><property><name>hive.querylog.location</name><value>/usr/local/hive-2.1.1/logs</value><description>Location of Hive run time structured log file</description></property><property><name>hive.server2.logging.operation.log.location</name><value>/usr/local/hive-2.1.1/logs</value><description>Top level directory where operation logs are stored if logging functionality is enabled</description></property><property><name>hive.metastore.warehouse.dir</name><value>/usr/hive/warehouse</value><description>location of default database for the warehouse</description></property>
<property><name>hive.metastore.uris</name><value>thrift://192.168.80.130:9083</value><description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.</description></property>
<property><name>hive.exec.scratchdir</name><value>/tmp/hive</value><description>HDFS root scratch dir for Hive jobs which gets created with write all (733) permission. For each connecting user, an HDFS scratch dir: ${hive.exec.scratchdir}/<username> is created, with ${hive.scratch.dir.permission}.</description></property><property><name>hive.exec.local.scratchdir</name><value>/usr/local/hive-2.1.1/tmp</value><description>Local scratch space for Hive jobs</description></property><property><name>hive.downloaded.resources.dir</name><value>/usr/local/hive-2.1.1/tmp/${hive.session.id}_resources</value><description>Temporary local directory for added resources in the remote file system.</description></property><property><name>javax.jdo.option.ConnectionURL</name><value>jdbc:mysql://192.168.80.130:3306/metastore?createDatabaseIfNotExist=true&useSSL=false</value><description>JDBC connect string for a JDBC metastore.To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL.For example, jdbc:postgresql://myhost/db?ssl=true for postgres database.</description></property><property><name>javax.jdo.option.ConnectionDriverName</name><value>com.mysql.jdbc.Driver</value><description>Driver class name for a JDBC metastore</description></property><property><name>javax.jdo.option.ConnectionUserName</name><value>hive</value><description>Username to use against metastore database</description></property><property><name>javax.jdo.option.ConnectionPassword</name><value>123456</value><description>password to use against metastore database</description></property>
<property><name>hive.hwi.listen.host</name><value>0.0.0.0</value><description>This is the host address the Hive Web Interface will listen on</description></property><property><name>hive.hwi.listen.port</name><value>9999</value><description>This is the port the Hive Web Interface will listen on</description></property><property><name>hive.server2.thrift.bind.host</name><value>0.0.0.0</value><description>Bind host on which to run the HiveServer2 Thrift service.</description></property>
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
- 70
- 71
- 72
- 73
- 74
- 75
- 76
- 77
- 78
- 79
- 80
- 81
- 82
- 83
- 84
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
- 70
- 71
- 72
- 73
- 74
- 75
- 76
- 77
- 78
- 79
- 80
- 81
- 82
- 83
- 84
4 拷貝mysql連接包到hive主目錄下的lib中
sudo mv ~/下載/mysql-connector-java-5.1.40-bin.jar /usr/local/hive-1.2.1/lib/
5 配置Hive的hwi網頁訪問方式?
下載hive-2.1.1源碼包
wget http://www-us.apache.org/dist/hive/hive-2.1.1/ apache-hive-1.2.1-src.tar.gz
tar -zxvf apache-hive-2.1.1-src.tar.gz
cd apache-hive-2.1.1-src
cd hwi/web
zip hive-hwi-2.1.1.zip .
拷貝tools包
sudo cp /usr/lib/jdk1.8.0_121/lib/tools.jar /usr/local/hive-2.1.1/lib
刪除lib下的ant-1.6.5.jar,否則瀏覽hwi網頁時會顯示錯誤信息,需要刷新兩次才能看到網頁。?
6 初始化hive
schematool -dbType mysql -initSchema
7 啟動hive服務?
啟動metaStore服務
--
啟動hive web界面
--
啟動thrift2服務
--
啟動hive shell
hive
hwi訪問網址
http:
---------------------------------------------------------------------------------------------------------------------------------------------
如果發現不行:可以按照一下在來一遍。
第一步:?
下載最新的hive,直接去apache 里面找hive2.1.0下載就行。
第二步,解壓到服務器
tar zxvf apache-hive-2.0.0-bin.tar.gz mv apache-hive-2.0.0-bin /home/hivecd /home/hive
第三步,修改conf。這里只關心hadoop和hive的配置,其他JAVA HBASE的配置根據自己來
vi /etc/profile
export HADOOP_HOME=/home/hadoop/hadoop-2.7.3
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
export PATH=$PATH:/home/hadoop/hadoop-2.7.3/bin
export PATH=$PATH:/home/hadoop/hadoop-2.7.3/sbin
export HIVE_HOME=/home/hive export PATH=$HADOOP_HOME/bin:$JAVA_HOME/bin:$HBASE_HOME/bin:$HIVE_HOME/bin:$PATH
第四步,下載并設置好jdbc connector?
我這里使用了最新的mysql-connector-java-5.1.40.tar.gz?
記住,將解壓出來的jar放入hive 的lib中
cp mysql-connector-java-5.1.36-bin.jar $HIVE_HOME/lib/
第五步,配置hive-site.xml文件?
文件來源于hive-default.xml.template?
即
cp hive-default.xml.template hive-site.xml
然后找到
<name>javax.jdo.option.ConnectionURL</name>
修改其value
<value>jdbc:mysql://139.196.xxx.xxx:3306/hive?characterEncoding=UTF8&useSSL=false&createDatabaseIfNotExist=true</value>
同時,注意修改對應數據庫的賬號密碼,否則會在執行hive時出錯
<property><name>javax.jdo.option.ConnectionUserName</name><value>root</value></property><property><name>javax.jdo.option.ConnectionPassword</name><value>123456</value></property>
第六步 運行hive客戶端
cd /home/hive/bin
hive
第七步,初始化DB?
schematool -initSchema -dbType mysql
第八步,查看成功后的元數據?
可以看到對應數據庫hive中,有了各種初始的表
第九步,啟動master,node節點
啟動單機?
hive?
啟動集群?
hive -hiveconf hbase.zookeeper.quorum=slave1,slave2,slave3
————————————————————————————————————————————?
以下是可能出現的錯誤:?
一,如果執行hive時候出現報出賬號密碼的錯誤?
那么記得修改hive-site.xml中賬號密碼,參考第五步中的內容。?
二,如果出現db沒有初始化,如
Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Hive metastore database is not initialized. Please use schematool (e.g. ./schematool -initSchema -dbType ...) to create the schema. If needed, don't forget to include the option to auto-create the underlying database in your JDBC connection string (e.g. ?createDatabaseIfNotExist=true for mysql))
這種錯誤的時候,請先執行初始化DB。?
三,如果提示?
SSL相關的內容,請在配置jdbc鏈接的時候設置ssl為false?
jdbc:mysql://139.196.xxx.xxx:3306/hive?useSSL=false&createDatabaseIfNotExist=true
Wed Nov 30 14:24:50 CST 2016 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Wed Nov 30 14:24:55 CST 2016 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
四,遇到hive出錯的時候
[Fatal Error] hive-site.xml:26:5: The element type "value" must be terminated by the matching end-tag "</value>".
Exception in thread "main" java.lang.RuntimeException: org.xml.sax.SAXParseException; systemId: file:/home/hive/conf/hive-site.xml; lineNumber: 26; columnNumber: 5; The element type "value" must be terminated by the matching end-tag "</value>".
Logging initialized using configuration in jar:file:/home/hive/lib/hive-common-2.1.0.jar!/hive-log4j2.properties Async: true
Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D
這里就是配置文件Hive-site.xml中,修改system:java.io.tmpdir,指定一個系統存在的目錄即可。
這里追加兩種啟動方式,方便各位用來進行hive測試。
hive提供了四種運行hive的方式,分別是:
**Hive CLI?
HiveServer2 和 Beeline**?
HCatalog?
WebHCat (Templeton)
這里主要介紹前兩種?
第一種,hive CLI
因為hive的bin目錄已經添加了path變量, 因此, 可以直接使用hive命令啟動:?
hive
?
輸入完命令后可以,直接可以進行hive操作。
第二種HiveServer2 和 Beeline
beeline提供多用戶, 更加安全的服務, 因此beeline用得比較多.?
hiveserver2啟動時默認的地址是”localhost:10000”, 因此, 在使用beeline連接的時候, 需要使用” jdbc:hive2://localhost:10000”作為參數.?
相關的命令如下:
hiveserver2
beeline -u jdbc:hive2:
同時也可以將 Beeline和HiveServer2在同一個進程里啟動, 用于測試:?
beeline -u jdbc:hive2://?
但是,這里如果用到自定義賬號密碼,必須在配置文件hive-site.xml中進行相關配置。?
上文第五步已經進行了相關介紹,可以參照。
----------------------------------------------------------------------------------------------------------------------------------------------------------
hive啟動時報Relative path in absolute URI:${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D解決辦法
hive啟動時遇到以下錯誤:
Exception in thread "main"java.lang.RuntimeException: java.lang.IllegalArgumentException:java.net.URISyntaxException: Relative path in absolute URI:${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D
解決辦法:
在hive下創建臨時IO的tmp文件夾。然后將路徑配置到hive-site.xml的下列參數中
- <property>
- ????<name>hive.querylog.location</name>
- ????<value>/usr/local/hive/iotmp</value>
- ????<description>Location of Hive run time structured log file</description>
- ??</property>
- ??
- ??<property>
- ????<name>hive.exec.local.scratchdir</name>
- ????<value>/usr/local/hive/iotmp</value>
- ????<description>Local scratch space for Hive jobs</description>
- ??</property>
- ??
- ??<property>
- ????<name>hive.downloaded.resources.dir</name>
- ????<value>/usr/local/hive/iotmp</value>
- ????<description>Temporary local directory for added resources in the remote file system.</description>
- ??</property>
保存,重啟hive即可。
[root@master ~]# hive
Logging initialized using configuration in jar:file:/usr/local/hive/lib/hive-common-1.2.0.jar!/hive-log4j.properties
hive> show databases;
OK
default
Time taken: 3.684 seconds, Fetched: 1 row(s)
hive>
hiveserver啟動方式
1, hive??命令行模式,直接輸入/hive/bin/hive的執行程序,或者輸入 hive –service cli
? ? ? ?用于linux平臺命令行查詢,查詢語句基本跟mysql查詢語句類似
?2,?hive??web界面的啟動方式,hive –service hwi ?
? ? ? 用于通過瀏覽器來訪問hive,感覺沒多大用途
3, hive??遠程服務 (端口號10000) 啟動方式, hive –service hiveserver??&?
? ? ? 用java等程序實現通過jdbc等驅動的訪問hive就用這種起動方式了,這個是程序員最需要的方式了
? 也可以自己指定端口 hive -service hiveserver -p 50000 &? (&表示后臺運行)
輸入完這些指令后終端就在運行hiveserver了,會卡住不動。其實已經在運行了,不用擔心。