当前位置：
首页 >
【hive】——metastore的三种模式
发布时间：2023/12/9
45
豆豆
生活随笔
收集整理的这篇文章主要介绍了
【hive】——metastore的三种模式
小编觉得挺不错的,现在分享给大家,帮大家做个参考.
Hive中metastore（元数据存储）的三种方式：
- 内嵌Derby方式
- Local方式
- Remote方式
[一]、内嵌Derby方式
这个是Hive默认的启动模式，一般用于单元测试，这种存储方式有一个缺点：在同一时间只能有一个进程连接使用数据库。
hive-site.xml 中jdbc URL、驱动、用户名、密码等的配置信息如下：
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 | <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:derby:;databaseName=metastore_db;create=true</value> <description>JDBC connect string for a JDBC metastore</description> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>org.apache.derby.jdbc.EmbeddedDriver</value> <description>Driver class name for a JDBC metastore</description> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>APP</value> <description>username to use against metastore database</description> </property> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>mine</value> <description>password to use against metastore database</description> </property> <property> <name>hive.metastore.warehouse.dir</name> <value>file:///Users/micmiu/tmp/hive/warehouse</value> <description>unit test data goes in here on your local filesystem</description> </property> <!-- micmiu.com --> |
执行初始化命令：schematool -dbType derby -initSchema
查看初始化后的信息： schematool -dbType derby -info
配置完成后就可在shell中以CLI的方式访问hive 进行操作验证。
[二]、Local方式
以本地Mysql数据库为例：创建好用户：hive；database：hive。
配置文件 hive-site.xml 中jdbc URL、驱动、用户名、密码等属性值配置如下：
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 | <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://localhost/hive?createDatabaseIfNotExist=true</value> <description>JDBC connect string for a JDBC metastore</description> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> <description>Driver class name for a JDBC metastore</description> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>hive</value> <description>username to use against metastore database</description> </property> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>micmiu</value> <description>password to use against metastore database</description> </property> <property> <name>hive.metastore.warehouse.dir</name> <!-- base hdfs path --> <value>/user/hive/warehouse</value> <description>location of default database for the warehouse</description> </property> <!-- micmiu.com --> |
ps：需要把mysql的驱动包copy到目录 <HIVE_HOME>/lib 中
如果是第一次需要执行初始化命令：schematool -dbType mysql -initSchema
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | micmiu-mbp:mysql micmiu$ schematool -dbType mysql -initSchema 14/02/17 14:46:08 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive 14/02/17 14:46:08 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize 14/02/17 14:46:08 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize 14/02/17 14:46:08 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack 14/02/17 14:46:08 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node 14/02/17 14:46:08 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces 14/02/17 14:46:08 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative Metastore connection URL: jdbc:mysql://localhost/hive?createDatabaseIfNotExist=true Metastore Connection Driver : com.mysql.jdbc.Driver Metastore connection User: hive Starting metastore schema initialization to 0.12.0 Initialization script hive-schema-0.12.0.mysql.sql Initialization script completed schemaTool completeted |
查看初始化后信息 schematool -dbType mysql -info
初始化后查看mysql中表情况：show tables;
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 | mysql> show tables; +---------------------------+ | Tables_in_hive | +---------------------------+ | BUCKETING_COLS | | CDS | | COLUMNS_V2 | | DATABASE_PARAMS | | DBS | | DB_PRIVS | | DELEGATION_TOKENS | | GLOBAL_PRIVS | | IDXS | | INDEX_PARAMS | | MASTER_KEYS | | NUCLEUS_TABLES | | PARTITIONS | | PARTITION_EVENTS | | PARTITION_KEYS | | PARTITION_KEY_VALS | | PARTITION_PARAMS | | PART_COL_PRIVS | | PART_COL_STATS | | PART_PRIVS | | ROLES | | ROLE_MAP | | SDS | | SD_PARAMS | | SEQUENCE_TABLE | | SERDES | | SERDE_PARAMS | | SKEWED_COL_NAMES | | SKEWED_COL_VALUE_LOC_MAP | | SKEWED_STRING_LIST | | SKEWED_STRING_LIST_VALUES | | SKEWED_VALUES | | SORT_COLS | | TABLE_PARAMS | | TAB_COL_STATS | | TBLS | | TBL_COL_PRIVS | | TBL_PRIVS | | TYPES | | TYPE_FIELDS | | VERSION | +---------------------------+ 41 rows in set (0.00 sec) |
配置完成后就可在shell中以CLI的方式访问hive 进行操作验证。
[三]、Remote方式
以Mysql数据库(192.168.6.77)为例：创建好用户：hive；database：hive_meta。Remote方式需要分别配置服务端和客户端的配置文件：
服务端的 hive-site.xml 中jdbc URL、驱动、用户名、密码等属性值配置如下：
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 | <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://192.168.6.77/hive_meta?createDatabaseIfNotExist=true</value> <description>JDBC connect string for a JDBC metastore</description> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> <description>Driver class name for a JDBC metastore</description> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>hive</value> <description>username to use against metastore database</description> </property> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>micmiu</value> <description>password to use against metastore database</description> </property> <property> <name>hive.metastore.warehouse.dir</name> <!-- base hdfs path --> <value>/user/hive/warehouse</value> <description>base hdfs path :location of default database for the warehouse</description> </property> <!-- micmiu.com --> |
ps：需要把mysql的驱动包copy到目录 <HIVE_HOME>/lib 中
如果是第一次需要执行初始化命令：schematool -dbType mysql -initSchema
客户端中配置内容修改如下：
| 1 2 3 4 5 6 7 8 9 10 11 12 13 | <!-- thrift://<host_name>:<port> 默认端口是9083 --> <property> <name>hive.metastore.uris</name> <value>thrift://192.168.6.77:9083</value> <description>Thrift uri for the remote metastore. Used by metastore client to connect to remote metastore.</description> </property> <!-- hive表的默认存储路径 --> <property> <name>hive.metastore.warehouse.dir</name> <value>/user/hive/warehouse</value> <description>location of default database for the warehouse</description> </property> |
hive metastore 服务端启动命令：
hive --service metastore -p <port_num>
如果不加端口默认启动：hive --service metastore，则默认监听端口是：9083 ，注意客户端中的端口配置需要和启动监听的端口一致。服务端启动正常后，客户端就可以执行hive操作了。
参考：
https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin
转载于:https://www.cnblogs.com/zhengrunjian/p/4546032.html
总结
以上是生活随笔为你收集整理的【hive】——metastore的三种模式的全部内容，希望文章能够帮你解决所遇到的问题。
- 上一篇： 2021年第十八届五一数学建模竞赛题目
- 下一篇： Fiddler 抓包详细使用教程