目录
目标:
像iceberg一样,在hive能直接查询hudi表数据。
一,导入jar
将hudi-hadoop-mr-bundle-0.9.0-SNAPSHOT.jar 放入到hive lib下面;
![]()
二,创建hudi表:
mor:
cow:
写入数据:
三,创建hive外部表
在hive shell执行,创建外部表:
CREATE EXTERNAL TABLE hudi_db.hudi_test2(
`_hoodie_commit_time` string,
`_hoodie_commit_seqno` string,
`_hoodie_record_key` string,
`_hoodie_partition_path` string,
`_hoodie_file_name` string,
`uuid` STRING,
`ts` bigint)
PARTITIONED BY (
`partition` string)
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
STORED AS INPUTFORMAT
'org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION
'hdfs://bi-524:8020/tmp/hudi/hudi_test2';添加分区
alter table hudi_db.hudi_test2 add if not exists partition(`partition`='par1') location 'hdfs://bi-524:8020/tmp/hudi/hudi_test2/par1'
四,查询结果
五,问题总结
不懂的看文章:
https://blog.csdn.net/u010848845/article/details/115701329
1,我这里是hive2.1版本,使用cow的方式查询到了数据, mor没有

官方解释:

Mor也会生成parquet 但是要等到compaction,默认checkpoint 5次才会触发(这个参数值也可以修改)。

也可以看看 org.apache.hudi.configuration.FlinkOptions 参数有没有设置的,官网参数地址:http://hudi.apache.org/docs/configurations.html#write-options-1
还可以参考文章:
https://blog.csdn.net/hjl18309163914/article/details/116057379
2,创建hudi表 好像必须要有ts字段
其他:
.
5,hudi目前是大小写敏感,错误信息会导致JM插入数据时候挂掉
Caused by: java.lang.IllegalArgumentException: No enum constant org.apache.hudi.common.model.HoodieTableType.Copy_On_Write
at java.lang.Enum.valueOf(Enum.java:238) ~[?:1.8.0_162]
at org.apache.hudi.common.model.HoodieTableType.valueOf(HoodieTableType.java:30) ~[hudi-flink-bundle_2.11-0.9.0-SNAPSHOT.jar:0.9.0-SNAPSHOT]
at org.apache.hudi.common.table.HoodieTableMetaClient$PropertyBuilder.setTableType(HoodieTableMetaClient.java:613) ~[hudi-flink-bundle_2.11-0.9.0-SNAPSHOT.jar:0.9.0-SNAPSHOT]
at org.apache.hudi.util.StreamerUtil.initTableIfNotExists(StreamerUtil.java:262) ~[hudi-flink-bundle_2.11-0.9.0-SNAPSHOT.jar:0.9.0-SNAPSHOT]
at org.apache.hudi.sink.StreamWriteOperatorCoordinator.start(StreamWriteOperatorCoordinator.java:154) ~[hudi-flink-bundle_2.11-0.9.0-SNAPSHOT.jar:0.9.0-SNAPSHOT]
at org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder.start(OperatorCoordinatorHolder.java:189) ~[flink-dist_2.11-1.12.2.jar:1.12.2]
at org.apache.flink.runtime.scheduler.SchedulerBase.startAllOperatorCoordinators(SchedulerBase.java:1253) ~[flink-dist_2.11-1.12.2.jar:1.12.2]
at org.apache.flink.runtime.scheduler.SchedulerBase.startScheduling(SchedulerBase.java:624) ~[flink-dist_2.11-1.12.2.jar:1.12.2]
at org.apache.flink.runtime.jobmaster.JobMaster.startScheduling(JobMaster.java:1032) ~[flink-dist_2.11-1.12.2.jar:1.12.2]
at java.util.concurrent.CompletableFuture.uniRun(CompletableFuture.java:705) ~[?:1.8.0_162]

最后提醒:
如果你创建表没问题,数据查询不到,试试建表类似设置为mor 或者cow试试
参考文章:
https://blog.csdn.net/u010848845/article/details/115701329








