Oozie编译

• 参考oozie编译

所需要的软件包及已安装软件

1. jdk 1.8
2. hadoop 2.7.5
3. mysql 5.7
4. apache-maven-3.6.3-bin.tar.gz
5. ext-2.2.zip
6. oozie-4.1.0.tar.gz
7. doxia-core-1.0-alpha-9.2y.jar
8. doxia-module-twiki-1.0-alpha-9.2y.jar
9. pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar

安装maven

1. 解压

tar -zxf /opt/pkg/apache-maven-3.6.3-bin.tar.gz  -C /opt/core/

vim ~/.bash_profile
export M2_HOME=/opt/core/apache-maven-3.6.3/
export PATH=$PATH:$M2_HOME/bin

source ~/.bash_profile
mvn -version

2. 修改setting文件，添加以下内容：
   vim /opt/core/apache-maven-3.6.3/conf/settings.xml

    <mirror>
        <id>alimaven</id>
        <mirrorOf>central</mirrorOf>
        <name>aliyun maven</name>
        <url>http://maven.aliyun.com/nexus/content/repositories/central/</url>
    </mirror>

编译oozie

1. 解压

tar -zxf /opt/pkg/oozie-4.1.0.tar.gz -C /opt/core/

2. 修改pom.xml，将Java版本修改为1.8

3. 编译

cd /opt/core/oozie-4.1.0/
bin/mkdistro.sh  -DskipTests

4. 将oozie目录备份

mv /opt/core/oozie-4.1.0 /opt/core/oozie-4.1.0_bak

5. 解压刚编译好的oozie

 tar -zxf /opt/core/oozie-4.1.0/distro/target/oozie-4.1.0-distro.tar.gz -C /opt/core/

6. 创建libext目录

mkdir /opt/core/oozie-4.1.0/libext

7. 把hadoop里面的包复制到libext里面

cp /opt/core/hadoop-2.7.5/share/hadoop/*/lib/*.jar /opt/core/oozie-4.1.0/libext
cp /opt/core/hadoop-2.7.5/share/hadoop/*/*.jar /opt/core/oozie-4.1.0/libext

8. 将mysql-connector的jar包和ext-2.2.zip也放到libext文件夹里面

cp /opt/pkg/mysql-connector-java-5.1.40-bin.jar /opt/core/oozie-4.1.0/libext
cp /opt/pkg/ext-2.2.zip /opt/core/oozie-4.1.0/libext

9. 把libext 里面一系列和hadoop冲突的包丢掉

mv /opt/core/oozie-4.1.0/libext/servlet-api-2.5.jar /opt/core/oozie-4.1.0/libext/servlet-api-2.5.jar.bak
mv /opt/core/oozie-4.1.0/libext/jsp-api-2.1.jar /opt/core/oozie-4.1.0/libext/jsp-api-2.1.jar.bak

10. 执行下面命令

cd /opt/core/oozie-4.1.0/
bin/oozie-setup.sh prepare-war

11. 修改oozie-env.sh

# Set Java hoem and hadoop prefix  
export JAVA_HOME=/opt/core/jdk1.8.0_65
export OOZIE_PREFIX=/opt/core/oozie-4.1.0/
 
 
# Set hadoop configuration path  
export OOZIE_CONF_DIR=/opt/core/oozie-4.1.0//conf/
export OOZIE_HOME=/opt/core/oozie-4.1.0/
 
 
# add hadoop package 
export CLASSPATH=$CLASSPATH:$OOZIE_HOME/libext/*.jar

source conf/oozie-env.sh

12. 解压oozie-4.1.0目录下的share lib

cd /opt/core/oozie-4.1.0/
tar -xzvf oozie-sharelib-4.1.0.tar.gz 

13. 把share lib放到oozie-site.xml配置的路径里去

cd /opt/core/oozie-4.1.0/
hdfs dfs -put share /user/root/

14. 在oozie关联mysql数据库

bin/ooziedb.sh create -sqlfile oozie.sql -run

15. 启动hadoop和job history

start-dfs.sh
start-yarn.sh
mr-jobhistory-daemon.sh start historyserve

16. 启动oozie

bin/oozied.sh start

17. 查看oozie状态

bin/oozie admin --oozie http://localhost:11000/oozie -status

错误锦集

1.[ERROR] Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:2.7:javadoc (default) on project oozie-client: An error has occurred in JavaDocs report generation:
原因：Javadoc编译错误
解决：修改pom.xml文件中的maven-javadoc-plugin插件，将版本升级到3.x之后，不然下面的参数 -Xdoclint:none 不会生效

                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-javadoc-plugin</artifactId>
                    <version>3.0.1</version>
                    <configuration>
                          <failOnError>false</failOnError>
                          <doclint>none</doclint>
                    </configuration>
                </plugin>

参考：Javadoc编译错误
2. [ERROR] Failed to execute goal org.apache.maven.plugins:maven-project-info-reports-plugin:2.3:dependencies (default) on project oozie-webapp: An error has occurred in Dependencies report generation.: error in opening zip file
解决：注释掉pom.xml中的maven-project-info-reports-plugin插件

                <!--<plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-project-info-reports-plugin</artifactId>
                    <version>2.3</version>
                </plugin>-->