Hive Developing，hivedeveloping

Building hive from source

clone and build from source

downlaod

git clone https://git-wip-us.apache.org/repos/asf/hive.git
or 
git clone git@github.com:wankunde/hive.git

build

git branch -va
// checkout the branch which you are intrest.
git checkout -b branch-0.14 origin/branch-0.14
// or git checkout --track origin/branch-0.14

// compile and dist
mvn clean install -DskipTests -Phadoop-2 -Pdist

//  generate protobuf code
cd ql
mvn clean install -DskipTests -Phadoop-2,protobuf

// generate Thrift code
mvn clean install -Phadoop-2,thriftif -DskipTests -Dthrift.home=/usr/local

Tips

• By default,before compile,maven will download many dependency packages and meet timeout exception.I use nexus and add <timeout>120000</timeout> configuration to <server> configuration item. Not Test
• Hive HiveDeveloperFAQ Wiki
• If you want to compile and test hive in Eclipse,continue the follow steps.
  
  • mvn eclipse:eclipse
  • import project
  • Configure - Convert to maven project
  • Some maven plugins may not work well,(may need to set local proxy host and port in eclipse),connect to m2e market and install m2e connectors what you need.(include antlr,build-helper)
    

You can access the m2e market place from the preferences: Preferences>Maven>Discovery>Open Catalog. Installing WTP integration solved most plugin issues for me.

Test Unit

Change hive log level

bin/hive -hiveconf hive.root.logger=DEBUG,console

Or change log4j properties.

cp conf/hive-log4j.properties.template conf/hive-log4j.properties

Connecting a Java Debugger to hive

Example java remote debug

• Run remote java program using script

JVM_OPTS="-server -XX:+UseParNewGC -XX:+HeapDumpOnOutOfMemoryError"
DEBUG="-Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=2345"
JVM_OPTS="$JVM_OPTS $DEBUG"
$JAVA_HOME/bin/java $JVM_OPTS -cp tools-1.0.jar com.wankun.tools.hdfs.Test2

• Add and run new remote debug configuration in eclipse

Start hive debug

help command
hive --help --debug

Run hive without a hadoop cluster

export HIVE_OPTS='--hiveconf mapred.job.tracker=local --hiveconf fs.default.name=file:///tmp \
    --hiveconf hive.metastore.warehouse.dir=file:///tmp/warehouse \
    --hiveconf javax.jdo.option.ConnectionURL=jdbc:derby:;databaseName=/tmp/metastore_db;create=true'

Hive test unit

Two kind of unit tests

• Normal unit test

mvn test -Dtest