kerberos下HBase访问Zookeeper的ACL问题，hbasezookeeper

kerberos下HBase访问Zookeeper的ACL问题，hbasezookeeper

最近公司HBase(CDH-4.6.0)遇到了一个麻烦问题，觉得有必要记录下整个解决的过程。

问题起因

用户在跑mapreduce任务，从hdfs读取文件想写入到hbase table的时候失败了（这是hbase提供的一种mapred能力）。这个问题发现在A环境(一个测试环境)，自从启用了kerberos之后。运行了用户给的程序和自己写的sample之后，发现程序最后挂在NullPointerException上。这个NPE指示的是服务端的一个叫currentKey的变量为null。

org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@58e395e8,java.io.IOException: java.io.IOException: java.lang.NullPointerException
at  org.apache.hadoop.hbase.security.token.AuthenticationTokenSecretManager.createPassword(AuthenticationTokenSecretManager.java:129) 
at org.apache.hadoop.hbase.security.token.AuthenticationTokenSecretManager.createPassword(AuthenticationTokenSecretManager.java:57)
at org.apache.hadoop.security.token.Token.<init>(Token.java:70)
at org.apache.hadoop.hbase.security.token.AuthenticationTokenSecretManager.generateToken(AuthenticationTokenSecretManager.java:162)
at org.apache.hadoop.hbase.security.token.TokenProvider.getAuthenticationToken(TokenProvider.java:91) 
at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.regionserver.HRegion.exec(HRegion.java:5610)
at org.apache.hadoop.hbase.regionserver.HRegionServer.execCoprocessor(HRegionServer.java:3918)
at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.SecureRpcEngine$Server.call(SecureRpcEngine.java:311)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1428)

AuthenticationTokenSecretManager：

  @Override
  protected byte[] createPassword(AuthenticationTokenIdentifier identifier) {
    long now = EnvironmentEdgeManager.currentTimeMillis();
    AuthenticationKey secretKey = currentKey;  //currentKey赋给secretKey
    identifier.setKeyId(secretKey.getKeyId()); //NPE在这里抛出的，也就是currentKey为null
    identifier.setIssueDate(now);
    identifier.setExpirationDate(now + tokenMaxLifetime);
    identifier.setSequenceNumber(tokenSeq.getAndIncrement());
    return createPassword(WritableUtils.toByteArray(identifier),
        secretKey.getKey());
  }

问题定位

既然currentKey为null，那我们就去找它在哪里赋值的。阅读源码之后，了解到整个过程是这样的：

1.在开启kerberos之后，每个RegionServer都会有一个AuthenticationTokenSecretManager用来管理token。

2.这些manager中，只有一个leader，只有它能生产token，然后放到zookeeper里。其它manager通过感知zookeeper的变化来同步leader生产的token。leader通过竞争产生，谁先在ZK上创建 /hbase/tokenauth/keymaster 节点，谁就是leader。

AuthenticationTokenSecretManager$LeaderElector：

    public void run() {
      zkLeader.start();
      zkLeader.waitToBecomeLeader();  //没有成为leader的人会一直阻塞在这里，直到感知到当前leader挂掉才会开始新一轮竞争
      isMaster = true;
      while (!stopped) {
        long now = EnvironmentEdgeManager.currentTimeMillis();
        // clear any expired
        removeExpiredKeys(); //清除过期的token，同时也把它从ZK上移除
        if (lastKeyUpdate + keyUpdateInterval < now) {  //默认的周期是1天
          // roll a new master key
          rollCurrentKey();  //就是这个函数产生新的token，替换currenKey
        }
        try {
          Thread.sleep(5000);
        } catch (InterruptedException ie) {
          if (LOG.isDebugEnabled()) {
            LOG.debug("Interrupted waiting for next update", ie);
          }
        }
      }
    }

AuthenticationTokenSecretManager：

  synchronized void rollCurrentKey() {
    if (!leaderElector.isMaster()) {
      LOG.info("Skipping rollCurrentKey() because not running as master.");
      return;
    }
    long now = EnvironmentEdgeManager.currentTimeMillis();
    AuthenticationKey prev = currentKey;
    AuthenticationKey newKey = new AuthenticationKey(++idSeq,
        Long.MAX_VALUE, // don't allow to expire until it's replaced by a new key
        generateSecret());
    allKeys.put(newKey.getKeyId(), newKey);
    currentKey = newKey;           //滚动currentKey，置为newKey
    zkWatcher.addKeyToZK(newKey);  //把新的token放到zookeeper
    lastKeyUpdate = now;
    if (prev != null) {
      // make sure previous key is still stored
      prev.setExpiration(now + tokenMaxLifetime); //prev是原来的newKey，是不会过期的，当有新的newKey替代它后，它的期限默认设置是7天
      allKeys.put(prev.getKeyId(), prev);
      zkWatcher.updateKeyInZK(prev);
    }
  }

3.既然token是由leader生产的，除非没有leader，才会没人生产。验证这个想法，我在zookeeper和一些region server启动当天的日志里找到了证据：
a) zk中的 /hbase/tokenauth/keymaster 节点用来存放leader的信息，然后进入zookeeper-client查看了下，根本没这个节点。
b) 一些尚有保留集群启动当天日志的region server上找到了如下异常：

org.apache.hadoop.hbase.security.token.AuthenticationTokenSecretManager: Zookeeper initialization failed
org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /hbase/tokenauth/keys
at org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:421)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:403)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1164)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1142)
at org.apache.hadoop.hbase.security.token.ZKSecretWatcher.start(ZKSecretWatcher.java:58)
at org.apache.hadoop.hbase.security.token.AuthenticationTokenSecretManager.start(AuthenticationTokenSecretManager.java:105)
at org.apache.hadoop.hbase.ipc.SecureR