本文主要包含hadoop实战,hadoop实战第3版pdf,hadoop实战第二版,hadoop实战 pdf,hadoop实战视频等服务器相关知识,网友希望可以进行参考
Hadoop实战 Hadoop Pipes运行C++程序问题解决，hadooppipes
说明：我使用的是hadoop-1.2.1，开发环境是OpenSuSE12.3 x64。 Hadoop安装在/usr/lib/hadoop下。并且我的hadoop的相关指令已经加入到了系统的path中。
下面四篇有我解决问题时，所主要参考的文档：
1、http://www.cnblogs.com/lanxuezaipiao/p/3648853.html 该博客指出64位的libhadooppipes.a和 libhadooputils.a这两个库应该由我们自己编译，官方提供的是32位的库。
2、http://guoyunsky.iteye.com/blog/1709654
3、http://blog.csdn.net/keljony/article/details/29872915
4、http://blog.csdn.net/sigxxl/article/details/12293435
第二、三、四篇博客指出了如何解决“Hadoop Pipes “Server failed to authenticate”错误，这也是我们的主要内容。
这里所说的是《hadoop实战》（第2版）中3.5节Hadoop Pipes的例子。
（1）下面附上书上的wordcount.cpp的代码，这里我完全基本完全照着书上敲的，为了方便只在前面加了名using namespace std;另外加了<vector>的头文件。
#include "hadoop/Pipes.hh"
#include "hadoop/TemplateFactory.hh"
#include "hadoop/StringUtils.hh"
#include <vector>
using namespace std;
const std::string WORDCOUNT = "WORDCOUNT";
const std::string INPUT_WORDS = "INPUT_WORDS";
const std::string OUTPUT_WORDS = "OUTPUT_WORDS";
class WordCountMap : public HadoopPipes::Mapper
{
public:
HadoopPipes::TaskContext::Counter* inputWords;
WordCountMap(HadoopPipes::TaskContext& context)
{
inputWords = context.getCounter(WORDCOUNT, INPUT_WORDS);
}
void map(HadoopPipes::MapContext& context)
{
vector<string> words = HadoopUtils::splitString(context.getInputValue(), " ");
for(unsigned int i = 0; i < words.size(); ++i)
{
context.emit(words[i], "1");
}
context.incrementCounter(inputWords, words.size());
}
};
class WordCountReduce : public HadoopPipes::Reducer
{
public:
HadoopPipes::TaskContext::Counter* outputWords;
WordCountReduce(HadoopPipes::TaskContext& context)
{
outputWords = context.getCounter(WORDCOUNT, OUTPUT_WORDS);
}
void reduce(HadoopPipes::ReduceContext& context)
{
int sum = 0;
while(context.nextValue())
{
sum += HadoopUtils::toInt(context.getInputValue());
}
context.emit(context.getInputKey(), HadoopUtils::toString(sum));
context.incrementCounter(outputWords, 1);
}
};
int main(int argc, char* argv[])
{
return HadoopPipes::runTask(HadoopPipes::TemplateFactory<WordCountMap, WordCountReduce>());
}
HADOOP_INSTALL=/usr/lib/hadoop-1.2.1
PLATFORM=Linux-i386-32
CC=g++
CPPFLAGS= -m32 -I$(HADOOP_INSTALL)/c++/$(PLATFORM)/include
wordcount: wordcount.cpp
$(CC) ${CPPFLAGS} $< -Wall -L$(HADOOP_INSTALL)/c++/$(PLATFORM)/lib -lhadooppipes -lhadooputils -lpthread -g -O2 -o $@
clean:
rm -f *.o wordcount
由于默认使用的库是32位的，如果你用的是64位的系统，应该将
PLATFORM=Linux-i386-32修改为
Linux-amd64-64。这里书上说得有点问题，书上说的是“如果是AMD的CPU，请使用Linux-amd64-64”,这里应该是如果是64位的系统的话，请使用Linux-amd64-64。并将
CPPFLAGS= -m32 -I$(HADOOP_INSTALL)/c++/$(PLATFORM)/include
改为
CPPFLAGS= -m64 -I$(HADOOP_INSTALL)/c++/$(PLATFORM)/include
修改完成后，输入make进行编译会报如下的错误：
g++ -m64 -I/usr/lib/hadoop-1.2.1/c++/Linux-amd64-64/include wordcount.cpp -Wall -L/usr/lib/hadoop-1.2.1/c++/Linux-amd64-64/lib -lhadooppipes -lhadooputils -lpthread -g -O2 -o wordcount /usr/lib/hadoop-1.2.1/c++/Linux-amd64-64/lib/libhadooppipes.a(HadoopPipes.o): In function `HadoopPipes::BinaryProtocol::createDigest(std::string&, std::string&)': /usr/lib/hadoop-1.2.1/src/c++/pipes/impl/HadoopPipes.cc:426: undefined reference to `EVP_sha1' /usr/lib/hadoop-1.2.1/src/c++/pipes/impl/HadoopPipes.cc:426: undefined reference to `HMAC_Init' /usr/lib/hadoop-1.2.1/src/c++/pipes/impl/HadoopPipes.cc:427: undefined reference to `HMAC_Update' /usr/lib/hadoop-1.2.1/src/c++/pipes/impl/HadoopPipes.cc:429: undefined reference to `HMAC_Final' /usr/lib/hadoop-1.2.1/src/c++/pipes/impl/HadoopPipes.cc:430: undefined reference to `HMAC_CTX_cleanup' /usr/lib/hadoop-1.2.1/src/c++/pipes/impl/HadoopPipes.cc:436: undefined reference to `BIO_f_base64' /usr/lib/hadoop-1.2.1/src/c++/pipes/impl/HadoopPipes.cc:436: undefined reference to `BIO_new' /usr/lib/hadoop-1.2.1/src/c++/pipes/impl/HadoopPipes.cc:437: undefined reference to `BIO_s_mem' /usr/lib/hadoop-1.2.1/src/c++/pipes/impl/HadoopPipes.cc:437: undefined reference to `BIO_new' /usr/lib/hadoop-1.2.1/src/c++/pipes/impl/HadoopPipes.cc:438: undefined reference to `BIO_push' /usr/lib/hadoop-1.2.1/src/c++/pipes/impl/HadoopPipes.cc:439: undefined reference to `BIO_write' /usr/lib/hadoop-1.2.1/src/c++/pipes/impl/HadoopPipes.cc:440: undefined reference to `BIO_ctrl' /usr/lib/hadoop-1.2.1/src/c++/pipes/impl/HadoopPipes.cc:441: undefined reference to `BIO_ctrl' /usr/lib/hadoop-1.2.1/src/c++/pipes/impl/HadoopPipes.cc:446: undefined reference to `BIO_free_all' /usr/lib/hadoop-1.2.1/src/c++/pipes/impl/HadoopPipes.cc:426: undefined reference to `EVP_sha1' /usr/lib/hadoop-1.2.1/src/c++/pipes/impl/HadoopPipes.cc:426: undefined reference to `HMAC_Init' /usr/lib/hadoop-1.2.1/src/c++/pipes/impl/HadoopPipes.cc:427: undefined reference to `HMAC_Update' /usr/lib/hadoop-1.2.1/src/c++/pipes/impl/HadoopPipes.cc:429: undefined reference to `HMAC_Final' /usr/lib/hadoop-1.2.1/src/c++/pipes/impl/HadoopPipes.cc:430: undefined reference to `HMAC_CTX_cleanup' /usr/lib/hadoop-1.2.1/src/c++/pipes/impl/HadoopPipes.cc:436: undefined reference to `BIO_f_base64' /usr/lib/hadoop-1.2.1/src/c++/pipes/impl/HadoopPipes.cc:436: undefined reference to `BIO_new' /usr/lib/hadoop-1.2.1/src/c++/pipes/impl/HadoopPipes.cc:437: undefined reference to `BIO_s_mem' /usr/lib/hadoop-1.2.1/src/c++/pipes/impl/HadoopPipes.cc:437: undefined reference to `BIO_new' /usr/lib/hadoop-1.2.1/src/c++/pipes/impl/HadoopPipes.cc:438: undefined reference to `BIO_push' /usr/lib/hadoop-1.2.1/src/c++/pipes/impl/HadoopPipes.cc:439: undefined reference to `BIO_write' /usr/lib/hadoop-1.2.1/src/c++/pipes/impl/HadoopPipes.cc:440: undefined reference to `BIO_ctrl' /usr/lib/hadoop-1.2.1/src/c++/pipes/impl/HadoopPipes.cc:441: undefined reference to `BIO_ctrl' /usr/lib/hadoop-1.2.1/src/c++/pipes/impl/HadoopPipes.cc:446: undefined reference to `BIO_free_all' collect2: error: ld returned 1 exit status make: *** [wordcount] Error 1出现该错误是因为在编译时少加了两个选项（-lcrypto -lssl）所导致的，应将
<