本文主要包含【甘道夫】Apache Hadoop 2.5.0-cdh5.2.0 HDFS Quotas 配额控制，hadoophdfs等服务器相关知识,网友希望可以进行参考
【甘道夫】Apache Hadoop 2.5.0-cdh5.2.0 HDFS Quotas 配额控制，hadoophdfs
前言 HDFS为管理员提供了针对目录的配额控制特性，可以控制名称配额（指定目录下的文件&文件夹总数），或者空间配额（占用磁盘空间的上限）。 本文探究了HDFS的配额控制特性，记录了各类配额控制场景的实验详细过程。 实验环境基于Apache Hadoop 2.5.0-cdh5.2.0。欢迎转载，请注明出处：http://blog.csdn.net/u010967382/article/details/44452485
名称配额功能试用
设置名称配额，即当前目录下文件和目录的最大数量： casliyang@singlehadoop:~$ hdfs dfsadmin -setQuota 3 /Workspace/quotas/ 15/03/18 14:53:52 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
上传文件： casliyang@singlehadoop:~$ hdfs dfs -put slf4j-log4j12-1.6.4.jar /Workspace/quotas/ 15/03/18 14:54:16 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
上传文件： casliyang@singlehadoop:~$ hdfs dfs -put dict.txt /Workspace/quotas/ 15/03/18 14:55:08 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
继续上传文件报错： casliyang@singlehadoop:~$ hdfs dfs -put examples.desktop /Workspace/quotas/ 15/03/18 14:55:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable put: The NameSpace quota (directories and files) of directory /Workspace/quotas is exceeded: quota=3 file count=4
查看该目录情况： casliyang@singlehadoop:~$ hdfs dfs -ls /Workspace/quotas 15/03/18 17:11:31 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Found 2 items -rw-r--r-- 3 casliyang supergroup 14 2015-03-18 14:55 /Workspace/quotas/dict.txt -rw-r--r-- 3 casliyang supergroup 9748 2015-03-18 14:38 /Workspace/quotas/slf4j-log4j12-1.6.4.jar 目录下只有两个文件。
查看配额情况： casliyang@singlehadoop:~$ hdfs dfs -count -q /Workspace/quotas 15/03/18 16:00:53 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 3 0 none inf 1 2 9762 /Workspace/quotas
注意几个数字的含义依次是：
名称配额剩余量的计算公式： 名称配额剩余量 = 名称配额总量 - （ 目录数 + 文件数 ）
基于上面配额查看结果得知： 名称配额总量=3 目录数=1 文件数=2
所以，名称配额剩余量=3-（1+2）=0 所以此时继续上传文件会超出名称配额的限制。
BTW，居然把根目录也计数了！
我们再来建一个没有设定quotas的目录，测试下是否也将根目录纳入count统计： casliyang@singlehadoop:~$ hdfs dfs -mkdir /Workspace/quotas1 15/03/18 17:19:21 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable casliyang@singlehadoop:~$ hdfs dfs -count -q /Workspace/quotas1 15/03/18 17:20:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable none inf none inf 1 0 0 /Workspace/quotas1
上面的实验看出，名称配额和空间配额都没设置，所以都显示none/inf，果然当前根目录还是计入统计了，目录数为1。
下面通过命令取消/Workspace/quotas目录的名称配额： casliyang@singlehadoop:~$ hdfs dfs -count -q /Workspace/quotas 15/03/18 17:28:25 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 3 0 none inf 1 2 9762 /Workspace/quotas casliyang@singlehadoop:~$ hdfs dfsadmin -clrQuota /Workspace/quotas 15/03/18 17:28:40 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable casliyang@singlehadoop:~$ hdfs dfs -count -q /Workspace/quotas 15/03/18 17:28:50 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable none inf none inf 1 2 9762 /Workspace/quotas
空间配额功能试用
清空目录/Workspace/quotas： casliyang@singlehadoop:~$ hdfs dfs -rm -r /Workspace/quotas/* 15/03/18 17:33:08 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 15/03/18 17:33:11 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes. Deleted /Workspace/quotas/dict.txt 15/03/18 17:33:11 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes. Deleted /Workspace/quotas/slf4j-log4j12-1.6.4.jar casliyang@singlehadoop:~$ hdfs dfs -ls /Workspace/quotas/ 15/03/18 17:33:36 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable casliyang@singlehadoop:~$ hdfs dfs -count -q /Workspace/quotas 15/03/18 17:33:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable none inf none inf 1 0 0 /Workspace/quotas
为目录设定空间限额： casliyang@singlehadoop:~$ hdfs dfsadmin -setSpaceQuota 8000 /Workspace/quotas 15/03/18 17:36:17 WARN util.NativeCodeLoader: Unable to load native