HDFS命令行操作

前提准备

熟悉HDFS的命令行操作，需要事先在机器上搭建好环境。（点击查看伪分布式搭建；点击查看HA完全分布式搭建）。

启动

然后把我们的集群启动起来（这里我就先启动了伪分布式的环境）：

start-dfs.sh 

所有的节点都启动起来后，输入以下命令会给出一系列命令的提示：

[root@node1 hadoop-2.6.5]# hdfs dfs
Usage: hadoop fs [generic options]
	[-appendToFile <localsrc> ... <dst>]
	[-cat [-ignoreCrc] <src> ...]
	[-checksum <src> ...]
	[-chgrp [-R] GROUP PATH...]
	[-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]
	[-chown [-R] [OWNER][:[GROUP]] PATH...]
	[-copyFromLocal [-f] [-p] [-l] <localsrc> ... <dst>]
	[-copyToLocal [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
	[-count [-q] [-h] <path> ...]
	[-cp [-f] [-p | -p[topax]] <src> ... <dst>]
	[-createSnapshot <snapshotDir> [<snapshotName>]]
	[-deleteSnapshot <snapshotDir> <snapshotName>]
	[-df [-h] [<path> ...]]
	[-du [-s] [-h] <path> ...]
	[-expunge]
	[-get [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
	[-getfacl [-R] <path>]
	[-getfattr [-R] {-n name | -d} [-e en] <path>]
	[-getmerge [-nl] <src> <localdst>]
	[-help [cmd ...]]
	[-ls [-d] [-h] [-R] [<path> ...]]
	[-mkdir [-p] <path> ...]
	[-moveFromLocal <localsrc> ... <dst>]
	[-moveToLocal <src> <localdst>]
	[-mv <src> ... <dst>]
	[-put [-f] [-p] [-l] <localsrc> ... <dst>]
	[-renameSnapshot <snapshotDir> <oldName> <newName>]
	[-rm [-f] [-r|-R] [-skipTrash] <src> ...]
	[-rmdir [--ignore-fail-on-non-empty] <dir> ...]
	[-setfacl [-R] [{-b|-k} {-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>]]
	[-setfattr {-n name [-v value] | -x name} <path>]
	[-setrep [-R] [-w] <rep> <path> ...]
	[-stat [format] <path> ...]
	[-tail [-f] <file>]
	[-test -[defsz] <path>]
	[-text [-ignoreCrc] <src> ...]
	[-touchz <path> ...]
	[-usage [cmd ...]]

Generic options supported are
-conf <configuration file>     specify an application configuration file
-D <property=value>            use value for given property
-fs <local|namenode:port>      specify a namenode
-jt <local|resourcemanager:port>    specify a ResourceManager
-files <comma separated list of files>    specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars>    specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives>    specify comma separated archives to be unarchived on the compute machines.

The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]

1.创建目录：

想创建目录就是"mkdir"，想创建多层就是"-p"

hdfs dfs -mkdir -p /user/root

2.上传文件

执行“-put”命令：

hdfs dfs -put ~/hadoop-2.6.5.tar.gz  /user/root

3.修改block大小

首先我们定义一个小文件：

for i in `seq 100000`;do echo "hello world $i" >>test.txt;done

然后在上传的时候设定block的大小：

hdfs dfs -D dfs.blocksize=1048576 -put test.txt /user/root

我们去到“/var/software/hadoop/local/dfs/data/current/BP-269914864-192.168.16.111-1560277309858/current/finalized/subdir0/subdir0
”目录下查看：

[root@node1 subdir0]# ll
总用量 198256
-rw-r--r-- 1 root root 134217728 6月  12 16:52 blk_1073741825
-rw-r--r-- 1 root root   1048583 6月  12 16:52 blk_1073741825_1001.meta
-rw-r--r-- 1 root root  65417541 6月  12 16:52 blk_1073741826
-rw-r--r-- 1 root root    511083 6月  12 16:52 blk_1073741826_1002.meta
-rw-r--r-- 1 root root   1048576 6月  12 16:59 blk_1073741827
-rw-r--r-- 1 root root      8199 6月  12 16:59 blk_1073741827_1003.meta
-rw-r--r-- 1 root root    740319 6月  12 16:59 blk_1073741828
-rw-r--r-- 1 root root      5791 6月  12 16:59 blk_1073741828_1004.meta

每个块旁边都有一个记录了元数据信息的小文件描述它，最重要的就是做验证用的MD5值。
其它的诸如"hdfs dfs -ls",“hdfs dfs -cat”,“hdfs dfs -get” “-chmod” ，“-chown”等和Linux命令无异，举一反三，这里就过多赘述了。