hadoop fs Commands

Help Command

Hadoop file system commands are ran using hadoop fs with all commands falling under that prefix. e.g. to get help we do hadoop fs -help

$ hadoop fs -help
Usage: hadoop fs [generic options]
        [-appendToFile <localsrc> ... <dst>]
        [-cat [-ignoreCrc] <src> ...]
        [-checksum <src> ...]
        [-chgrp [-R] GROUP PATH...]
        [-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]
        [-chown [-R] [OWNER][:[GROUP]] PATH...]
        [-copyFromLocal [-f] [-p] [-l] <localsrc> ... <dst>]
        [-copyToLocal [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
        [-count [-q] [-h] <path> ...]
        [-cp [-f] [-p | -p[topax]] <src> ... <dst>]
        [-createSnapshot <snapshotDir> [<snapshotName>]]
        [-deleteSnapshot <snapshotDir> <snapshotName>]
        [-df [-h] [<path> ...]]
        [-du [-s] [-h] <path> ...]
        [-expunge]
        [-get [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
        [-getfacl [-R] <path>]
        [-getfattr [-R] {-n name | -d} [-e en] <path>]
        [-getmerge [-nl] <src> <localdst>]
        [-help [cmd ...]]
        [-ls [-d] [-h] [-R] [<path> ...]]
        [-mkdir [-p] <path> ...]
        [-moveFromLocal <localsrc> ... <dst>]
        [-moveToLocal <src> <localdst>]
        [-mv <src> ... <dst>]
        [-put [-f] [-p] [-l] <localsrc> ... <dst>]
        [-renameSnapshot <snapshotDir> <oldName> <newName>]
        [-rm [-f] [-r|-R] [-skipTrash] <src> ...]
        [-rmdir [--ignore-fail-on-non-empty] <dir> ...]
...

hadoop fs -put

Put allows uploading a file from your local file system to the hadoop file system.

hadoop fs -put <local-src> <hdfs-destination>

$ echo "hello world" > a.txt
$ hadoop fs -put a.txt /b.txt

hadoop fs -ls

Once the file is uploaded we can check that it exists using the hadoop ls command, that lists the contents of a directory.

hadoop fs -ls <path>

$ hadoop fs -ls /
Found 1 items
-rw-r--r--   1 hduser supergroup         12 2016-08-04 12:59 /b.txt
$ hadoop fs -cat /b.txt
hello world
$ hadoop fs -du /
12  /b.txt

hdfs Copy File to New Directory

For copying a file to a new directory we will use two commands, mkdir and cp.

hadoop fs -mkdir <foldername> - Creates a new directory.

hadoop fs -cp <src> <destintation> - Copies a file from a src path to a destination.

$ hadoop fs -mkdir /mydir
$ hadoop fs -cp /b.txt /mydir/c.txt
$ hadoop fs -ls /mydir
Found 1 items
-rw-r--r--   1 hduser supergroup         12 2016-08-04 13:06 /mydir/c.txt

hadoop fs -get

To download a file from the hadoop file system to our local file system we can use the get command.

hadoop fs -get <hdfs-src> <local-destination>

$ hadoop fs -get /mydir/c.txt he.txt
$ cat he.txt
hello world

hadoop fsck Commands

Another useful command in hadoop is the fsck command. We can for instance use it to get details on blocks and files:

$ hdfs fsck / -files -blocks -locations
Connecting to namenode via http://localhost:50070
FSCK started by hduser (auth:SIMPLE) from /127.0.0.1 for path / at Thu Aug 04 13:09:50 UTC 2016
/ <dir>
/b.txt 12 bytes, 1 block(s):  OK
0. BP-1463267769-172.31.17.190-1470307592433:blk_1073741825_1001 len=12 repl=1 [127.0.0.1:50010]

/mydir <dir>
/mydir/c.txt 12 bytes, 1 block(s):  OK
0. BP-1463267769-172.31.17.190-1470307592433:blk_1073741826_1002 len=12 repl=1 [127.0.0.1:50010]

Status: HEALTHY
 Total size:    24 B
 Total dirs:    2
 Total files:   2
 Total symlinks:                0
 Total blocks (validated):      2 (avg. block size 12 B)
 Minimally replicated blocks:   2 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    1
 Average block replication:     1.0
 Corrupt blocks:                0
 Missing replicas:              0 (0.0 %)
 Number of data-nodes:          1
 Number of racks:               1
FSCK ended at Thu Aug 04 13:09:50 UTC 2016 in 2 milliseconds


The filesystem under path '/' is HEALTHY

Delete Files and Directories

To delete a file or directory from the hadoop file system we use the rm command.

hadoop fs -rm <file>
hadoop fs -rm -r <folder>

$ hadoop fs -ls /
Found 2 items
-rw-r--r--   1 hduser supergroup         12 2016-08-04 12:59 /b.txt
drwxr-xr-x   - hduser supergroup          0 2016-08-04 13:06 /mydir
$ hadoop fs -rm /b.txt
16/08/04 13:15:39 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
Deleted /b.txt
$ hadoop fs -rm -r /mydir
16/08/04 13:15:49 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
Deleted /mydir
$ hadoop fs -ls /

Summary of Commands Used

hadoop fs -help

echo "hello world" > a.txt
hadoop fs -put a.txt /b.txt

hadoop fs -ls /
hadoop fs -cat /b.txt
hadoop fs -du /

# hdfs Copy File to New Directory
hadoop fs -mkdir /mydir
hadoop fs -cp /b.txt /mydir/c.txt
hadoop fs -ls /mydir

hadoop fs -get /mydir/c.txt he.txt
cat he.txt


hdfs fsck / -files -blocks -locations

# DELETE File/Folders
hadoop fs -ls /
hadoop fs -rm /b.txt
hadoop fs -rm -r /mydir
hadoop fs -ls /

Hadoop HDFS Commands

Contents

Running Hadoop HDFS Commands