| Demonstrations of dirtop, the Linux eBPF/bcc version. |
| |
| |
| dirtop shows reads and writes by directory. For example: |
| |
| # ./dirtop.py -d '/hdfs/uuid/*/yarn' |
| Tracing... Output every 1 secs. Hit Ctrl-C to end |
| |
| 14:28:12 loadavg: 25.00 22.85 21.22 31/2921 66450 |
| |
| READS WRITES R_Kb W_Kb PATH |
| 1030 2852 8 147341 /hdfs/uuid/c11da291-28de-4a77-873e-44bb452d238b/yarn |
| 3308 2459 10980 24893 /hdfs/uuid/bf829d08-1455-45b8-81fa-05c3303e8c45/yarn |
| 2227 7165 6484 11157 /hdfs/uuid/76dc0b77-e2fd-4476-818f-2b5c3c452396/yarn |
| 1985 9576 6431 6616 /hdfs/uuid/99c178d5-a209-4af2-8467-7382c7f03c1b/yarn |
| 1986 398 6474 6486 /hdfs/uuid/7d512fe7-b20d-464c-a75a-dbf8b687ee1c/yarn |
| 764 3685 5 7069 /hdfs/uuid/250b21c8-1714-45fe-8c08-d45d0271c6bd/yarn |
| 432 1603 259 6402 /hdfs/uuid/4a833770-767e-43b3-b696-dc98901bce26/yarn |
| 993 5856 320 129 /hdfs/uuid/b94cbf3f-76b1-4ced-9043-02d450b9887c/yarn |
| 612 5645 4 249 /hdfs/uuid/8138a53b-b942-44d3-82df-51575f1a3901/yarn |
| 818 21 6 166 /hdfs/uuid/fada8004-53ff-48df-9396-165d8e42925b/yarn |
| 174 23 1 171 /hdfs/uuid/d04fccd8-bc72-4ed9-bda4-c5b6893f1405/yarn |
| 376 6281 2 97 /hdfs/uuid/0cc3683f-4800-4c73-8075-8d77dc7cf116/yarn |
| 370 4588 2 96 /hdfs/uuid/a78f846a-58c4-4d10-a9f5-42f16a6134a0/yarn |
| 190 6420 1 86 /hdfs/uuid/2c6a7223-cb18-4916-a1b6-8cd02bda1d31/yarn |
| 178 123 1 17 /hdfs/uuid/b3b2a2ed-f6c1-4641-86bf-2989dd932411/yarn |
| [...] |
| |
| This shows various directories read and written when hadoop runs. |
| By default the output is sorted by the total read size in Kbytes (R_Kb). |
| Sorting order can be changed via -s option. |
| This is instrumenting at the VFS interface, so this is reads and writes that |
| may return entirely from the file system cache (page cache). |
| |
| While not printed, the average read and write size can be calculated by |
| dividing R_Kb by READS, and the same for writes. |
| |
| This script works by tracing the vfs_read() and vfs_write() functions using |
| kernel dynamic tracing, which instruments explicit read and write calls. If |
| files are read or written using another means (eg, via mmap()), then they |
| will not be visible using this tool. |
| |
| This should be useful for file system workload characterization when analyzing |
| the performance of applications. |
| |
| Note that tracing VFS level reads and writes can be a frequent activity, and |
| this tool can begin to cost measurable overhead at high I/O rates. |
| |
| |
| A -C option will stop clearing the screen, and -r with a number will restrict |
| the output to that many rows (20 by default). For example, not clearing |
| the screen and showing the top 5 only: |
| |
| # ./dirtop -d '/hdfs/uuid/*/yarn' -Cr 5 |
| Tracing... Output every 1 secs. Hit Ctrl-C to end |
| |
| 14:29:08 loadavg: 25.66 23.42 21.51 17/2850 67167 |
| |
| READS WRITES R_Kb W_Kb PATH |
| 100 8429 0 48243 /hdfs/uuid/b94cbf3f-76b1-4ced-9043-02d450b9887c/yarn |
| 2066 4091 8176 26457 /hdfs/uuid/d04fccd8-bc72-4ed9-bda4-c5b6893f1405/yarn |
| 10 2043 0 8172 /hdfs/uuid/b3b2a2ed-f6c1-4641-86bf-2989dd932411/yarn |
| 38 1368 0 2652 /hdfs/uuid/a78f846a-58c4-4d10-a9f5-42f16a6134a0/yarn |
| 86 19 0 123 /hdfs/uuid/c11da291-28de-4a77-873e-44bb452d238b/yarn |
| |
| 14:29:09 loadavg: 25.66 23.42 21.51 15/2849 67170 |
| |
| READS WRITES R_Kb W_Kb PATH |
| 1204 5619 4388 33767 /hdfs/uuid/b94cbf3f-76b1-4ced-9043-02d450b9887c/yarn |
| 2208 3511 8744 22992 /hdfs/uuid/d04fccd8-bc72-4ed9-bda4-c5b6893f1405/yarn |
| 62 4010 0 21181 /hdfs/uuid/8138a53b-b942-44d3-82df-51575f1a3901/yarn |
| 22 2187 0 8748 /hdfs/uuid/b3b2a2ed-f6c1-4641-86bf-2989dd932411/yarn |
| 74 1097 0 4388 /hdfs/uuid/4a833770-767e-43b3-b696-dc98901bce26/yarn |
| |
| [..] |
| |
| |
| |
| USAGE message: |
| |
| # ./dirtop.py -h |
| usage: dirtop.py [-h] [-C] [-r MAXROWS] [-s {all,reads,writes,rbytes,wbytes}] |
| [-p PID] -d ROOTDIRS |
| [interval] [count] |
| |
| File reads and writes by process |
| |
| positional arguments: |
| interval output interval, in seconds |
| count number of outputs |
| |
| optional arguments: |
| -h, --help show this help message and exit |
| -C, --noclear don't clear the screen |
| -r MAXROWS, --maxrows MAXROWS |
| maximum rows to print, default 20 |
| -s {all,reads,writes,rbytes,wbytes}, --sort {all,reads,writes,rbytes,wbytes} |
| sort column, default all |
| -p PID, --pid PID trace this PID only |
| -d ROOTDIRS, --root-directories ROOTDIRS |
| select the directories to observe, separated by commas |
| |
| examples: |
| ./dirtop -d '/hdfs/uuid/*/yarn' # directory I/O top, 1 second refresh |
| ./dirtop -d '/hdfs/uuid/*/yarn' -C # don't clear the screen |
| ./dirtop -d '/hdfs/uuid/*/yarn' 5 # 5 second summaries |
| ./dirtop -d '/hdfs/uuid/*/yarn' 5 10 # 5 second summaries, 10 times only |
| ./dirtop -d '/hdfs/uuid/*/yarn,/hdfs/uuid/*/data' # Running dirtop on two set of directories |
