I have a large number of stream files that need to be imported into a database file so that they can be reprocessed. What makes this fun is that the data processing is quite slow and I have a lot of data to load. Because of this, I need to know the total number of lines across multiple stream files so that I can ensure that I don’t exceed the available processing window.
It turns out that Qshell has the answer.
This POSIX compliant command environment provides a set of commands and utilities that allow you to, among other things, manage files in any IFS supported file system.
The wc utility…
displays the number of lines, words, and bytes contained in each input file (or standard input, by default) to standard output. A line is defined as a string of characters delimited by a newline character. A word is defined as a string of characters delimited by white space characters. If more than one input file is specified, a line of cumulative counts for all the files is displayed on a separate line after the output for the last file.
In my case, it’s the number of lines I want to know about, so I need to use the
To get a list of all files, their line counts and a final total, this command:
wc -l IMP*
… gives me this output:
... 2028 IMP2016W48.TXT 2034 IMP2016W49.TXT 2662 IMP2016W50.TXT 3807 IMP2016W51.TXT 2848 IMP2016W52.TXT 4220 IMP2016W53.TXT 503232 total
And, as with every Qshell output, this can be directed to a text file which I can use to handle the information automatically.
It’s worth noting that the wildcard (*) can go anywhere. So this command:
wc -l IMP*M*
… looks for file names starting with IMP and with a M somewhere in the filename:
5987 IMP2016M01.TXT 9564 IMP2016M02.TXT 9344 IMP2016M03.TXT 5757 IMP2016M04.TXT 3263 IMP2016M05.txt 2521 IMP2016M06.txt 5807 IMP2016M07.TXT 6456 IMP2016M08.TXT 10439 IMP2016M09.TXT 8823 IMP2016M10.TXT 8119 IMP2016M11.TXT 8007 IMP2016M12.TXT 84087 total
Qshell provides a powerful and flexible environment and is well worth getting to grips with if you want to effectively manage files in the IFS.