Counting lines in QShell

I have a large number of stream files that need to be imported into a database file so that they can be reprocessed. What makes this fun is that the data processing is quite slow and I have a lot of data to load. Because of this, I need to know the total number of lines across multiple stream files so that I can ensure that I don’t exceed the available processing window.

It turns out that Qshell has the answer.

This POSIX compliant command environment provides a set of commands and utilities that allow you to, among other things, manage files in any IFS supported file system.

The wc utility…

displays the number of lines, words, and bytes contained in each input file (or standard input, by default) to standard output. A line is defined as a string of characters delimited by a newline character. A word is defined as a string of characters delimited by white space characters. If more than one input file is specified, a line of cumulative counts for all the files is displayed on a separate line after the output for the last file.

In my case, it’s the number of lines I want to know about, so I need to use the -l parameter.

To get a list of all files, their line counts and a final total, this command:

wc -l IMP*

… gives me this output:

    ...
    2028 IMP2016W48.TXT
    2034 IMP2016W49.TXT
    2662 IMP2016W50.TXT
    3807 IMP2016W51.TXT
    2848 IMP2016W52.TXT
    4220 IMP2016W53.TXT
  503232 total

And, as with every Qshell output, this can be directed to a text file which I can use to handle the information automatically.

It’s worth noting that the wildcard (*) can go anywhere. So this command:

wc -l IMP*M*

… looks for file names starting with IMP and with a M somewhere in the filename:

    5987 IMP2016M01.TXT
    9564 IMP2016M02.TXT
    9344 IMP2016M03.TXT
    5757 IMP2016M04.TXT
    3263 IMP2016M05.txt
    2521 IMP2016M06.txt
    5807 IMP2016M07.TXT
    6456 IMP2016M08.TXT
   10439 IMP2016M09.TXT
    8823 IMP2016M10.TXT
    8119 IMP2016M11.TXT
    8007 IMP2016M12.TXT
   84087 total

Qshell provides a powerful and flexible environment and is well worth getting to grips with if you want to effectively manage files in the IFS.