QShell – LS Consulting

I have a large number of stream files that need to be imported into a database file so that they can be reprocessed. What makes this fun is that the data processing is quite slow and I have a lot of data to load. Because of this, I need to know the total number of lines across multiple stream files so that I can ensure that I don’t exceed the available processing window.

It turns out that Qshell has the answer.

This POSIX compliant command environment provides a set of commands and utilities that allow you to, among other things, manage files in any IFS supported file system.

The wc utility…

displays the number of lines, words, and bytes contained in each input file (or standard input, by default) to standard output. A line is defined as a string of characters delimited by a newline character. A word is defined as a string of characters delimited by white space characters. If more than one input file is specified, a line of cumulative counts for all the files is displayed on a separate line after the output for the last file.

In my case, it’s the number of lines I want to know about, so I need to use the -l parameter.

To get a list of all files, their line counts and a final total, this command:

wc -l IMP*

… gives me this output:

    ...
    2028 IMP2016W48.TXT
    2034 IMP2016W49.TXT
    2662 IMP2016W50.TXT
    3807 IMP2016W51.TXT
    2848 IMP2016W52.TXT
    4220 IMP2016W53.TXT
  503232 total

And, as with every Qshell output, this can be directed to a text file which I can use to handle the information automatically.

It’s worth noting that the wildcard (*) can go anywhere. So this command:

wc -l IMP*M*

… looks for file names starting with IMP and with a M somewhere in the filename:

    5987 IMP2016M01.TXT
    9564 IMP2016M02.TXT
    9344 IMP2016M03.TXT
    5757 IMP2016M04.TXT
    3263 IMP2016M05.txt
    2521 IMP2016M06.txt
    5807 IMP2016M07.TXT
    6456 IMP2016M08.TXT
   10439 IMP2016M09.TXT
    8823 IMP2016M10.TXT
    8119 IMP2016M11.TXT
    8007 IMP2016M12.TXT
   84087 total

Qshell provides a powerful and flexible environment and is well worth getting to grips with if you want to effectively manage files in the IFS.

It turns out that there is no simple way of generating a list of files currently residing in a folder on the IBM i IFS. Simple, in this case, would be a command like DSPLNK OUTPUT(*FILE). An API does exist, but it turns out that the same results can be achieved both quickly and simply if you are willing to spend a bit of time in Qshell.

This post is a simplified reworking of an issue I encountered some time ago, but it’s worth documenting.

The issue is that I have a number of files being sent, via FTP, to a folder in the IFS. I need to be able to copy these files into an import file for subsequent processing and then archive them. The problem, of course, is that I don’t know what the file names will be beforehand.

Here’s a solution:

First create an extract file in QTEMP. You will need this in order to compile the program. If you are going to submit the compile in batch, you will need to create the file in some other library:

crtpf file(QTEMP/EXTRACTP) rcdlen(20)

And then the CL program:

pgm                                                                             
dcl &fromfile *char 50                                                          
dcl &tombr *char 50 value('/qsys.lib/LSCLIB.lib/IMPORTP.file/IMPORTP.mbr')  
dcl &dltfile *char 100                                                          
dcl &cpyfile *char 100                                                          
dclf EXTRACTP                                                                   
                                                                                
/* First, I use QShell to list the files in the Extract folder                */
/* The output of this is redirected to ExtractedFiles.txt.                    */
qsh cmd('ls /home/PAUL/Extract/ > /home/PAUL/ExtractedFiles.txt')     
                                                                                
/* In order to use this information, create the extract file in qtemp and     */
/* copy the ExtractedFiles.txt stream file into it.                           */
/* If the file already exists, the MONMSG clears it.                          */
crtpf file(QTEMP/EXTRACTP) rcdlen(20)                                           
monmsg CPF7302 exec(clrpfm QTEMP/EXTRACTP)                                      
cpyfrmstmf fromstmf('/home/PAUL/ExtractedFiles.txt') +                     
           tombr('/qsys.lib/qtemp.lib/EXTRACTP.file/EXTRACTP.mbr') +            
           mbropt(*REPLACE)                                                     
                                                                              
/* And now I can use QTEMP/EXTRACTP to drive my way through the Extract       */
/* folder                                                                     */
dowhile '1'                                                                     
    rcvf                                                                        
    monmsg msgid(CPF0864) exec(LEAVE)                                           
                                                                                
    /* Copy the next stream file to the IMPORTP physical file                 */
    chgvar &fromfile value('/home/PAUL/Extract/' *tcat &EXTRACTP)          
    cpyfrmstmf fromstmf(&fromfile) tombr(&tombr) mbropt(*add) +                 
               STMFCCSID(819)                                                   
                                                                                
    /* Copy the stream file to the Archive                                    */
    chgvar &cpyfile value('cp /home/PAUL/Extract/' *tcat &EXTRACTP +       
                    *bcat '/home/PAUL/Archive')                            
    qsh cmd(&cpyfile)                                                           
                                                                                
    /* Then remove the stream file from the Extract folder                    */
    chgvar &dltfile value('rm /home/PAUL/Extract/' *tcat &EXTRACTP)        
    qsh cmd(&dltfile)         
                                                   
    enddo                                          
                                                   
/* Clean up and exit                               
qsh cmd('rm /home/PAUL/ExtractedFiles.txt')   
dltf qtemp/EXTRACTP                                
                                                   
endpgm

It should go without saying that some of the names have been changed and that the above program should be treated as a sample only.

Being able to move information between the QShell/IFS and traditional i on Power environments is both useful and (in my experience) increasingly important. Although it does take a bit of thinking about, it isn’t difficult which is why I find that the oft-seen solution of “buy this tool” is both disappointing and (often) overkill.

Tag: QShell

Counting lines in QShell

Using Qshell and CL to work with Stream files in the IFS