Recipe 16.9 Controlling the Input, Output, and Error of Another Program

16.9.1 Problem

You want full control over a command's input, output, and error streams.

16.9.2 Solution

Carefully use the standard IPC::Open3 module, possibly in conjunction with the standard IO::Select module.

16.9.3 Discussion

If you're interested in only one of the program's STDIN, STDOUT, or STDERR, the task is simple. When you want to manage two or more of these, however, it abruptly stops being simple. Multiplexing multiple I/O streams is never a pretty picture. Here's an easy workaround:

@all = `($cmd | sed -e 's/^/stdout: /' ) 2>&1`;
for (@all) { push @{ s/stdout: // ? \@outlines : \@errlines }, $_ }
print "STDOUT:\n", @outlines, "\n";
print "STDERR:\n", @errlines, "\n";

If you don't have sed on your system, you'll find that for simple cases like this, perl -pe works just as well as sed -e.

However, that's not really simultaneous processing. All we're doing is marking STDOUT lines with "stdout:" and then stripping them back out once we've read all the STDOUT and STDERR the program produced.

You can use the standard IPC::Open3 module for this. Mysteriously, the argument order is different for IPC::Open3 than for IPC::Open2.

open3($write_me, $read_me, $errors, "program to run");

Using this has even more potential for chaos than using open2. If you're reading the program's STDERR as it is trying to write more than one buffer's worth to its STDOUT, the program will block on the write because its buffers are full, and you will block on the read because there's nothing available.

You can avoid this deadlock by mimicking open3 with fork, open, and exec; making all filehandles unbuffered; and using sysread, syswrite, and select to decide which readable filehandle to read a byte from. This makes your program slower and bulkier, though, and it doesn't solve the classic open2 deadlock where each program is expecting the other to say something.

use IPC::Open3;
$pid = open3($child_in, $child_out, $child_err, $cmd);
close $child_in;  # give end-of-file to kid
@outlines = <$child_out>;              # read till EOF
@errlines = <$child_err>;              # XXX: block potential if massive
print "STDOUT:\n", @outlines, "\n";
print "STDERR:\n", @errlines, "\n";

As if deadlock weren't bad enough, this approach is subtly error-prone. There are at least three worrisome situations: both the child and the parent trying to read at the same time, causing deadlock; full buffers causing the child to block as it tries to write to STDERR while the parent is blocked trying to read from the child's STDOUT; and full buffers causing the parent to block writing to the child's STDIN while the child is blocked writing to either its STDOUT or STDERR. The first problem is generally unsolvable, although you can work around it by setting timers with alarm and preventing blocking operations from restarting if a SIGALRM is received.

We use the IO::Select module (you could also do this with the built-in function select) to learn which filehandles (if any) can be read from. This solves the second problem, but not the third. To solve the third, you also need alarm and non-restarting SIGALRM.

If you want to send input to the program, read its output, and either read or ignore its error, you need to work much harder. (See Example 16-2.)

Example 16-2. cmd3sel

  #!/usr/bin/perl
  # cmd3sel - control all three of kids in, out, and error.
  use IPC::Open3;
  use IO::Select;
  
  $cmd = "grep vt33 /none/such - /etc/termcap";
  $pid = open3($cmd_in, $cmd_out, $cmd_err, $cmd);
  
  $SIG{CHLD} = sub {
      print "REAPER: status $? on $pid\n" if waitpid($pid, 0) > 0
  };
  
  print $cmd_in "This line has a vt33 lurking in it\n";
  close $cmd_in;
  
  $selector = IO::Select->new( );
  $selector->add($cmd_err, $cmd_out);
  
  while (@ready = $selector->can_read) {
      foreach $fh (@ready) {
          if (fileno($fh) =  = fileno($cmd_err)) {print "STDERR: ", scalar <$cmd_err>}
          else                                 {print "STDOUT: ", scalar <$cmd_out>}
          $selector->remove($fh) if eof($fh);
      }
  }
  close $cmd_out;
  close $cmd_err;

We sent only a short line as input, then closed the handle. This avoids the deadlock situation of two processes each waiting for the other to write something.

16.9.4 See Also

The documentation for the standard IO::Select, IPC::Open2, and IPC::Open3 modules; the alarm function in Chapter 29 of Programming Perl or perlfunc(1); Recipe 16.8; Recipe 16.15; Recipe 16.16