Bad joining of STDERR + STDOUT in windows [rt.cpan.org #55164]

Migrated from rt.cpan.org#55164 (status was 'open')

Requestors:

• alexandrfedorov@gmail.com

From alexandrfedorov@gmail.com on 2010-03-03 05:51:37:

I'm trying to execute windows utility "wmic", if success it's return
data in utf-16 encoding, in some sort of errors - utf-16 too, and
sometimes in ANSI encoding.
When error is going with utf-16 encoding, BOM header in printed to
STDOUT and all that remains is printed to STDERR. When using qxx sub BOM
header - don't join to STDERR stream and i can't simply decode that
string, please repair this. 

From dagolden@cpan.org on 2010-03-03 13:28:27:

On Wed, Mar 3, 2010 at 12:51 AM, Alexandr Fedorov via RT
<bug-IO-CaptureOutput@rt.cpan.org> wrote:
> I'm trying to execute windows utility "wmic", if success it's return
> data in utf-16 encoding, in some sort of errors - utf-16 too, and
> sometimes in ANSI encoding.
> When error is going with utf-16 encoding, BOM header in printed to
> STDOUT and all that remains is printed to STDERR. When using qxx sub BOM
> header - don't join to STDERR stream and i can't simply decode that
> string, please repair this.

IO::CaptureOutput is not getting active support these days as I've
been working on a replacement called Capture::Tiny.

Capture::Tiny has some better support for utf, but I don't know if it
will solve your particular problem.  Would you mind trying it and
letting me know?

Also, per your other ticket, getting quoting right on Windows is hard.
 I suggest you try either IPC::System::Simple or IPC::Cmd and see if
they get it right. Then, with Capture:Tiny, your code would be like
this:

  use IPC::System::Simple 'systemx';
  use Capture::Tiny 'capture';
  my ($output, $error) = capture { systemx($cmd, @args) };

You might also want to read http://www.perlmonks.org/?node_id=722963
which suggests that piping wmic to another process forces ANSI output,
which might help you.

-- David

From alexandrfedorov@gmail.com on 2010-03-17 20:19:01:

Срд Мар 03 08:28:27 2010, DAGOLDEN писал:
> On Wed, Mar 3, 2010 at 12:51 AM, Alexandr Fedorov via RT

>   use IPC::System::Simple 'systemx';
>   use Capture::Tiny 'capture';
>   my ($output, $error) = capture { systemx($cmd, @args) };

I tried this and many other variants... There is some problem with your
module and IPC::System::Simple,
1. My script in utf8 encoding
2. I'm using utf8 pragma
3. I'm running something like this:

$rslt = capture_merged {
    systemx('cmd','/c',encode 'cp1251','echo Не работает');
};

(important part - with cyrillic letters (in utf8 encoding))
4. script silently dies

If i add:
binmode $_,':encoding(866)' for *STDOUT,*STDERR

script works.
Can you explain for me this?

With Perl's "system"  your module works well.

From dagolden@cpan.org on 2010-03-18 03:15:26:

Alexandr, I have to admit I'm stumped.  So I'm copying the author of
IPC::System::Simple in the hopes that he has some ideas about what is
happening.

Paul, please see the conversation below and/or the original RT ticket.
 It seems like IPC::System::Simple might be doing something to the
output handles.

Regards,

-- David


On Wed, Mar 17, 2010 at 4:19 PM, Alexandr Fedorov via RT
<bug-IO-CaptureOutput@rt.cpan.org> wrote:
>       Queue: IO-CaptureOutput
>  Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=55164 >
>
> Срд Мар 03 08:28:27 2010, DAGOLDEN писал:
>> On Wed, Mar 3, 2010 at 12:51 AM, Alexandr Fedorov via RT
>
>>   use IPC::System::Simple 'systemx';
>>   use Capture::Tiny 'capture';
>>   my ($output, $error) = capture { systemx($cmd, @args) };
>
> I tried this and many other variants... There is some problem with your
> module and IPC::System::Simple,
> 1. My script in utf8 encoding
> 2. I'm using utf8 pragma
> 3. I'm running something like this:
>
> $rslt = capture_merged {
>        systemx('cmd','/c',encode 'cp1251','echo Не работает');
> };
>
> (important part - with cyrillic letters (in utf8 encoding))
> 4. script silently dies
>
> If i add:
> binmode $_,':encoding(866)' for *STDOUT,*STDERR
>
> script works.
> Can you explain for me this?
>
> With Perl's "system"  your module works well.
>

From pjf@cpan.org on 2010-03-19 01:11:53:

G'day Alexandr / David,

On Wed Mar 17 23:15:26 2010, DAGOLDEN wrote:

> Paul, please see the conversation below and/or the original RT ticket.
>  It seems like IPC::System::Simple might be doing something to the
> output handles.

If we were using IPC::System::Simple's capture/capturex here, then I'd
say yes absolutely weird things are happening to output filehandles. 
But we're not. We're using systemx(), and letting Capture::Tiny grab the
output.

For systemx() on Windows, we use Win32::Process::Create to spawn the
command, but otherwise don't touch filehandles at all.  There is a bit
of extra poking around to try and determine where that could should be
($ENV{PATH} searching is normally done by the shell, but none of that
involves changing filehandles.

I'd be *very* interested to see if a call to Win32::Process::Create()
has the same issues, since that's what I'm fundamentally using underneath.

Best wishes,

   Paul