cygwin / mintty stdin is not readable with libuv

[papamarkou]

Towards the end of Julia's native build on Win7 (x64) using msys2 I got the following segmentation fault (I used the Julia commit d073213, which builds without giving this segfault on msys, so the error is msys2-specific) :

C:\mingw64\bin\ar.exe: creating librbio.a
    CC src/jltypes.o
    CC src/gf.o
    CC src/support/hashing.o
    CC src/support/timefuncs.o
    CC src/support/ptrhash.o
    CC src/support/operators.o
    CC src/support/utf8.o
    CC src/support/ios.o
    CC src/support/htable.o
    CC src/support/bitvector.o
    CC src/support/int2str.o
    CC src/support/libsupportinit.o
    CC src/support/arraylist.o
    CC src/support/asprintf.o
    CC src/support/wcwidth.o
    CC src/support/_setjmp.win64.o
    CC src/support/_longjmp.win64.o
    LINK src/support/libsupport.a
    CC src/flisp/flisp.o
    CC src/flisp/builtins.o
    CC src/flisp/string.o
    CC src/flisp/equalhash.o
    CC src/flisp/table.o
    CC src/flisp/iostream.o
    CC src/flisp/julia_extensions.o
    LINK src/flisp/libflisp.a
    CC src/flisp/flmain.o
    CC src/flisp/flisp
    FLISP src/julia_flisp.boot
    FLISP src/julia_flisp.boot.inc
    CC src/ast.o
    CC src/builtins.o
    CC src/module.o
    CC src/codegen.o
    CC src/interpreter.o
    CC src/alloc.o
    CC src/dlload.o
    CC src/sys.o
    CC src/init.o
    CC src/task.o
    CC src/array.o
    CC src/dump.o
    CC src/toplevel.o
    CC src/jl_uv.o
    CC src/jlapi.o
    CC src/profile.o
    CC src/gc.o
    LINK usr/bin/libjulia.dll
    PERL base/pcre_h.jl
    PERL base/errno_h.jl
    PERL base/build_h.jl.phony
    PERL base/fenv_constants.jl
    PERL base/file_constants.jl
    PERL base/uv_constants.jl
    CC ui/repl.o
    CC ui/repl-readline.o
    LINK usr/bin/julia-readline.exe
    CC ui/repl-basic.o
    LINK usr/bin/julia-basic.exe
/bin/sh: line 1:  6264 Segmentation fault      /home/theodore/opt/julia/usr/bin/julia-readline.exe -bf sysimg.jl
Makefile:69: recipe for target '/home/theodore/opt/julia/usr/bin/sys0.ji' failed
make[1]: *** [/home/theodore/opt/julia/usr/bin/sys0.ji] Error 139
Makefile:34: recipe for target 'release' failed
make: *** [release] Error 2

[JeffBezanson]

@vtjnash what do we make of this?

[vtjnash]

that you shouldn't use windows for serious work?

oh, you meant about getting it fixed.

i stopped looking at all issue reports after the summer, so I don't think i saw this one.

I also haven't built julia for windows since the summer -- loladiro switch the virtualization program, and I couldn't get windows to install correctly (although I did find at least half a dozen ways to corrupt or crash the installer). I'll PM you.

[quinnj]

FWIW, I've continued to build on windows without any problems. Though it seems every once in a while something gets screwy and I have to start over from scratch. At those times, I will usually make sure my msys2 build is on the latest (new releases get pushed pretty frequently) and just start the build from scratch. I don't think I've run into anything totally blocking for a while now. I'm on windows 8 though, so I'm not sure if that makes a difference. What msys2 release are you using?

[ViralBShah]

@loladiro How do we fix the virtualization stuff?

@vtjnash Would it help if I ship you a laptop for running windows?

[vtjnash]

the virtualization program is excellent. The problem was that windows didn't support most of the options. For example: if the installation disk was smaller than 90GB, it wouldn't let me log in after the installation was complete. If I forgot the network card, the setup program after installation would crash and I would have to reinstall from scratch. If I emulated Haswell, or chose the wrong type of hardware, the desktop would crash after setup and then it couldn't boot and I would have to reinstall from scratch. After a few hours, I ran out of time to deal with it.

@ViralBShah no. the problem is that I don't like using window, so I don't have much time or interest to work on supporting it. I have plenty of licenses and virtual installations of windows around. Remote-ing into a virtual setup reduced that effort somewhat.

@karbarcca I recall something about PCRE failing tests when built this way, was that ever resolved?

I agree that rebuilding from scratch is necessary more frequently on Windows. I have not had good experiences when switching compilers, so if you change versions of mingw, it is sometimes also necessary to rebuild-the-world.

[quinnj]

Yeah, the failing PCRE tests is the one additional thing when building with msys2. The solution is to just put an empty file named checked in the PCRE directory.

[StefanKarpinski]

Wow, that's a really weird solution.

[Keno]

Not really, that's what our build system does to avoid checking everything every time you do make.

[ViralBShah]

Does upgrading to the latest pcre help?

[StefanKarpinski]

Ah, that makes sense.

On Nov 24, 2013, at 11:08 PM, Keno Fischer [email protected] wrote:

Not really, that's what our build system does to avoid checking everything every time you do make.

—
Reply to this email directly or view it on GitHub.

[quinnj]

@ViralBShah, I think somebody tried the latest PCRE, but it didn't seem to make a difference.

[papamarkou]

I haven't built Julia on Windows recently to contribute to the discussion (due to moving house and changing academic institution I have access to a Linux machine only at the moment). I will let you know if I get the chance to rebuild on Windows.

[ihnorton]

I tried upgrading pcre and it didn't help. I decided to give up on msys2 for a while after BSOD'ing my laptop three times during builds (the only time this has ever happened on Win7). Should probably try again though, because they were fixing things quickly.

[quinnj]

@Scidom, when you're back up and running, I'd suggest going with @Alexpux's suggestion for mingw64 build here:
http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/mingw-builds/4.8.2/threads-win32/seh/

I've been building native windows 8.1 with that mingw64 and msys2.

[Alexpux]

Hi, all!
I'm switch MSYS2 disturb to use package manager ported from Arch Linux - pacman. Now I don't release full msys2 archives - just base system (https://sourceforge.net/projects/msys2/files/Base/) that can be easily extended with packages that you want and upgraded when you want. Also I create two repositories with mingw packages 32 and 64 bit where prebuilded toolchains located too.

Regards,
Alexey.

[vtjnash]

I think you need the sjlj tool chain, or c++ code may crash, because of limitations of error handlers on JIT code

Msys2 just keeps getting better! Thanks Alexey.

[vtjnash]

Correction: dwarf may have issues with JIT code. seh should be fine. (sjlj is also fine).

I tried to test this, but couldn't get LLVM to compile on my XP-64 VM.

[jiahao]

I got a bit further with mingw-build/x64-4.8.1-win32-seh-rev5 and msys2/x86_64-20131113:

$make debug
...
LINK usr/bin/julia-debug-basic.exe
Error initializing stdio in uv_pipe_open (5, 7)
/bin/sh: line 1:  4108 Segmentation fault     /home/julia/julia/usr/bin/julia-debug-readline.exe --build /home/julia/julia/usr/bin/sys0 sysimg.jl
...
$gdb --args /home/julia/julia/usr/bin/julia-debug-readline.exe --build /home/julia/julia/usr/bin/sys0 sysimg.jl
...
(gdb) r
...
Program received signal SIGSEGV, Segmentation fault.
0x000000006369443a in jl_enter_handler (eh=0xeff8f0) in builtins.c:97
97         eh->prev = jl_current_task->eh;
(gdb) bt
#0  0x000000006369443a in jl_enter_handler (eh=0xeff8f0) at builtins.c:97
#1  0x00000000636cb0c9 in uv_atexit_hook() at init.c:352
#2  0x00000000636db400 in jl_exit (exitcode=1) at jl_uv.c:643
#3  0x00000000636941cc in jl_errorf (
    fmt=0x63fe22f8 <llvm::sys::IsLittleEndianHost+2843> "Error initializing stdio in uv_pipe_open (%d, %d)\n") at builtins.c:54
#4 0x00000000636cb412 in init_stdio_handle (fd=5, readable=1) at init.c:468
#5 0x00000000636cb51e in init_stdio() at init.c:498
#6 0x00000000636cb710 in julia_init (imageFile=0x0, build_mode=1)
   at init.c:637
#7 0x00000000004020e5 in __fu19___stack_chk_guard () at repl.c:268
(gdb) p jl_current_task
$1 = (jl_task_t * volatile) 0x0

[ihnorton]

I can replicate @jiahao's error in both cross-compile and mingw64 build.

The uv_pipe_open failure is in SetNamedHandleState:

Breakpoint 3, uv_set_pipe_handle (handle=handle@entry=0x3ae5c0,
    pipeHandle=pipeHandle@entry=0x1e0, duplex_flags=duplex_flags@entry=0,
    loop=<optimized out>) at src/win/pipe.c:354
354     static int uv_set_pipe_handle(uv_loop_t* loop, uv_pipe_t* handle,
(gdb) n
361       if (!SetNamedPipeHandleState(pipeHandle, &mode, NULL, NULL)) {

However, it appears that the underlying cause may be stack corruption in uv_pipe_init. This might also explain why the error handler is segfaulting.

Breakpoint 1, init_stdio_handle (fd=1, readable=0) at init.c:427
427     {
(gdb) info frame
Stack level 0, frame at 0xe3fbf0:
 rip = 0x6e40c3f8 in init_stdio_handle (init.c:427); saved rip 0x6e40c68d
 called by frame at 0xe3fc30
 source language c.
 Arglist at 0xe3fbe0, args: fd=1, readable=0
 Locals at 0xe3fbe0, Previous frame's sp is 0xe3fbf0
 Saved registers:
  rip at 0xe3fbe8, xmm15 at 0xe3fbe8
(gdb) n
429         uv_handle_type type = uv_guess_handle(fd);
(gdb) n
443         switch(type) {
(gdb) n
462                 handle = malloc(sizeof(uv_pipe_t));
(gdb) n
463                 if (uv_pipe_init(jl_io_loop, (uv_pipe_t*)handle, (readable?UV_PIPE_READABLE:UV_PIPE_WRITABLE))) {
(gdb) n
467                 if (uv_pipe_open((uv_pipe_t*)handle,fd)) {
(gdb) info frame
Stack level 0, frame at 0xe3fbf0:
 rip = 0x6e40c557 in init_stdio_handle (init.c:467); saved rip 0x6e40c68d
 called by frame at 0xe3fc30
 source language c.
 Arglist at 0xe3fb80, args: fd=0, readable=1
 Locals at 0xe3fb80, Previous frame's sp is 0xe3fbf0
 Saved registers:
  rbx at 0xe3fbd8, rbp at 0xe3fbe0, rip at 0xe3fbe8, xmm15 at 0xe3fbe8

The value of fd is different after uv_pipe_init. If I change fd back with set variable fd=1 then it doesn't fail at SetNamedPipeHandleState, and I can proceed into image loading one command at a time (didn't go too far). However, if I continue then I get

(gdb) cont
Continuing.
[New Thread 308744.0x9f70]
[Inferior 1 (process 308744) exited with code 01]

@jiahao and I are using different mingw and msys builds, although both fairly new. However, this also fails with a known-good xcompile setup. I also tried building v0.2.0 with this stack and I get the same segfault as seen at the top of this thread. So this is weird.

[ihnorton]

I can run image creation and the resulting julia successfully under msys-0828 after compiling with both x86_64-4.8.2-release-win32-seh-rt_v3 and x64-4.8.1-release-win32-seh-rev5. under msys-1208.

The same binary segfaults with same error under msys-1208.

This points toward a library problem/conflict with something from msys, but I'm unsure what library. Still baffled at the segfault in cross-compile.

[Alexpux]

did you update MSYS2 packages via pacman?

[vtjnash]

@ihnorton what happened to the call fd = _dup(fd) that should have happened after entering the init_stdio_handle function and before uv_pipe_init?

[ihnorton]

@vtjnash I had commented it out for that build, but it doesn't make a difference.

@Alexpux no. trying that now.

[vtjnash]

I see what is going wrong, I'm attempting a fix now.

[vtjnash]

The problem occurs because of a recent (circa 2011?) change to cygwin / mintty that started to create stdin as a pipe. This would be fine, except cygwin doesn't give libuv enough permission to configure the read end of the pipe (we expect to have FILE_WRITE_ATTRIBUTES, which allows us to ensure the pipe is in blocking, byte-mode), so it gets ERROR_ACCESS_DENIED when it tries to configure the pipe.

I would classify this as a bug in cygwin since there isn't much the client can do about it, except stay within the cygwin sandbox.

It I disable the error check, Julia works, although it doesn't print the prompt since it detects that it is reading from a pipe.

[vtjnash]

We can detect that it is a cygwin pty (instead of a pipe) by using the following patch which compares the name of the pipe server against the following regex ^\\cygwin-[^-]+-[pt]ty[1-9][0-9]*-
http://sourceware-org.1504.n7.nabble.com/PATCH-0-3-V3-Test-mingw32-GDB-in-cygwin-td239693.html

[vtjnash]

I looked into this more today -- the issues here are actually with our hacked up version of readline, cygwin, and mintty. But by using a normal version of readline with a normal version of ncurses, and the following command line to start julia (using a cygwin stty program), we can get closer to a working repl prompt in mintty:
TERM=xterm ./julia-readline -P 'run(stty -icanon -echo -icrnl -inlcr)'