Upgrade GCC to 5.2.0 for Windows x86 and x86_64
ClosedPublic

Authored by Phyx on Aug 2 2015, 2:39 PM.

Details

Summary

This patch does a few things

  • Moved GHC x86 to MinGW-w64 (Using Awson's patch)
  • Moves Both GHCs to MSYS2 toolchains
  • Completely removes the dependencies on the git tarball repo
    • Downloads only the required tarball for the architecture building for
    • Downloads the perl tarball is missing as well
    • Fixed a few bugs in the linker to fix tests on Windows

The links currently point to repo.msys2.org and GitHub, it might be more desirable to mirror them on
http://downloads.haskell.org/~ghc/mingw/ as with the previous patch attempt.

For more details on what the MSYS2 packages I include see Trac #10726 (Awson's comment). but it should contain all we need
and no python or fortran, which makes the uncompressed tar a 1-2 hundreds mb smaller.

The GCC 5.2.0 in the package supports libgcc as a shared library, this is a problem since
when compiling with -shared the produced dll now has a dependency on libgcc_s_sjlj-1.dll.
To solve this the flag -static-libgcc is now being used for all GCC calls on windows.

Test Plan

./validate was ran both on x86 and x86_64 windows and compared against the baseline.

A few test were failing due to Ld no longer being noisy. These were updated.

The changes to the configure script *should* be validated by the build bots for the other platforms before landing

Diff Detail

Repository
rGHC Glasgow Haskell Compiler
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
Phyx updated this revision to Diff 3734.Aug 2 2015, 2:39 PM
Phyx retitled this revision from to Upgrade GCC to 5.1.1 for Windows x86 and x86_64.
Phyx updated this object.
Phyx edited the test plan for this revision. (Show Details)
Phyx added reviewers: thomie, austin.
Phyx updated the Trac tickets for this revision.
Phyx added a subscriber: awson.
Phyx added a comment.Aug 2 2015, 2:49 PM

Validate on x86_64 looks identical to what it was before the upgrade, but I'm waiting for the latest round to finish.
For x86 I have a new failing test:

ghci/prog003                       prog003 [bad exit code] (ghci)

which is caused by a segfault.

Running this test manually (inputting the commands in ghci) works, but running it via the test script always segfaults..
This test however has and was already failing on x86_64 which was already using MinGW-w64.

I am not sure what to do about this.

The test output is:

Wrong exit code (expected 0 , actual 1 )                                      
Stdout:                                                                       
Run 1                                                                         
a :: Int -> Int                                                               
168                                                                           
Run 2                                                                         
(A.a,B.b,C.c,D.d)                                                             
  :: (Float -> Float, Float -> Float, Float -> Float, Float -> Float)         
28.0                                                                          
Run 3                                                                         
(A.a,B.b,C.c,D.d)                                                             
  :: (Float -> Float, Float -> Float, Float -> Float, Float -> Float)         
28.0                                                                          
Run 4                                                                         
(A.a,B.b,C.c,D.d)                                                             
  :: (Float -> Float, Float -> Float, Float -> Float, Float -> Float)         
28.0                                                                          
Run 5                                                                         
(A.a,B.b,C.c,D.d)                                                             
  :: (Float -> Float, Float -> Float, Float -> Float, Float -> Float)         
28.0                                                                          
Run 6                                                                         
(A.a,B.b,C.c,D.d)                                                             
  :: (Float -> Float, Float -> Float, Float -> Float, Float -> Float)         
28.0                                                                          
Run 7                                                                         
Segmentation fault/access violation in generated code
Phyx updated this revision to Diff 3735.Aug 2 2015, 2:57 PM
Phyx edited edge metadata.

Removed trailing white spaces

thomie edited edge metadata.Aug 5 2015, 2:56 AM

I didn't look at this patch in great detail yet.

@Phyx: I see you just download perl here, good. What are the other differences between this patch and Gintas' Diff in D339? Especially those linker changes are scary. Do both Diffs just apply @awson's patch without modification? If I were to do a side by side comparison with D339, what would I see?

We need to ask @austin to host those files from sourceforge on the downloads.haskell.org server.

Phyx planned changes to this revision.EditedAug 5 2015, 5:16 AM

I forgot to set it to Plan changes as I'm currently doing the request Awson made in Trac #10726 to use the msys2 builds instead.
This will require a different set of binaries so @austin Should probably wait until that has been done.

I also cleaned up and changed the configure script a bit due to the different way the msys2 builds are packaged.

@awson I have completed the update to GCC 5.2.0 using the msys2 build packages instead but am getting two new test failures which Is why I haven't pushed the diff here.

@thomie Yes this is an application of @awson's patch unmodified (except for a consistent outlining of the / at the end as can be seen in the diff).
If you compare this diff side by side with D339 what you'd see are:

  • driver/gcc/gcc.c In D339 this modification was undeeded and actually the file unused, since the mingw-w64 build do not use /mingw32 as their base directly. The line that used this file was actually removed from the configure script so the file was unneeded. In this diff, the msys2 build like the original mingw build assume the gcc libs to be in /mingw32 and /mingw64 respectively, which would make it point to C:/mingw32 instead of inplace. Hence the trick with the driver is needed once again for this toolchain.
  • compiler/main/SysTools.hs Because of the newer GCC and how it's configured, when compiled with -shared it also compiled in libgcc as shared. Which I think is undesirable, as it's not consistent with what we had before and makes a few test fail on not being able to find libgcc. So instead I now consistently tell it to statically link libgcc in all circumstances.
  • rts/Linker.c This is the same patch as there was before. However because of the continued development I had to manually apply the patch. But I double checked the changes. Most of the changes just deal with the different/fixed name mangings and they just drop RTS_WIN64_ONLY and remove the RTS_WIN32_ONLY entries on some places. The weak reference changes I don't really follow but @awson can check to see if they're still needed.
  • ghc.mk I don't drop perl, so no change was needed here
  • mk/config.mk.in I don't need this, originally the configure script did this by modifying the PATH. Since this line was removed in D339 he needed it here. I leave it in the configure script.
  • configure.ac A slightly different approach, and I no longer (in the diff to be submitted) use 7z, since the files are once again tar files.

To answer a question asked in D339, What this means for the current developers is that they need too rerun configure. If they want to reclaim the disk space from not needing the old tarballs, they can remove ghc-tarballs (and remove the git history etc) and just rerun configure which will recreate the inplace/mingw and perl folder and download the needed packages.

awson added a comment.Aug 5 2015, 8:08 AM

COMDAT support (for which I reused existing weak symbols support) was absolutely necessary for gcc > 4.8. At least gcc 4.9.x didn't work without it. I didn't check gcc 5+ though.

Phyx updated this revision to Diff 3804.Aug 8 2015, 9:13 AM
Phyx edited edge metadata.
  • Upgrade to 5.2.0 and the MSYS2 builds now done for i686
  • Removed tarballs file since no longer seems relevant
  • Added MD5 check on downloads and enabled x86_64 support
  • Fixed MD5sum check
  • Removed a debug line from the configure.ac line
  • Fixed a bug in Linker which would result in undefined behaviour and it crashed.
  • Ignored .rdata$zzz section but load any other section.
  • Removed trailing whitespaces
Phyx added a comment.Aug 8 2015, 9:33 AM

Updated the toolchains to the MSYS2 builds on GCC 5.2.0.
All outstanding issues on x86_64 have been fixed and there's actually one less failing test now.
I have also found two bugs in the linker which have been fixed here as well (one introduced by @awson 's patch and one that was already there), these are pointed out in the comments below.

x86 however has one less unexpected passes but two more unexpected failures. Both in ghci.

The first one is ghci/prog003 which I've mentioned before. the second one is ghci/prog004 which is complaining:

ghc-stage2.exe: panic! (the 'impossible' happened)
  (GHC version 7.11.20150808 for i386-unknown-mingw32):
        loadObj "ctest.o": failed

Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug

ghc-stage2.exe: Misaligned section .eh_frame: 036a030f

But according to objdump it's properly aligned to 4:

9 .eh_frame     00000038  00000000  00000000  0000030f  2**2
                CONTENTS, ALLOC, LOAD, RELOC, READONLY, DATA

So this is most likely another bug in the Linker, which was only revealed by the upgrade. (I suspect the very old mingw compiler used before did not have a .eh_frame section)

As it stands I think the changes can be reviewed and I will try to find the cause of the failures.

If @austin can put the files on http://downloads.haskell.org/~ghc/ I will update the urls.

rts/Linker.c
2391

This was a genuine bug that upgrading the toolchains pointed out. repeated calls to unload would try to free an invalid pointer. Since this is undefined behavior it now crashed the program in gcc 5.2.0.

4390

Most of the .rdata data sections we want to load and consider a SECTIONKIND_CODE_OR_RODATA section. With the exception of .rdata$zzz which is just a section containing the GCC version:

Contents of section .rdata$zzz:
 0000 4743433a 20285265 76332c20 4275696c  GCC: (Rev3, Buil
 0010 74206279 204d5359 53322070 726f6a65  t by MSYS2 proje
 0020 63742920 352e322e 30000000 00000000  ct) 5.2.0.......
Phyx retitled this revision from Upgrade GCC to 5.1.1 for Windows x86 and x86_64 to Upgrade GCC to 5.2.0 for Windows x86 and x86_64.Aug 8 2015, 9:34 AM
Phyx edited edge metadata.

This is looking pretty good. Great job fixing those bugs.

Please see the comments I made. And please update the Diff description, as that is what will end up in the commit message.

The weak reference changes I don't really follow but @awson can check to see if they're still needed.

Adding Simon as a reviewer for this.

compiler/main/SysTools.hs
844

Please turn this into a real (non-inline) Note, explain *why* you want static compilation of libgcc, and link it from here with See Note [Windows static libGCC].

rts/Linker.c
20

Is this still needed?https://ghc.haskell.org/trac/ghc/ticket/9726 makes me think it isn't, so I suggest removing it.

2105–2108

Comment: See Note [mingw-w64 name decoration scheme]

3986

Comment: See Note [mingw-w64 name decoration scheme]

4004

Can you add a Note [mingw-w64 name decoration scheme] to this file, containing the following text from https://ghc.haskell.org/trac/ghc/ticket/9218#comment:12:

What's going on with name decoration? Well, original code have some crufty and ad-hocish paths related mostly to very old mingw gcc/binutils/runtime combinations. Now mingw-w64 offers pretty uniform and MS-compatible decoration scheme across its tools and runtime.

The scheme is pretty straightforward: on 32 bit objects symbols are exported with underscore prepended (and @ + stack size suffix appended for stdcall functions), on 64 bits no underscore is prepended and no suffix is appended because we have no stdcall convention on 64 bits.

Then add a See Note [mingw-w64 name decoration scheme] here.

4390

Please turn this into a code comment or a Note.

Phyx updated this revision to Diff 3824.EditedAug 11 2015, 1:11 PM
Phyx marked 5 inline comments as done.
  • Turned comment in SysTools into proper note
  • Removed _CRT_NON_CONFORMING_SWPRINTFS
  • Added notes to linker
  • Add note about the .rdata section groups
Phyx marked 2 inline comments as done.Aug 11 2015, 1:13 PM

Updated to add notes and other requested changes.

Phyx updated this object.Aug 11 2015, 1:18 PM
Phyx edited edge metadata.
Phyx added a subscriber: Restricted Project.
Phyx updated this revision to Diff 3825.Aug 11 2015, 1:22 PM
  • Added a missing note
Phyx added a comment.Aug 11 2015, 1:26 PM

Macro noice:

That should take care of all the comments for now

thomie accepted this revision.Aug 11 2015, 3:49 PM
thomie edited edge metadata.

Excellent. I think we could go ahead and land this, but let's give others some more time to comment first.

@Phyx, great work on this! I'm going to test and merge this. When doing so I'll update mingw_base_url to point to https://downloads.haskell.org/~ghc/mingw, which now contains the necessary tarballs.

I'm not sure why but I'm seeing this on my Windows test box. @Phyx, does this look familiar to you?

$ make
===--- building phase 0
make --no-print-directory -f ghc.mk phase=0 phase_0_builds
make[1]: Nothing to be done for 'phase_0_builds'.
===--- building phase 1
make --no-print-directory -f ghc.mk phase=1 phase_1_builds
utils/touchy/ghc.mk:18: utils/touchy/dist/build/.depend.c_asm: No such file or directory
utils/unlit/ghc.mk:19: utils/unlit/dist/build/.depend.c_asm: No such file or directory
utils/hp2ps/ghc.mk:25: utils/hp2ps/dist/build/.depend.c_asm: No such file or directory
driver/ghc/ghc.mk:21: driver/ghc/dist/build/.depend.c_asm: No such file or directory
driver/haddock/ghc.mk:21: driver/haddock/dist/build/.depend.c_asm: No such file or directory
utils/genapply/ghc.mk:27: utils/genapply/dist/build/.depend.haskell: No such file or directory
utils/genapply/ghc.mk:27: utils/genapply/dist/build/.depend.c_asm: No such file or directory
libraries/hpc/ghc.mk:3: libraries/hpc/dist-boot/build/.depend-v.haskell: No such file or directory
libraries/hpc/ghc.mk:3: libraries/hpc/dist-boot/build/.depend-v.c_asm: No such file or directory
libraries/template-haskell/ghc.mk:3: libraries/template-haskell/dist-boot/build/.depend-v.haskell: No such file or directory
libraries/template-haskell/ghc.mk:3: libraries/template-haskell/dist-boot/build/.depend-v.c_asm: No such file or directory
libraries/binary/ghc.mk:3: libraries/binary/dist-boot/build/.depend-v.haskell: No such file or directory
libraries/binary/ghc.mk:3: libraries/binary/dist-boot/build/.depend-v.c_asm: No such file or directory
libraries/Cabal/Cabal/ghc.mk:3: libraries/Cabal/Cabal/dist-boot/build/.depend-v.haskell: No such file or directory
libraries/Cabal/Cabal/ghc.mk:3: libraries/Cabal/Cabal/dist-boot/build/.depend-v.c_asm: No such file or directory
libraries/bin-package-db/ghc.mk:3: libraries/bin-package-db/dist-boot/build/.depend-v.haskell: No such file or directory
libraries/bin-package-db/ghc.mk:3: libraries/bin-package-db/dist-boot/build/.depend-v.c_asm: No such file or directory
libraries/hoopl/ghc.mk:3: libraries/hoopl/dist-boot/build/.depend-v.haskell: No such file or directory
libraries/hoopl/ghc.mk:3: libraries/hoopl/dist-boot/build/.depend-v.c_asm: No such file or directory
libraries/transformers/ghc.mk:3: libraries/transformers/dist-boot/build/.depend-v.haskell: No such file or directory
libraries/transformers/ghc.mk:3: libraries/transformers/dist-boot/build/.depend-v.c_asm: No such file or directory
compiler/ghc.mk:654: compiler/stage1/build/.depend-v.haskell: No such file or directory
compiler/ghc.mk:654: compiler/stage1/build/.depend-v.c_asm: No such file or directory
"rm" -f compiler/stage1/build/.depend-v.c_asm.tmp
D:/msys64/home/Administrator/ghc/inplace/mingw/bin/gcc.exe -E  -DMAKING_GHC_BUILD_SYSTEM_DEPENDENCIES  -fno-stack-protector -Wall   -Icompiler/stage1/build/autogen -Icompiler/. -Icompiler/parser -Icompiler/utils -Icompiler/stage1     -isystem'D:\msys64\build-7.8.4\mingw64\ghc-7.10.2\libraries\process\include' -isystem'D:\msys64\build-7.8.4\mingw64\ghc-7.10.2\libraries\directory\include' -isystem'D:\msys64\build-7.8.4\mingw64\ghc-7.10.2\libraries\time\lib/include' -isystem'D:\msys64\build-7.8.4\mingw64\ghc-7.10.2\libraries\containers\include' -isystem'D:\msys64\build-7.8.4\mingw64\ghc-7.10.2\libraries\Win32\include' -isystem'D:\msys64\build-7.8.4\mingw64\ghc-7.10.2\libraries\bytestring\include' -isystem'D:\msys64\build-7.8.4\mingw64\ghc-7.10.2\libraries\base\include' -isystem'D:\msys64\build-7.8.4\mingw64\ghc-7.10.2\libraries\integer-gmp2\include' -isystem'D:/msys64/build-7.8.4/mingw64/ghc-7.10.2/rts/dist/build' -isystem'D:/msys64/build-7.8.4/mingw64/ghc-7.10.2/includes' -isystem'D:/msys64/build-7.8.4/mingw64/ghc-7.10.2/includes/dist-derivedconstants/header'   -Wno-error=inline      -MM -x c compiler/ghci/keepCAFsForGHCi.c -MF compiler/stage1/build/.depend-v.c_asm.bit
compiler/ghci/keepCAFsForGHCi.c:1:17: fatal error: Rts.h: No such file or directory
compilation terminated.
compiler/ghc.mk:654: recipe for target 'compiler/stage1/build/.depend-v.c_asm' failed
make[1]: *** [compiler/stage1/build/.depend-v.c_asm] Error 1
Makefile:103: recipe for target 'all' failed
make: *** [all] Error 2
Phyx added a comment.Aug 12 2015, 9:05 AM

@bgamari hmm no I've never seen that before. Boot and configure both ran successfully?

Does git status show anything odd? Find it weird that it can't find Rts.h. Makes me think the checkout isn't complete or somehow the paths are being messed up.

Fwiw, this patch built successfully for me on windows 64bit using msys2.

This revision was automatically updated to reflect the committed changes.

Alright, well given that @thomie managed to build it I'm going to go ahead and merge.

I'm not sure what happened in my configuration but includes/Rts.h certainly exists. It's curious that -Iincludes appears nowhere in the compiler command line.