It doesn't validated on Windows, most likely because of Trac #10672 (D1244).

ghc: internal error: checkProddableBlock: invalid fixup in runtime linker: 0000000006aa7320
    (GHC version 7.11.20150919 for x86_64_unknown_mingw32)
    Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.
bindisttest/ghc.mk:21: recipe for target 'test_bindist' failed

Ok, good news. Using both D1244 and this patch, the build does succeed on Windows 64 bit. The testsuite only reports a few failing tests, some of which are known to fail:

Unexpected passes:
   ghci/scripts  T9878b (ghci)

Unexpected failures:
   rts         T7037 [bad stdout] (normal)
   rts         outofmem [bad stdout] (normal)
   rts/T10672  T10672_x64 [bad exit code] (normal)

Unexpected stat failures:
   perf/haddock  haddock.base [stat too good] (normal)

I suggest we first review and land D1244, and then come back to this patch.

Hmmm

rts/T10672  T10672_x64 [bad exit code] (normal)

That shouldn't be failing though.. I'll take a look at that..

perf.haskell.org results:

nofib/time/k-nucleotide 	4.382	- 3.65%	4.222	seconds

nofib/size/compress 	953121	- 10.88%	849389	bytes
nofib/size/grep 	912623	- 4.92%	867733	bytes
nofib/size/knights 	922645	- 3.33%	891955	bytes
nofib/size/veritas 	1803176	- 5.25%	1708470	bytes

@kgardas: do you have the Solaris/x86/x64/SPARC results?

@thomie solaris results will be done this week (up to Wednesday). Last week I needed to be out of town. W.r.t binutils < 2.17 I just signal that Solaris is using 2.23 and OpenBSD/amd64 is using 2.17 -- so from those two platforms it's safe to remove mentioned hack.

There might be some interaction between this patch and D975, but I haven't tested it. My worry is that the linker will end up making a million tiny mmap calls.

In D1242#37060, @simonmar wrote:

There might be some interaction between this patch and D975, but I haven't tested it. My worry is that the linker will end up making a million tiny mmap calls.

I take it you would need to test the combination of the patches with your specific project to be able to tell if it works?

The total number of sections with split-sections (and the size of each section) should be comparable to what you get with split-objs though, so if split-objs works with your patch (and project) I would expect split-sections to work too. If I understand D975 correctly, small sections should get allocated with the m32 allocator and be able to share pages.

Well, one difference is that with -split-objs, I can make a single object using ld -r and then use that with the RTS linker. With this patch I can still do that, but the object file will have a million tiny sections each of which will get mmap'd separately by the RTS linker. Is there any way to combine the sections into a single text section using ld?

In D1242#37132, @simonmar wrote:

Well, one difference is that with -split-objs, I can make a single object using ld -r and then use that with the RTS linker. With this patch I can still do that, but the object file will have a million tiny sections each of which will get mmap'd separately by the RTS linker. Is there any way to combine the sections into a single text section using ld?

ld -r does indeed produce an object with all the sections. I can't see a flag for telling ld to merge sections, but it can be done with a linker script (e.g. https://github.com/olsner/ghc/commit/D1242_merge_sections). This is fine when using GNU ld, and Mac OS doesn't need the script anyway (it doesn't use sections), but if e.g. Solaris or BSD doesn't use GNU ld those platforms would still get lots of sections to load (as I believe they don't support linker scripts?).

I also just noticed that without the merging, disabling DYNAMIC_GHC_PROGRAMS and enabling sections gives a broken ghci. The RTS linker only supports 64k ELF sections, and HSbase.o gets more sections than that...

I had a patch lined up for fixing this in the linker, but didn't include it here since everything seemingly worked fine without it. I thought it had just become unnecessary due to changes to the splitting code, but apparently not :)
I'll aim to clean that up for review this evening. Should I submit the linker fix as another diff that we can merge separately, or just add it to this diff?

Sorry for delay, I've done promised Solaris 11 testing on i386 and amd64 on HEAD + arc patch D1242 and have not found any new regression. To my surprise arc patch D1242 did from previously checked out HEAD something more older so I've also needed to manually patch issue in testlib.py to make it compatible with python 2.6 (original patch by Thomas is already in HEAD).
So far I've not been able to test SPARC NCG, as this does have its own set of troubles.

In D1242#37263, @olsner wrote:

ld -r does indeed produce an object with all the sections. I can't see a flag for telling ld to merge sections, but it can be done with a linker script (e.g. https://github.com/olsner/ghc/commit/D1242_merge_sections). This is fine when using GNU ld, and Mac OS doesn't need the script anyway (it doesn't use sections), but if e.g. Solaris or BSD doesn't use GNU ld those platforms would still get lots of sections to load (as I believe they don't support linker scripts?).

Yes, Solaris 11 is not using GNU ld and yes it does not support GNU ld linker scripts. On the other hand my OpenBSD 5.8 is using GNU ld coming from binutils 2.17 so I guess it also supports linker scripts at least on a level of 2.17 release.

Any progress on this @olsner? It would be great to have this for 8.0.

I can't see a flag for telling ld to merge sections, but it can be done with a linker script (e.g. https://github.com/olsner/ghc/commit/D1242_merge_sections).

driver/utils/merge_sections.ld seems like a fine location to me.

Should I submit the linker fix as another diff that we can merge separately

Yes please.

So, with the updated patch the merging for ghci libs is done on GNU ld platforms, and the RTS linker fix is submitted as D1357. Other ELF platforms will not properly support -split-sections in ghci until that's in.
I also did some testing without the merging (and with the linker fix) to see if there was any bad interaction with D975, but didn't notice anything.

Windows should be saved by ld -r too since it uses GNU ld, but otherwise I think many-sections support would require linking libraries with the -mbig-obj flag (which switches to a different format of some headers that has room for more than 32k sections), as well as adding RTS linker support for that new "big object" format. (Although it sounds scary, it looks like the big object format reuses almost all of the rest of COFF, so it shouldn't be too difficult if we end up needing it.)

Several nits and a few questions, but overall this looks good to me.

Nice work. Some small comments. Only address them if you still feel like, because this patch has been delayed quite a bit already.

function-sections has a bad interaction with D1375: the link-info section added by GHC is garbage collected by --gc-sections when the PROGBITS section type is used (confirmed by @olsner with --print-gc-sections).

A fix would be to revert D1375 (i.e. use NOTE section type to avoid garbage section collection) but to use the correct NOTE format this time. Then to read it back, we could either parse "readelf -n" output or directly read the binary. I am much in favor of the latter approach and I could easily add a few lines to D1381 to do it. Before I do it, we must arbitrate between the readelf approach (cf D1326) and D1381.

D1381 is now in so this should be ready to test again.

In D1242#42682, @bgamari wrote:

In D1242#42663, @olsner wrote:

Also found a problem when combining -split-sections with -g. Fixed by not
splitting debug info when splitting sections.

Can you describe this problem a bit more?

The labels emitted at the start of each debug section (dwarfLineLabel, etc) get duplicated once for each split - with -split-objs those end up in different assembly files compiled to different object files (and they'll be needed once in each file), but in this case where all the sections will be assembled in the same file you get conflicting symbol errors instead.

Doing something like section splitting on the debug info itself is not very useful (and after some reading on that it seems it would not have worked anyway). When linking debug info with --gc-sections ld rather keeps all the debug info and replaces references to GC'd functions with placeholder values. (and tools like gdb are aware of that workaround)