Add gen-dll as replacement for dll-split
ClosedPublic

Authored by Phyx on Aug 24 2017, 2:40 AM.

Details

Summary

This tool can be used to generate dll's for any list of object files
given to it. It will then repartition them automatically to fit within
a dll and generates as many dll's as needed to do this. Cyclic dependencies
between these generated dlls are handle automatically so there is no need
to tell it how to partition.

It is also a lot more general than dll-split as it is able to split any
package not just libGHC. It also uses a trick using GNU style import libraries
to hide the splitting from the rest of the pipeline. Which means come linking time
you don't need to know which dll contains what symbol or how many split dlls were
created.

The import libraries are by default created with libtool. However since libtool is BFD
based it is very slow. So if present and detected by configure the genlib tool
from the msys2 project is used. This makes a difference of about ~45 minutes when compiling.

To install genlib run pacman -Sy mingw-w64-$(uname -m)-tools-git.

More detailed explaination of the process can be found here
https://ghc.haskell.org/trac/ghc/wiki/WindowsDynamicLinking

Test Plan

./validate

Diff Detail

Repository
rGHC Glasgow Haskell Compiler
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
Phyx created this revision.Aug 24 2017, 2:40 AM
Phyx updated this revision to Diff 13613.Aug 24 2017, 2:44 AM

Remove 8.2.1 release not changes.

bgamari edited edge metadata.Aug 24 2017, 8:18 AM
bgamari added a subscriber: snowleopard.

Adding @snowleopard as this will need to be reflected in Hadrian as well.

This looks reasonable to me. Just some documentation requests inline.

To install genlib run pacman -Sy mingw-w64-$(uname -m)-tools-git.

Thanks for adding this to the Wiki already! It might also be a good idea to send a message to ghc-devs when this is merged to ensure that everyone realizes the change.

configure.ac
394

Don't we also need to do this in distrib/configure.ac.in?

utils/gen-dll/Main.hs
5

It would be great if there were a Note here explaining the goal of this utility and roughly outlining how it works. Be sure to reference Trac #5987 and briefly explain how this interacts (eventually?) with SxS. Also perhaps reference Note [BFD import library].

52

What is symbol padding?

82

What is this Char?

284

Should we perhaps catch the case where the string is neither "yes" nor "no"? Seems like we should fail in this case.

Phyx updated this revision to Diff 13624.Aug 24 2017, 4:40 PM
Phyx marked 4 inline comments as done.

Add documentation and process feedback.

Phyx added inline comments.Aug 24 2017, 4:44 PM
configure.ac
394

No, gen-dll won't be distributed, it's only needed to build stage2. In fact the paths ./configure finds for these tools here are baked into the the gen-dll.exe so it can't be moved to another machine.

utils/gen-dll/Main.hs
82

It's the type of the symbol, e.g. the Single letter descriptions nm uses to indicate what the symbol is. It's important to split out data and function, as the way we access them is different.

Phyx updated this revision to Diff 13625.Aug 24 2017, 5:01 PM

clarify SxS

Phyx updated this revision to Diff 13626.Aug 24 2017, 5:02 PM

Clarify SxS

Phyx updated this revision to Diff 13632.Aug 25 2017, 4:18 PM

relax process version bounds.

Phyx updated this revision to Diff 13638.Aug 25 2017, 9:21 PM
  • Update process bounds instead of filepath
bgamari accepted this revision.Aug 29 2017, 1:39 PM

Alright, this looks good to me.

configure.ac
394

So if I'm not mistaken this means that user libraries with many symbols will still fail?

This revision is now accepted and ready to land.Aug 29 2017, 1:39 PM
Phyx added inline comments.Aug 29 2017, 1:47 PM
configure.ac
394

Yes, but that wouldn't be a problem unique to GHC in that case. Theoretically gen-dll could be generalized to supposed user libraries, but atm it's very specialized to the build system requirements of GHC.

You also need a fairly large codebase in order for you to hit this issue. At that point you can probably get around it by just splitting your package.

This revision was automatically updated to reflect the committed changes.
trofi added a subscriber: trofi.Sep 20 2017, 4:49 PM
trofi added inline comments.
aclocal.m4
1149

POSIX shell syntax requires no spaces in assignments.
Does this code have correct if condition? It looks inverted from here.

The only reason I noticed is warning these lines on linux:

$ ./configure --target=sparc-unknown-linux-gnu
...
./configure: line 9708: cygpath: command not found
./configure: line 9708: ArCmd: command not found
trofi added inline comments.Sep 20 2017, 5:21 PM
aclocal.m4
1149

Pushed tweak as: https://git.haskell.org/ghc.git/commitdiff/d7705f2fbead058700940321982a6315450dbcd1

I hope I didn't break windows as a result.

Phyx added a comment.Sep 25 2017, 12:54 PM

POSIX shell syntax requires no spaces in assignments.

I did not know this, I'll keep this in mind going forward.

Thanks @trofi