Add a RTS option -xp to load PIC object anywhere in address space
ClosedPublic

Authored by watashi on Oct 2 2018, 11:12 AM.

Details

Summary

Add a RTS option -xp to load PIC object anywhere in address space. We do
this by relaxing the requirement of <0x80000000 result of mmapForLinker
and implying USE_CONTIGUOUS_MMAP.

We also need to change calls to ocInit and ocGetNames to avoid dangling
pointers when the address of oc->image is changed by ocAllocateSymbolExtra.

Test Plan
$ uname -a
Linux localhost 4.18.8-arch1-1-ARCH #1 SMP PREEMPT Sat Sep 15 20:34:48 UTC 2018 x86_64 GNU/Linux
$ cat mk/build.mk
DYNAMIC_GHC_PROGRAMS = NO
DYNAMIC_BY_DEFAULT = NO

GhcRTSWays += thr_debug
EXTRA_HC_OPTS += -debug
WAY_p_HC_OPTS += -fPIC -fexternal-dynamic-refs
$ inplace/bin/ghc-stage2 --interactive -prof +RTS -xp
GHCi, version 8.7.20180928: http://www.haskell.org/ghc/  :? for help
ghc-stage2: R_X86_64_32 relocation out of range:
ghczmprim_GHCziTypes_ZMZN_closure = 7f690bffab59
Recompile /data/users/watashi/ghc/libraries/ghc-prim/dist-install/build/HSghc-prim-0.5.3.o with -fPIC -fexternal-dynamic-refs.
ghc-stage2: unable to load package `ghc-prim-0.5.3'
$ strace -f -e open,mmap inplace/bin/ghc-stage2 --interactive -prof -fexternal-interpreter  -opti+RTS -opti-xp
...
[pid 1355283] open("/data/users/watashi/ghc/libraries/base/dist-install/build/libHSbase-4.12.0.0_p.a", O_RDONLY) = 14
[pid 1355283] mmap(NULL, 8192, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6a84842000
[pid 1355283] open("/data/users/watashi/ghc/libraries/base/dist-install/build/libHSbase-4.12.0.0_p.a", O_RDONLY) = 14
[pid 1355283] mmap(NULL, 8192, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6a84676000
...
Prelude> System.Posix.Process.getProcessID
...
[pid 1355283] open("/data/users/watashi/ghc/libraries/unix/dist-install/build/libHSunix-2.7.2.2_p.a", O_RDONLY) = 14
[pid 1355283] mmap(NULL, 45056, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6a67d60000
[pid 1355283] open("/data/users/watashi/ghc/libraries/unix/dist-install/build/libHSunix-2.7.2.2_p.a", O_RDONLY) = 14
[pid 1355283] mmap(NULL, 57344, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6a67d52000
...
$ uname -a
Darwin watashis-iMac.local 18.0.0 Darwin Kernel Version 18.0.0: Wed Aug 22 20:13:40 PDT 2018; root:xnu-4903.201.2~1/RELEASE_X86_64 x86_64
$ mv /Users/watashi/gao/ghc/libraries/integer-gmp/dist-install/build/HSinteger-gmp-1.0.2.0.o{,._DISABLE_GHC_ISSUE_15105}
$ inplace/bin/ghc-stage2 --interactive  +RTS -xp
GHCi, version 8.7.20181003: http://www.haskell.org/ghc/  :? for help
Prelude> System.Posix.Process.getProcessID
42791
Prelude> Data.Set.fromList [1 .. 10]
fromList [1,2,3,4,5,6,7,8,9,10]
Prelude> 
Leaving GHCi.
$ inplace/bin/ghc-stage2 --interactive -prof -fexternal-interpreter
GHCi, version 8.7.20181003: http://www.haskell.org/ghc/  :? for help
Prelude> System.Posix.Process.getProcessID
42806
Prelude> Data.Set.fromList [1 .. 10]
fromList [1,2,3,4,5,6,7,8,9,10]
Prelude> 
Leaving GHCi.

Also test with something that used to hit the 2Gb limit and it loads and runs without problem.

Diff Detail

Repository
rGHC Glasgow Haskell Compiler
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
watashi created this revision.Oct 2 2018, 11:12 AM
Phyx added a comment.Oct 2 2018, 12:11 PM

I don't know which dangling pointer you're talking about.... If it's something your change introduced then you must take care of it there. flipping the order like this will just segfault on stage2 on Windows.
ocAllocateSymbolExtras_PEi386 accesses oc->info->image https://github.com/ghc/ghc/blob/5840734379da5d494a368d0b8a6edf1b1216a7f4/rts/linker/PEi386.c#L1785 which is set in ocGetNames_PEi386 https://github.com/ghc/ghc/blob/5840734379da5d494a368d0b8a6edf1b1216a7f4/rts/linker/PEi386.c#L1579 if the linker decides it's going to keep the object file. So I don't see how flipping the order can work at all.

@Phyx Thanks for pointing this out, so we cannot flip the order for PEi386 and I will keep it untouched, but it was a problem for ELF when !RTS_LINKER_USE_MMAP || USE_CONTIGUOUS_MMAP is true.

We may create a new image with larger size in

https://phabricator.haskell.org/diffusion/GHC/browse/master/rts/linker/SymbolExtras.c;21efbc7599e39ec93b8b13b7d7b84811226e6f6f$62

or

https://phabricator.haskell.org/diffusion/GHC/browse/master/rts/linker/SymbolExtras.c;21efbc7599e39ec93b8b13b7d7b84811226e6f6f$88

and free the old image, but the pointer set in

https://phabricator.haskell.org/diffusion/GHC/browse/master/rts/linker/Elf.c;21efbc7599e39ec93b8b13b7d7b84811226e6f6f$930

will still point to somewhere in the old image and cause segfault.

watashi updated this revision to Diff 18181.Oct 2 2018, 1:28 PM

Avoid changing the order for Windows

Phyx added a comment.Oct 2 2018, 1:54 PM

@watashi Thanks, that makes more sense. The behaviors are different so for now splitting them up is fine, Add a /* See Note [loadOc orderings]. */ in front of the PE one.
I think I'll just move the call into ocGetNames_PEi386 so that Linker.c doesn't diverge. But for now this is fine. Thanks!

simonmar added inline comments.Oct 3 2018, 2:45 AM
docs/users_guide/runtime_control.rst
247–251

Suggest some rewording and expansion here:

On 64-bit machines, the runtime linker usually needs to map object code into the low
2Gb of the address space, due to the x86_64 small memory model where most symbol references are 32 bits. The problem is that this 2Gb of address space can fill up, especially if you're loading a very large number of object files into GHCi.

This flag offers a workaround, albeit a slightly convoluted one. To be able to load an object file outside of the low 2Gb, it needs to be compiled with -fPIC -fexternal-dynamic-refs. When the +RTS -xp flag is passed, the linker will assume that all object files were compiled with -fPIC -fexternal-dynamic-refs and load them anywhere in the address space. It's up to you to arrange that the object files you load (including all packages) were compiled in the right way. If this is not the case for an object, the linker will probably fail with an error message when the problem is detected.

rts/Linker.c
1033–1036

Perhaps get rid of ALWAYS_PIC and set linkerAlwaysPic to true by default on OS X instead? That would eliminate some CPP, which is always a good thing.

rts/RtsFlags.c
264

You need to add it to the usage text here too.

watashi updated this revision to Diff 18199.Oct 3 2018, 5:01 PM
watashi marked 3 inline comments as done.

clean up ALWAYS_PIC macro and comments

watashi edited the test plan for this revision. (Show Details)Oct 3 2018, 7:30 PM
watashi edited the test plan for this revision. (Show Details)
watashi edited the test plan for this revision. (Show Details)Oct 3 2018, 7:32 PM
watashi updated this revision to Diff 18201.Oct 3 2018, 7:52 PM

comments

simonmar accepted this revision.Oct 4 2018, 1:58 AM

Looks good to go, provided it passes validate.

This revision is now accepted and ready to land.Oct 4 2018, 1:58 AM
This revision was automatically updated to reflect the committed changes.