Fix GHCi on Arm (#10375).
ClosedPublic

Authored by erikd on Oct 11 2015, 5:10 PM.

Details

Summary

Arm has two instruction sets, Arm and Thumb, and an execution mode for each.
Executing Arm code in Thumb mode or vice-versa will likely result in an
Illegal instruction exception.

Furthermore, Haskell code compiled via LLVM was generating Arm instructions
while C code compiled via GCC was generating Thumb code by default. When
these two object code types were being linked by the system linker, all was
fine, because the system linker knows how to jump and call from one
instruction set to the other.

The first problem was with GHCi's object code loader which did not know
about Thumb vs Arm. When loading an object file StgCRun would jump
into the loaded object which could change the mode causing a crash after
it returned. This was fixed by forcing all C code to generate Arm
instructions by passing -marm to GCC.

The second problem was the mkJumpToAddr function which was generating
Thumb instructions. Changing that to generate Arm instructions instead
results in a working GHCi on Arm.

Test Plan

validate on x86_64 and arm

Diff Detail

Repository
rGHC Glasgow Haskell Compiler
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
erikd updated this revision to Diff 4474.Oct 11 2015, 5:10 PM
erikd retitled this revision from to Fix GHCi on Arm (#10375)..
erikd updated this object.
erikd edited the test plan for this revision. (Show Details)
erikd added a reviewer: bgamari.
erikd updated the Trac tickets for this revision.
bgamari edited edge metadata.Oct 11 2015, 6:40 PM

Yay! Very well done; these issues are devilishly difficult to sort out.

That being said, it would be nice if we could catch the case of Thumb code, perhaps just by barfing. IIRC this is marked by a set low-bit in the relocation address. In principle we should now never see Thumb code with your patch, but we should make sure that this deficiency is very clearly signposted.

thomie added a subscriber: kgardas.Oct 12 2015, 5:15 AM

A Note [Arm vs Thumb] would be helpful, including the summary above and why we don't generate Thumb.

git grep Thumb returns quite a few results. Is that code still needed (or also wrong?). cc @kgardas as the author of f3f8c90638e38088c3c8ea251f5c8e54e4047fe7.

Good work, especially if this really makes GHCi usable on ARM. Just a nitpick, what makes you to decide on usage of ARM instead of Thumb Eric? I'm asking since for example Ubuntu in the past went different route arm -> thumb so I'm just curious. It would also be good to see ARM versus Thumb nofib results, but well, this is really out of the scope of this patch anyway. Just for ARM fan curious about the results... :-)
Anyway, thanks for this patch!

erikd added a comment.Oct 12 2015, 2:25 PM

Good work, especially if this really makes GHCi usable on ARM. Just a nitpick, what makes you to decide on usage of ARM instead of Thumb Eric? I'm asking since for example Ubuntu in the past went different route arm -> thumb so I'm just curious. It would also be good to see ARM versus Thumb nofib results, but well, this is really out of the scope of this patch anyway. Just for ARM fan curious about the results... :-)
Anyway, thanks for this patch!

I chose Arm instead of Thumb for three reasons:

  • The LLVM backend was generating Arm by default (as does Clang).
  • My understanding is that when allowed to generate Thumb code, GCC produces a mixture of Thumb and Arm code, because not everything can be encoded purely as Thumb,
  • If GHC wanted to allow use of mixed Thumb and Arm code, we need to put a lot more work into the run time linker to make sure it does the right thing with interop between the two.
erikd added a comment.Oct 12 2015, 6:17 PM
  • My understanding is that when allowed to generate Thumb code, GCC produces a mixture of Thumb and Arm code, because not everything can be encoded purely as Thumb,

Turns out that while that may be true of the original Thumb instruction set it is not true of Thumb2 which in fact what we would be using.

It may be possible to get LLVM to generate Thumb2 and do the same for GCC and then make sure the linker and ghci still work, but I don't think I have time to explore those avenues. What we have in this patch works now.

erikd added a comment.Oct 12 2015, 7:10 PM
In D1323#37606, @thomie wrote:

git grep Thumb returns quite a few results. Is that code still needed (or also wrong?).

Its becoming clearer that I need to do some more work on this.

Firstly I need to see if we can support Thumb2 only inplace of Arm. If that works, its probably a better option because i has performance benefits (most instructions encoded in two bytes instead of 4).

Secondly, since we have no control over where the object code modules loaded by GHCi come from, its probably worthwhile seeing if we can get proper interop working between Thumb and Arm.

However, I wonder if maybe we shouldn't get this commited and applied to the 7.10 branch so that 7.10.3 can be released with a working ghci before tackling the issues above.

I'm going to update the current commit so that the linker barfs on Thumb relocations as suggested by @bgamari while I investigate the other options.

bgamari accepted this revision.Oct 12 2015, 8:13 PM
bgamari edited edge metadata.
In D1323#37633, @erikd wrote:
In D1323#37606, @thomie wrote:

git grep Thumb returns quite a few results. Is that code still needed (or also wrong?).

Its becoming clearer that I need to do some more work on this.

Firstly I need to see if we can support Thumb2 only inplace of Arm. If that works, its probably a better option because i has performance benefits (most instructions encoded in two bytes instead of 4).

Secondly, since we have no control over where the object code modules loaded by GHCi come from, its probably worthwhile seeing if we can get proper interop working between Thumb and Arm.

However, I wonder if maybe we shouldn't get this commited and applied to the 7.10 branch so that 7.10.3 can be released with a working ghci before tackling the issues above.

I'm going to update the current commit so that the linker barfs on Thumb relocations as suggested by @bgamari while I investigate the other options.

Great, I'm fine with committing this after it's updated to fail on Thumb. That being said, I think @erikd is right that full Thumb support would be useful. That being said, I don't think we'll ever be able to support linking of Thumb Haskell code with ARM Haskell code. The reason for this is that interop requires the use of a trampoline, which breaks tables-next-to-code.

This revision is now accepted and ready to land.Oct 12 2015, 8:13 PM

However, I wonder if maybe we shouldn't get this commited and applied to the 7.10 branch so that 7.10.3 can be released with a working ghci before tackling the issues above.

Sounds good to me.

@erikd Erik, you really did great job on this and thanks for also considering Thumb2 support. In fact while talking about thumb in my original remark I've been talking about thumb2 of course. Thumb is already a history...

This revision was automatically updated to reflect the committed changes.
erikd added a comment.Oct 14 2015, 7:11 PM

Lodged ticket https://ghc.haskell.org/trac/ghc/ticket/10969 to track any futher work on Thumb2/Arm interop.