Build utilities with the bootstrap compiler when cross compiling
AbandonedPublic

Authored by angerman on Sep 29 2017, 1:49 AM.

Details

Reviewers
bgamari
trofi
hvr
austin
Trac Issues
#14297
Summary

This should fix Trac Trac #14297. When building a cross compiler, we have rather little use
of utilities that do not run on the host, where the compiler runs. As such we should
build the utilities with the bootstrap (stage 0) compiler rather than witht he
in-tree (stage 1) compiler when CrossCompiling.

This used to results in the utilities we ship in the binary distribution to be built for
the wrong host. This diff tries to rectify the situation and allow the binary distribution
to contain the utilities for the host when CrossCompiling.

angerman created this revision.Sep 29 2017, 1:49 AM

The corresponding patch to hsc2hs has already been apply to the repository.

angerman updated this revision to Diff 14193.Sep 29 2017, 2:01 AM
  • bump hsc2hs
trofi edited edge metadata.Sep 29 2017, 3:50 AM

The commit message does not explain what exactly changes in:

  • build process
  • what changes in installed artifacts

GHC supports a few flavours of crosscompiler (see Note [CrossCompiling vs Stage1Only], http://git.haskell.org/ghc.git/blob/HEAD:/mk/config.mk.in#l586)
that sould be considered here:

  1. installing cross-compiler itself: make install CrossCompiling=YES Stage1Only=YES
  2. installing crossbuilt native target make install CrossCompiling=YES Stage1Only=NO

Normally you can install both flavours after building stage2.

In look like your change building some tools wth stage0 twice.

Let's consider unlit (i believe it already works after ff84d052850b637b03bbb98cf05202e44886257d).
It clearly already builds stage 0 with $(eval $(call build-prog,utils/unlit,dist,0)) and installs it in Stage1Only1 as:

ifeq "$(Stage1Only)" "YES"
utils/unlit_dist_INSTALL         = YES
utils/unlit_dist-install_INSTALL = NO
else
utils/unlit_dist_INSTALL         = NO
utils/unlit_dist-install_INSTALL = YES
endif

Why do you need another $(eval $(call build-prog,utils/unlit,dist-install,0))?

Thanks for the review @trofi!

When building a cross compiler for distribution with make binary-dist given a quick-cross flavour.
The final .tar.xz will (exemplary on ghc-8.3.20170929-aarch64-apple-ios.tar.xz) contain the following:

$ find . -type f -perm -a=x -print |xargs file
./config.guess:                                             POSIX shell script text executable, ASCII text
./config.sub:                                               POSIX shell script text executable, ASCII text
./configure:                                                POSIX shell script text executable, ASCII text
./ghc/stage1/build/tmp/ghc-stage1:                          Mach-O 64-bit executable x86_64
./inplace/bin/mkdirhier:                                    POSIX shell script text executable, ASCII text
./inplace/lib/bin/ghc-split:                                a /usr/bin/perl script text executable, ASCII text
./install-sh:                                               POSIX shell script text executable, ASCII text
./iserv/stage2/build/tmp/ghc-iserv:                         Mach-O 64-bit executable arm64
./utils/ghc-cabal/dist-install/build/tmp/ghc-cabal:         Mach-O 64-bit executable arm64
./utils/ghc-cabal/dist-install/build/tmp/ghc-cabal-bindist: POSIX shell script text executable, ASCII text
./utils/ghc-pkg/dist/build/tmp/ghc-pkg:                     Mach-O 64-bit executable x86_64
./utils/hp2ps/dist/build/tmp/hp2ps:                         Mach-O 64-bit executable x86_64
./utils/hpc/dist-install/build/tmp/hpc:                     Mach-O 64-bit executable arm64
./utils/hsc2hs/dist-install/build/tmp/hsc2hs:               Mach-O 64-bit executable arm64
./utils/unlit/dist/build/tmp/unlit:                         Mach-O 64-bit executable x86_64

And indeed we seem to be putting only the wrong ghc-cabal, hpc and hsc2hs
into the distribution file. (ghc-iserv is correct, as that one would need to run on the
target anyway).

This results in the installed version (xz -d; tar -xf; make install) to fail first
because ghc-cabal can't be executed on the host. And if one copies ghc-cabal
from a full stage2 built in, later hsc2hs fails when called by cabal. Similarly hpc
is of no use after being installed either, because it won't be able to run on the
host.

I see, that I might have been a bit to quick to change every dist-install for each tool.

Are you proposing to put the dist instead of dist-install version into the
bindist as you did in ff84d052850b637b03bbb98cf05202e44886257d?

I've very likely overestimated the meaning of the "dist-install" name.

angerman updated this revision to Diff 14194.Sep 29 2017, 4:35 AM
  • drop unlit and hp2ps
angerman updated this revision to Diff 14195.Sep 29 2017, 4:44 AM
  • cleanup
trofi added a comment.Sep 29 2017, 5:09 AM

dist-install is just a name of intermediate directory nearby source of a tool or library.
The directory name itself does not carry semantic meaning and is just a suffix for rest
of variables like utils/hpc_dist-install_INSTALL = YES (vs, say utils/hpc_dist_INSTALL = YES)

The real behaviour is defined by by actual variables like utils/hpc_dist-install_SHELL_WRAPPER (or _INSTALL, etc.).
Normally I would suggest never mixing the same intermediate directory for two
distinct install flavours and instead of having

ifeq "$(Stage1Only) $(CrossCompiling)" "YES YES"
# compile with stage 0 (bootstrap compiler)
$(eval $(call build-prog,utils/hpc,dist-install,0))
else
$(eval $(call build-prog,utils/hpc,dist-install,1))
endif

define both installed targets unconditionally
without any if guards as:

$(eval $(call build-prog,utils/hpc,dist,0))
$(eval $(call build-prog,utils/hpc,dist-install,1))

and tweak variables to change installation behaviour.

[ Vaguely related: I think real decision to build or not to build a package is defined by
PACKAGES_STAGE<N> list in /ghc.mk thus it's completely fine to have targets
not used in some contexts ]

That way subsequent runs of make and make Stage1Only=NO should result in slightly
less inconsistent state in intermediate directories
[ inplace/bin/... contents might still be bit of a mess due to wrapper override, we will fix it one day ]

I hope that makes any sense.

angerman updated this revision to Diff 14254.Oct 3 2017, 8:58 PM
  • update

The hsc2hs repo would need to have the following patch

From 94af7d9a27307f40a8b18da0f8e0fd9e9d77e818 Mon Sep 17 00:00:00 2001
From: Moritz Angermann <moritz.angermann@gmail.com>
Date: Sun, 1 Oct 2017 18:07:17 +0800
Subject: [PATCH] Use the same conditional install logic from unlit

This removes the make concurrency bug from the CrossCompilation fix, and uses the
same appraoch `utils/unlit` already uses.
---
 ghc.mk | 21 ++++++++-------------
 1 file changed, 8 insertions(+), 13 deletions(-)

diff --git a/ghc.mk b/ghc.mk
index c5f7473..8221829 100644
--- a/ghc.mk
+++ b/ghc.mk
@@ -5,27 +5,22 @@ utils/hsc2hs_dist_PROGNAME         = hsc2hs
 utils/hsc2hs_dist-install_PROGNAME = hsc2hs
 
 utils/hsc2hs_dist_SHELL_WRAPPER = YES
-utils/hsc2hs_dist_INSTALL = NO
 utils/hsc2hs_dist_INSTALL_INPLACE = YES
 
 utils/hsc2hs_dist-install_SHELL_WRAPPER = YES
-utils/hsc2hs_dist-install_INSTALL = YES
 utils/hsc2hs_dist-install_INSTALL_INPLACE = NO
 
-$(eval $(call build-prog,utils/hsc2hs,dist,0))
-
-# When CrossCompiling, we want to ship the binary for the
-# host, not for the target.  As such we need to compile
-# with the Bootstrap compiler rather than with the in-tree
-# stage1 compiler, which would result in a binary that
-# won't run on the host.
-ifeq "$(CrossCompiling)" "YES"
-# compile with stage 0 (bootstrap compiler)
-$(eval $(call build-prog,utils/hsc2hs,dist-install,0))
+ifeq "$(Stage1Only)" "YES"
+utils/hsc2hs_dist_INSTALL         = YES
+utils/hsc2hs_dist-install_INSTALL = NO
 else
-$(eval $(call build-prog,utils/hsc2hs,dist-install,1))
+utils/hsc2hs_dist_INSTALL         = NO
+utils/hsc2hs_dist-install_INSTALL = YES
 endif
 
+$(eval $(call build-prog,utils/hsc2hs,dist,0))
+$(eval $(call build-prog,utils/hsc2hs,dist-install,1))
+
 # After build-prog above
 utils/hsc2hs_dist_MODULES += Paths_hsc2hs
 utils/hsc2hs_dist-install_MODULES = $(utils/hsc2hs_dist_MODULES)
-- 
2.14.1

applied, to be consistent with the changes in updated changes in here.

bgamari added a comment.EditedOct 3 2017, 11:30 PM

I'll admit I'm not terribly happy about including things build with the stage0 compiler in a bindist. One of the points of our staged compilation is to isolate ourselves from the stage0 compiler. By shipping artifacts built by stage0 we break this isolation in the most direct way possible.

In general our approach of using the stage1 compiler for cross compilation seems like a hack. I think ultimately (read "post Hadrian") what we should do is build a stage1 targeting the host, then use that to build a stage2 targeting the cross compilation target. We could then build a stage3 native target compiler if we wanted.

However, if we can agree that this is a teMcCrary solution then I'm okay with merging it.

I'll admit I'm not terribly happy about including things build with the stage0 compiler in a bindist. One of the points of our staged compilation is to isolate ourselves from the stage0 compiler. By shipping artifacts built by stage0 we break this isolation in the most direct way possible.

In general our approach of using the stage1 compiler for cross compilation seems like a hack. I think ultimately (read "post Hadrian") what we should do is build a stage1 targeting the host, then use that to build a stage2 targeting the cross compilation target. We could then build a stage3 native target compiler if we wanted.

However, if we can agree that this is a teMcCrary solution then I'm okay with merging it.

Yes, this is not an ideal solution. There is some discussion of why we do the stage1 cross compiler logic on the wiki at https://ghc.haskell.org/trac/ghc/wiki/CrossCompilation
In general, I end up building a full stage2 compiler, and use that to build the cross compilers, which is somewhat similar to building a stage2 cross compiler.

Again, I'm not terribly happy with this solution. Yet the current status quo, packaging a binary for the wrong architecture seems even worse to me, as it leads to an un-installable binary distribution.

bgamari accepted this revision.Oct 25 2017, 1:28 PM
bgamari added a subscriber: snowleopard.

Alright, fair enough. CCing @snowleopard since this will require work in Hadrian as well.

This revision is now accepted and ready to land.Oct 25 2017, 1:28 PM

Alright, fair enough. CCing @snowleopard since this will require work in Hadrian as well.

I would hope this does not affect Hadrian just yet. As this is all only for the bin-dist logic that's still missing, iirc.

On the other hand, I've started heavily mutilating Hadrian in a local branch, to make it produce staged output
in the first place. (e.g. no more inplace stuff. You could just use _build/stage<N> as a relocatable install)
that would elevate the issue for hadrian completely.

e.g. no more inplace stuff. You could just use _build/stage<N> as a relocatable install)
that would elevate the issue for hadrian completely.

Oh, that would be nice. This inplace stuff is one of the most confusing places in Hadrian. Can we get rid of it altogether?

e.g. no more inplace stuff. You could just use _build/stage<N> as a relocatable install)
that would elevate the issue for hadrian completely.

Oh, that would be nice. This inplace stuff is one of the most confusing places in Hadrian. Can we get rid of it altogether?

That's the idea. However it involves mutilating hadrian quite a bit right now.
E.g. programs built in stageN are put into stage(N+1)/bin and similarly for libraries.

This will render using the make and hadrian build systems concurrently useless though. As the make build system will remain
to expect the inplace logic.

However for various reasons I layed out in the relocatable bindist email,
I'd rather have relocatable bindists and a clean source tree with build artifacts completely outside of it, instead of dirtying my source tree with each
build.

My intention is to have something workable sometime next week.

This will render using the make and hadrian build systems concurrently useless though. As the make build system will remain
to expect the inplace logic.

Well, they are mostly incompatible anyway. The only important bit is the ability to run the Make-based testsuite after Hadrian. This needs to be supported in some way until we de-Make-ify the testsuite.

Hmm, I tried validating this and got an error from the dynamic linker about libHSCabal missing. Specifically:

...
bindisttest/ghc.mk:21: recipe for target 'test_bindist' failed
utils/ghc-cabal/dist-install/build/tmp/ghc-cabal: error while loading shared libraries: libHSCabal-2.0.0.2-ghc8.3.20171030.so: cannot open shared object file: No such file or directory
bgamari requested changes to this revision.Oct 29 2017, 9:54 PM
This revision now requires changes to proceed.Oct 29 2017, 9:54 PM

As I've switched over to hacking on hadrian mostly. I wonder how much I should keep pursuing these patches.

ghc.mk
1130

@bgamari I guess this is the culprit.

austin resigned from this revision.Nov 9 2017, 5:40 PM
angerman abandoned this revision.Nov 9 2017, 8:36 PM

I'll focus on doing this in my hadrian fork instead.

trofi added a comment.Nov 29 2017, 4:49 PM

The hsc2hs repo would need to have the following patch

From 94af7d9a27307f40a8b18da0f8e0fd9e9d77e818 Mon Sep 17 00:00:00 2001
From: Moritz Angermann <moritz.angermann@gmail.com>
Date: Sun, 1 Oct 2017 18:07:17 +0800
Subject: [PATCH] Use the same conditional install logic from unlit

This removes the make concurrency bug from the CrossCompilation fix, and uses the
same appraoch `utils/unlit` already uses.
---
 ghc.mk | 21 ++++++++-------------
 1 file changed, 8 insertions(+), 13 deletions(-)

diff --git a/ghc.mk b/ghc.mk
index c5f7473..8221829 100644
--- a/ghc.mk
+++ b/ghc.mk
@@ -5,27 +5,22 @@ utils/hsc2hs_dist_PROGNAME         = hsc2hs
 utils/hsc2hs_dist-install_PROGNAME = hsc2hs
 
 utils/hsc2hs_dist_SHELL_WRAPPER = YES
-utils/hsc2hs_dist_INSTALL = NO
 utils/hsc2hs_dist_INSTALL_INPLACE = YES
 
 utils/hsc2hs_dist-install_SHELL_WRAPPER = YES
-utils/hsc2hs_dist-install_INSTALL = YES
 utils/hsc2hs_dist-install_INSTALL_INPLACE = NO
 
-$(eval $(call build-prog,utils/hsc2hs,dist,0))
-
-# When CrossCompiling, we want to ship the binary for the
-# host, not for the target.  As such we need to compile
-# with the Bootstrap compiler rather than with the in-tree
-# stage1 compiler, which would result in a binary that
-# won't run on the host.
-ifeq "$(CrossCompiling)" "YES"
-# compile with stage 0 (bootstrap compiler)
-$(eval $(call build-prog,utils/hsc2hs,dist-install,0))
+ifeq "$(Stage1Only)" "YES"
+utils/hsc2hs_dist_INSTALL         = YES
+utils/hsc2hs_dist-install_INSTALL = NO
 else
-$(eval $(call build-prog,utils/hsc2hs,dist-install,1))
+utils/hsc2hs_dist_INSTALL         = NO
+utils/hsc2hs_dist-install_INSTALL = YES
 endif
 
+$(eval $(call build-prog,utils/hsc2hs,dist,0))
+$(eval $(call build-prog,utils/hsc2hs,dist-install,1))
+
 # After build-prog above
 utils/hsc2hs_dist_MODULES += Paths_hsc2hs
 utils/hsc2hs_dist-install_MODULES = $(utils/hsc2hs_dist_MODULES)
-- 
2.14.1

applied, to be consistent with the changes in updated changes in here.

I don't see it applied at http://git.haskell.org/hsc2hs.git/history/HEAD:/ghc.mk
Looks like current ghc-HEAD has cross-compiling broken. Simplest

./configure --target=s390x-unknown-linux-gnu && make

fails as:

utils/hsc2hs/ghc.mk:24: utils/hsc2hs/dist-install/package-data.mk: No such file or directory
make[1]: *** [utils/ghc-pkg/ghc.mk:70: utils/ghc-pkg/dist/build/tmp/ghc-pkg] Error 1
make: *** [Makefile:123: all] Error 2

Reverting http://git.haskell.org/hsc2hs.git/commitdiff/ecdac062b5cf1d284906487849c56f4e149b3c8e locally helps build to progress further.