Hadrian: ability to run a subset of the testsuite
ClosedPublic

Authored by alpmestan on Dec 10 2018, 12:25 PM.

Details

Summary

This was supposed to be working already but didn't work when we
specified several tests with --only. This patch not only fixes this
but also makes it possible to specify a subset of tests to run with the
TEST environment variable, like the make build system. Here are some
examples:

hadrian/build.sh test --only=plugins01
hadrian/build.sh test --only="plugins01 plugins02"
TEST="plugins01 plugins02" hadrian/build.sh test
TEST=plugins03 hadrian/build.sh test --only="plugins01 plugins02"

When both the TEST environment variable and the --only flag are used,
we simply concatenate the list of tests from both sources and ask the
testsuite driver to run them all.

This patch addresses Trac #16026.

Test Plan

hadrian/build.sh test --only="plugins01 plugins02"

Diff Detail

Repository
rGHC Glasgow Haskell Compiler
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
alpmestan created this revision.Dec 10 2018, 12:25 PM
snowleopard accepted this revision.Dec 10 2018, 12:38 PM

This looks great. Thank you for the patch!

This revision is now accepted and ready to land.Dec 10 2018, 12:38 PM
bgamari requested changes to this revision.Dec 10 2018, 2:40 PM

Thank you for doing this!

However, we certainly need to document it somewhere.

This revision now requires changes to proceed.Dec 10 2018, 2:40 PM

However, we certainly need to document it somewhere.

I was planning on updating the Hadrian quickstart page with a dedicated testsuite section, explaining everything one can tweak when running the testsuite.

(And also a word about the simpler targets for libs and exes, when it lands.)

Where else do you want this to be documented?

Where else do you want this to be documented?

There is a README plus some files in the doc directory in Hadrian, where we could be documenting the build system, and where the information about the testsuite could go. This has the benefit of being tracked together with the code/patches etc., so it's less likely to go out of date. However, it seems that documenting things on Trac is a lot more common among GHC developers, so I'm not sure what's the right answer is.

If the answer is: we document everything on Trac, then I guess we should just delete the README and doc/* since they will go out of date very quickly.

Where else do you want this to be documented?

There is a README plus some files in the doc directory in Hadrian, where we could be documenting the build system, and where the information about the testsuite could go. This has the benefit of being tracked together with the code/patches etc., so it's less likely to go out of date. However, it seems that documenting things on Trac is a lot more common among GHC developers, so I'm not sure what's the right answer is.

If the answer is: we document everything on Trac, then I guess we should just delete the README and doc/* since they will go out of date very quickly.

Personally I am in favor of rather documenting everything in the tree; when we move to GitLab users can easily link to and view the documentation online anyways.

alpmestan added a comment.EditedDec 11 2018, 1:23 AM

Hmm, right, well the quickstart page being on trac and with the move to gitlab, I guess it's not such a great idea to focus the documentation effort there anyway. It's just absolutely not the direction I've been taking in the past few months. I'll add a few words about what this patch allows in the README and then perhaps start thinking about enhancing the README and doc/ directory with the contents that I ended up putting in the quickstart page. This latter part would be in another patch though of course.

alpmestan updated this revision to Diff 19086.Dec 11 2018, 5:11 AM
  • in-tree documentation for the testsuite rule and its options

I just updated the patch with a new file, hadrian/doc/testsuite.md, with a bunch of illustrated examples for most of the testsuite options (all the important ones anyway). And hadrian's readme now links to it in the testing section.

bgamari accepted this revision.Dec 11 2018, 12:11 PM

Great documentation!

This revision is now accepted and ready to land.Dec 11 2018, 12:11 PM
snowleopard accepted this revision.Dec 11 2018, 1:07 PM

Great work!

This revision was automatically updated to reflect the committed changes.