Allow setting explicit thread affinity masks from a file.
Needs RevisionPublic

Authored by fryguybob on Apr 3 2015, 12:22 PM.

Details

Reviewers
simonmar
austin
Trac Issues
#10229
Summary

The existing thread affinity mechanism is limited to a specific mapping of RTS capabilities to hardware cores. This patch allows a user specified mapping via a file where each line is the thread affinity mask and capabilities are mapped to masks by indexing into lines (wrapping around if there are more capabilities then lines in the file). Each mask is given by specifying the allowed cores by number (zero-indexed) separated by spaces.

I've been using this for the last three months and it is critical to getting consistent results on a large machine, in my case, 72-threads across two sockets.

This patch is not comprehensive and only applies to Linux. There may be better directions to go, but hopefully this will start the conversation and through some discussion we can land at a good solution.

Things that still need to be addressed:

  • Supporting other OSs.
  • Documentation.
  • Tests for parsing.
  • RTS flags for convenient common configurations.
Test Plan

I've testing the effectiveness of the patch by hand, running a program that makes N cores busy then watching in htop to see if the right cores become busy. Several tests should be written for parsing.

Diff Detail

Repository
rGHC Glasgow Haskell Compiler
Lint
Lint Skipped
Unit
Unit Tests Skipped
Build Status
Buildable 3626
Build 3656: GHC Patch Validation (amd64/Linux)
fryguybob updated this revision to Diff 2645.Apr 3 2015, 12:22 PM
fryguybob retitled this revision from to Allow setting explicit thread affinity masks from a file..
fryguybob updated this object.
fryguybob edited the test plan for this revision. (Show Details)
fryguybob set the repository for this revision to rGHC Glasgow Haskell Compiler.
fryguybob updated the Trac tickets for this revision.
fryguybob updated this revision to Diff 2650.Apr 3 2015, 7:34 PM
fryguybob edited edge metadata.

Fix build warning of unused set variable and wrong type for lines counter.

fryguybob updated this revision to Diff 2651.Apr 3 2015, 9:19 PM

Another attempt to fix warning.

simonmar requested changes to this revision.Apr 5 2015, 2:21 PM
simonmar added a reviewer: simonmar.
simonmar added a subscriber: simonmar.

Generalising this is good in principle. There are a couple of things I'd like to see done differently, though.

  1. There should be a way to do this with command-line options. I don't mind if we also have the file method. They might as well have the same syntax, though. e.g.
prog +RTS -N2 -qa0=1,2,3,4;1=5,6,7,8
  1. The platform-specific code should go in rts/posix and be exposed via a generic API.
This revision now requires changes to proceed.Apr 5 2015, 2:21 PM

@simonmar Those are good suggestions. I initially considered just using the command-line, but for my use, 72 threads was a bit unwieldy. For lower thread count that will be convenient. Another format that can be useful in the middle is a hex bit mask especially if you want to choose multiple threads per capability. In my use, I'm always pinning each capability to exactly one thread.

bgamari added a subscriber: bgamari.Nov 3 2015, 9:51 AM
carter added a subscriber: carter.May 6 2016, 8:30 PM

whats needed to move this or something similar to this along?

@carter See the "Things that still need to be addressed:" part of the Summary. The file format that I'm reading here is rather ad hoc and aimed at being the simplest thing to do while reading it in C. I would hope for some better user facing interface such as command-line options for at least some common configurations while still allowing full mappings to be specified. I don't have much time at the moment to expand on those ideas and the patch as it is works for (and is quite critical for) my research.