Add NUMA support for Windows
ClosedPublic

Authored by Phyx on Sep 17 2016, 10:51 AM.

Details

Summary
NOTE: I have been able to do simple testing on emulated NUMA nodes. Real hardware would be needed for a proper test.

D2199 Added NUMA support for Linux, I have just filled in the missing pieces following
the description of the Linux APIs.

Test Plan

Use bcdedit.exe /set groupsize 2 to modify the kernel again (Similar to D2533).

This generates some NUMA nodes:

Logical Processor to NUMA Node Map:
NUMA Node 0:
**
--
NUMA Node 1:
--
**

Approximate Cross-NUMA Node Access Cost (relative to fastest):
     00  01
00: 1.1 1.1
01: 1.0 1.0

run ../test-numa.exe +RTS --numa -RTS

and check PerfMon for NUMA allocations.

Diff Detail

Repository
rGHC Glasgow Haskell Compiler
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
Phyx updated this revision to Diff 8732.Sep 17 2016, 10:51 AM
Phyx retitled this revision from to Add NUMA support for Windows.
Phyx updated this object.
Phyx edited the test plan for this revision. (Show Details)
Phyx updated the Trac tickets for this revision.
Phyx added a subscriber: Restricted Project.
Phyx updated this revision to Diff 8734.Sep 18 2016, 2:20 PM
Phyx edited edge metadata.
  • NUMA: fix off by one
simonmar requested changes to this revision.Sep 18 2016, 9:15 PM
simonmar edited edge metadata.
simonmar added inline comments.
rts/win32/OSMem.c
509–518

You're calling osNumaAvailable() in various places which calls osNumaNodes(), so it would be better to compute these results once and cache them.

540

Perhaps you want to do this once in osMemInit() and save the result, rather than during every call to osBindMBlocksToNode()?

This revision now requires changes to proceed.Sep 18 2016, 9:15 PM
Phyx updated this revision to Diff 8768.Sep 24 2016, 2:11 PM
Phyx edited edge metadata.
  • T12602: cache numa values.
Phyx planned changes to this revision.Sep 25 2016, 12:38 PM

NUMA check seems to be called before osMemInit. Need to refactor.

Phyx updated this revision to Diff 8781.Sep 25 2016, 2:01 PM
Phyx edited edge metadata.
  • T12602: Fix NUMA caching
  • T12602: added note.
Phyx edited the test plan for this revision. (Show Details)Sep 25 2016, 2:07 PM
Phyx edited edge metadata.
Phyx updated this object.
Phyx edited the test plan for this revision. (Show Details)
Phyx added a comment.Sep 25 2016, 2:10 PM

@simonmar Should ../test-numa.exe +RTS --numa -N -qa -RTS be working?

There is a weird race condition in initCapabilities where it dies with unless I slow it down by e.g. attaching a debugger:

$ ../test-numa.exe +RTS --numa -N -qa -RTS
test-numa.exe: internal error: (null): available NUMA node set is empty
    (GHC version 8.1.20160906 for x86_64_unknown_mingw32)
    Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

The code isn't really thread-safe, but isn't init always done single threaded?

simonmar accepted this revision.Sep 26 2016, 7:51 AM
simonmar edited edge metadata.

Patch looks fine. I have no idea where that race condition could come from - what test program are you using?

This revision is now accepted and ready to land.Sep 26 2016, 7:51 AM
Phyx added a comment.Sep 26 2016, 1:26 PM

@simonmar Thanks for the review!

My test case is

import Control.Concurrent
import Control.Monad

main = do ids <- replicateM 9 (forkIO $ print $ length $ primesToG 100000000000)
          spin

spin = do threadDelay 1000
          yield
          spin       

primesToG m = 2 : sieve [3,5..m]
  where
    sieve (p:xs) 
       | p*p > m   = p : xs
       | otherwise = p : sieve (xs `minus` [p*p, p*p+2*p..])
       
minus (x:xs) (y:ys) = case (compare x y) of 
           LT -> x : minus  xs  (y:ys)
           EQ ->     minus  xs     ys 
           GT ->     minus (x:xs)  ys
minus  xs     _     = xs

I looked briefly but couldn't understand where the race condition was coming from.
I'll make a note of it.

Phyx updated this revision to Diff 8841.Oct 1 2016, 5:15 PM
Phyx edited edge metadata.
rebased
Phyx updated this object.Oct 1 2016, 5:19 PM
Phyx edited edge metadata.
This revision was automatically updated to reflect the committed changes.