RTS: Decrease MBLOCK_SPACE_SIZE size to 1/4 TB
ClosedPublic

Authored by erikd on Aug 24 2015, 9:28 PM.

Details

Summary

Commit 0d1a8d09f4 added a two step allocator for 64 bit systems. This
allocator mmaps a huge (originally 1 TB) chunk of memory out of which
it does smaller allocations. On AArch64/Arm64 linux, this mmap was
failing due to the Arm64 Linux kernel parameter CONFIG_ARM64_VA_BITS
defaulting to 39. Reducing the allocation from 1 TB to 1/4 TB fixed it.

Diff Detail

Repository
rGHC Glasgow Haskell Compiler
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
erikd updated this revision to Diff 3915.Aug 24 2015, 9:28 PM
erikd retitled this revision from to RTS: Decrease MBLOCK_SPACE_SIZE size to 1/4 TB.
erikd updated this object.
erikd edited the test plan for this revision. (Show Details)
erikd added a reviewer: ezyang.
erikd updated the Trac tickets for this revision.
ezyang accepted this revision.Aug 24 2015, 11:49 PM
ezyang edited edge metadata.

Maybe we should make the size configurable, in case there are Linux users that need more than a quarter terabyte of memory.

erikd added a comment.Aug 25 2015, 1:47 AM

Is there really anyone using GHC on hardware where they actually have 1/4 TB of virtual memory?

I would have thought that was still a good couple of years away and if its that far away, this code will surely churn anyway.

I have run Haskell code on machines in the past with several hundred gigabytes of physical memory. In this particular case I was using the machine for its cores, not its memory, but nevertheless I can easily imagine this changing in the future.

Hundreds of gigabytes of memory really isn't all that odd anymore. For 10000 USD you can have a machine with these sorts of numbers (memory alone costing around 5000 USD).

The point is, I think we should definitely leave this knob open to the users. Would we want this to be a compile-time option or a runtime RTS option?

Making it runtime has a cost, since the HEAP_ALLOCED test now has to do a memory reference to find the space size. On the other hand, we already do a memory ref to find out where the space starts, so it might not cost too much.

@ezyang, IIRC if you ensure the two sit in the same cache line the access is essentially free.

austin requested changes to this revision.Aug 25 2015, 2:01 PM
austin edited edge metadata.
austin added a subscriber: simonmar.

I can literally allocate/lease a machine with 1/2 TB of physical RAM in about 5 minutes, on your command, so yes this stuff does exist.

Personally I'm in favor of making it runtime, because that's far more sensible anyway. With a change like that, we can easily have the RTS default to some adjusted value e.g. on aarch64_HOST_ARCH, and on amd64_HOST_ARCH default to what we currently have. Just looking at the patch, since we have to load mblock_address_space_begin out of the data anyway, adding mblock_space_size for another load seems like it would be pretty cheap for the most part. I'd hope locality would be in our favor here too.

The real way to check of course is to look at what the assembly says for HEAP_ALLOCED, and also do a benchmark. @simonmar will probably have to follow through on that one.

Alternatively, I can probably figure out a workload with the aforementioned machine-with-512GB-RAM to make these sort of 'small cases' become very large and problematic. Maybe we should have some stress tests for 'insanely large heaps' that we can run things like this over? Can anyone think of something 'obvious' that we could scale to that much memory?

This revision now requires changes to proceed.Aug 25 2015, 2:01 PM

Oh, and at the *minimum*, we should change this even if it doesn't go to runtime: just #ifdef MBLOCK_SPACE_SIZE and make it 1/4TB on AArch64, or otherwise 1TB.

erikd updated this revision to Diff 3928.Aug 25 2015, 9:37 PM
erikd edited edge metadata.

MBLOCK_SPACE_SIZE only changes for aarch64_HOST_ARCH.

simonmar accepted this revision.Aug 27 2015, 9:41 AM
simonmar added a reviewer: simonmar.

Is there really anyone using GHC on hardware where they actually have 1/4 TB of virtual memory?

this is my dev box:

$ cat /proc/meminfo 
MemTotal:       263921216 kB

Let's go with the compile-time fix for AArch64 only.

bgamari accepted this revision.Aug 28 2015, 5:06 AM
bgamari edited edge metadata.

Is there really anyone using GHC on hardware where they actually have 1/4 TB of virtual memory?

this is my dev box:

$ cat /proc/meminfo 
MemTotal:       263921216 kB

Um, wow.

Let's go with the compile-time fix for AArch64 only.

Sounds reasoanble to me.

This revision was automatically updated to reflect the committed changes.

this is my dev box:

$ cat /proc/meminfo 
MemTotal:       263921216 kB

ohhrapbattle