GHCi: Fix load/reload space leaks (#4029)
ClosedPublic

Authored by jme on Feb 24 2016, 9:11 PM.

Details

Summary

This patch addresses GHCi load/reload space leaks which could be
fixed without adversely affecting performance.

Test Plan

make test "TEST=T4029"

Diff Detail

Repository
rGHC Glasgow Haskell Compiler
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
jme updated this revision to Diff 6758.Feb 24 2016, 9:11 PM
jme retitled this revision from to GHCi: Fix load/reload space leaks (#4029).
jme updated this object.
jme edited the test plan for this revision. (Show Details)
jme updated the Trac tickets for this revision.
jme added a comment.Feb 24 2016, 11:58 PM

Please hold off on reviewing this...I just noticed there are some failures in the DEBUG build.

bgamari edited edge metadata.Feb 25 2016, 7:12 AM

Thanks for looking at this @jme, great finds!

I just built a version of GHC with this patch applied and now GHCi is actually usable when loading haskell-src-exts into the repl. Could we please merge it into 8.0? I think it will benefit a lot of users.

@mpickering, sounds reasonable to me.

bgamari requested changes to this revision.Feb 25 2016, 12:19 PM
bgamari edited edge metadata.

I'm going to request changes while @jre confirms and/or fixes the the -DDEBUG issues.

This revision now requires changes to proceed.Feb 25 2016, 12:19 PM

See also D1951 for current test failures with -DDEBUG in HEAD. If you're seeing those, you can ignore them.

I am running the tests now to check.

jme added a comment.Feb 25 2016, 1:23 PM

@mpickering, thanks for the token--I'm glad the patch helps.

I do have a fix for the assertions in GhcMake.hs, but I still have to track down why Data.Map and Data.IntMap have been reversed in T11071.stderr.

mpickering added a comment.EditedFeb 25 2016, 2:33 PM

Here are the failing tests

Unexpected results from:
TEST="T5611 MoreEx T11224b T11071 rnfail047 rnfail055 TcCoercibleFail T3468 recomp001 T11523 MonoidsTF T11480b ClosedFam3 prog006 T6132"

OVERALL SUMMARY for test run started at Thu Feb 25 18:57:52 2016 GMT
 1:31:26 spent to go through
    5032 total tests, which gave rise to
   15811 test cases, of which
   10756 were skipped

      71 had missing libraries
    4853 expected passes
     116 expected failures

       0 caused framework failures
       0 unexpected passes
      15 unexpected failures
       0 unexpected stat failures

Unexpected failures:
   concurrent/should_run      T5611 [bad stderr] (normal)
   driver/recomp001           recomp001 [bad stderr] (normal)
   ghci/prog006               prog006 [bad stderr] (ghci)
   indexed-types/should_fail  ClosedFam3 [stderr mismatch] (normal)
   patsyn/should_compile      MoreEx [exit code non-0] (normal)
   patsyn/should_compile      T11224b [exit code non-0] (normal)
   polykinds                  MonoidsTF [exit code non-0] (normal)
   polykinds                  T11480b [exit code non-0] (normal)
   polykinds                  T11523 [exit code non-0] (normal)
   rename/should_fail         T11071 [stderr mismatch] (normal)
   rename/should_fail         rnfail047 [stderr mismatch] (normal)
   rename/should_fail         rnfail055 [stderr mismatch] (normal)
   runghc                     T6132 [exit code non-0] (normal)
   typecheck/should_fail      T3468 [stderr mismatch] (normal)
   typecheck/should_fail      TcCoercibleFail [stderr mismatch] (normal)
=====> T5611(normal) 1 of 15 [0, 0, 0] 
cd ./concurrent/should_run &&  "/Users/matt/Documents/haskell/ghc/inplace/test   spaces/ghc-stage2" -o T5611 T5611.hs -fforce-recomp -dcore-lint -dcmm-lint -dno-debug-output -no-user-package-db -rtsopts -fno-warn-tabs -fno-warn-missed-specialisations -fno-ghci-history   > T5611.comp.stderr 2>&1
cd ./concurrent/should_run && ./T5611    </dev/null > T5611.run.stdout 2> T5611.run.stderr
=====> recomp001(normal) 2 of 15 [0, 0, 0] 
cd ./driver/recomp001 && $MAKE -s --no-print-directory recomp001    </dev/null > recomp001.run.stdout 2> recomp001.run.stderr
cd ./driver/recomp001 && $MAKE -s clean
=====> prog006(ghci) 3 of 15 [0, 0, 0] 
cd ./ghci/prog006 && HC="/Users/matt/Documents/haskell/ghc/inplace/test   spaces/ghc-stage2" HC_OPTS="-dcore-lint -dcmm-lint -dno-debug-output -no-user-package-db -rtsopts -fno-warn-tabs -fno-warn-missed-specialisations -fno-ghci-history " "/Users/matt/Documents/haskell/ghc/inplace/test   spaces/ghc-stage2" -dcore-lint -dcmm-lint -dno-debug-output -no-user-package-db -rtsopts -fno-warn-tabs -fno-warn-missed-specialisations -fno-ghci-history  --interactive -v0 -ignore-dot-ghci +RTS -I0.1 -RTS    <prog006.script > prog006.run.stdout 2> prog006.run.stderr
Actual stderr output differs from expected:
--- ./ghci/prog006/prog006.stderr.normalised	2016-02-25 20:38:39.000000000 +0000
+++ ./ghci/prog006/prog006.run.stderr.normalised	2016-02-25 20:38:39.000000000 +0000
@@ -2,3 +2,4 @@
 Boot.hs:5:13:
     Not a data constructor: ‘forall’
     Perhaps you intended to use ExistentialQuantification
\ No newline at end of file
+*** Exception: ASSERT failed! file compiler/main/GhcMake.hs, line 370
\ No newline at end of file
*** unexpected failure for prog006(ghci)
=====> ClosedFam3(normal) 4 of 15 [0, 1, 0] 
cd ./indexed-types/should_fail &&  "/Users/matt/Documents/haskell/ghc/inplace/test   spaces/ghc-stage2" --make  ClosedFam3 -fforce-recomp -dcore-lint -dcmm-lint -dno-debug-output -no-user-package-db -rtsopts -fno-warn-tabs -fno-warn-missed-specialisations -fno-ghci-history  -v0 > ClosedFam3.comp.stderr 2>&1
Actual stderr output differs from expected:
--- ./indexed-types/should_fail/ClosedFam3.stderr.normalised	2016-02-25 20:38:39.000000000 +0000
+++ ./indexed-types/should_fail/ClosedFam3.comp.stderr.normalised	2016-02-25 20:38:39.000000000 +0000
@@ -29,3 +29,8 @@
     Boot file:   type family Baz (a :: k) :: *
                    where Baz * Int = Bool
     The types have different kinds
\ No newline at end of file
+ghc: panic! (the 'impossible' happened)
+  (GHC version 8.1.20160225 for x86_64-apple-darwin):
+	ASSERT failed! file compiler/main/GhcMake.hs, line 370
+
+Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug
\ No newline at end of file
*** unexpected failure for ClosedFam3(normal)
=====> T11224b(normal) 5 of 15 [0, 2, 0] 
cd ./patsyn/should_compile &&  "/Users/matt/Documents/haskell/ghc/inplace/test   spaces/ghc-stage2" -c T11224b.hs -fforce-recomp -dcore-lint -dcmm-lint -dno-debug-output -no-user-package-db -rtsopts -fno-warn-tabs -fno-warn-missed-specialisations -fno-ghci-history   > T11224b.comp.stderr 2>&1
Compile failed (status 256) errors were:
ghc-stage2: panic! (the 'impossible' happened)
  (GHC version 8.1.20160225 for x86_64-apple-darwin):
	ASSERT failed!
  CallStack (from HasCallStack):
  assertPprPanic, called at compiler/types/TyCoRep.hs:1944:28 in ghc:TyCoRep
  checkValidSubst, called at compiler/types/TyCoRep.hs:2003:17 in ghc:TyCoRep
  substTys, called at compiler/types/TyCoRep.hs:2024:14 in ghc:TyCoRep
  substTheta, called at compiler/typecheck/TcPatSyn.hs:255:20 in ghc:TcPatSyn
  in_scope InScope {d_ap3 c_apy}
  tenv [ap4 :-> c_apy[tau:5]]
  tenvFVs [apv :-> t_apv[tau:1], apy :-> c_apy[tau:5]]
  cenv []
  cenvFVs []
  tys []
  cos []

Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug


*** unexpected failure for T11224b(normal)
=====> MoreEx(normal) 6 of 15 [0, 3, 0] 
cd ./patsyn/should_compile &&  "/Users/matt/Documents/haskell/ghc/inplace/test   spaces/ghc-stage2" -c MoreEx.hs -fforce-recomp -dcore-lint -dcmm-lint -dno-debug-output -no-user-package-db -rtsopts -fno-warn-tabs -fno-warn-missed-specialisations -fno-ghci-history   > MoreEx.comp.stderr 2>&1
Compile failed (status 256) errors were:
ghc-stage2: panic! (the 'impossible' happened)
  (GHC version 8.1.20160225 for x86_64-apple-darwin):
	ASSERT failed!
  CallStack (from HasCallStack):
  assertPprPanic, called at compiler/types/TyCoRep.hs:1944:28 in ghc:TyCoRep
  checkValidSubst, called at compiler/types/TyCoRep.hs:2003:17 in ghc:TyCoRep
  substTys, called at compiler/types/TyCoRep.hs:2024:14 in ghc:TyCoRep
  substTheta, called at compiler/typecheck/TcPatSyn.hs:255:20 in ghc:TcPatSyn
  in_scope InScope {b_axe c_axf}
  tenv [aw9 :-> b_axe[tau:5], awa :-> c_axf[tau:5]]
  tenvFVs [ax0 :-> t_ax0[tau:1], ax2 :-> t_ax2[tau:1],
           axe :-> b_axe[tau:5], axf :-> c_axf[tau:5]]
  cenv []
  cenvFVs []
  tys []
  cos []

Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug


*** unexpected failure for MoreEx(normal)
=====> MonoidsTF(normal) 7 of 15 [0, 4, 0] 
cd ./polykinds &&  "/Users/matt/Documents/haskell/ghc/inplace/test   spaces/ghc-stage2" -o MonoidsTF MonoidsTF.hs -fforce-recomp -dcore-lint -dcmm-lint -dno-debug-output -no-user-package-db -rtsopts -fno-warn-tabs -fno-warn-missed-specialisations -fno-ghci-history   > MonoidsTF.comp.stderr 2>&1
Compile failed (status 256) errors were:
[1 of 1] Compiling Main             ( MonoidsTF.hs, MonoidsTF.o )
ghc-stage2: panic! (the 'impossible' happened)
  (GHC version 8.1.20160225 for x86_64-apple-darwin):
	ASSERT failed!
  CallStack (from HasCallStack):
  assertPprPanic, called at compiler/types/TyCoRep.hs:1944:28 in ghc:TyCoRep
  checkValidSubst, called at compiler/types/TyCoRep.hs:1980:17 in ghc:TyCoRep
  substTy, called at compiler/typecheck/TcHsType.hs:852:27 in ghc:TcHsType
  in_scope InScope {k1_a1dy k0_a1dz}
  tenv [aVB :-> k0_a1dz[tau:1], aVC :-> k1_a1dy[tau:1]]
  tenvFVs [a1dv :-> t_a1dv[tau:1], a1dy :-> k1_a1dy[tau:1],
           a1dz :-> k0_a1dz[tau:1]]
  cenv []
  cenvFVs []
  tys [(k0_aVB[sk] -> k1_aVC[sk] -> *)
       -> k1_aVC[sk] -> k1_aVC[sk] -> k1_aVC[sk] -> k0_aVB[sk]]
  cos []

Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug


*** unexpected failure for MonoidsTF(normal)
=====> T11480b(normal) 8 of 15 [0, 5, 0] 
cd ./polykinds &&  "/Users/matt/Documents/haskell/ghc/inplace/test   spaces/ghc-stage2" -c T11480b.hs -fforce-recomp -dcore-lint -dcmm-lint -dno-debug-output -no-user-package-db -rtsopts -fno-warn-tabs -fno-warn-missed-specialisations -fno-ghci-history   > T11480b.comp.stderr 2>&1
Compile failed (status 256) errors were:
ghc-stage2: panic! (the 'impossible' happened)
  (GHC version 8.1.20160225 for x86_64-apple-darwin):
	ASSERT failed!
  CallStack (from HasCallStack):
  assertPprPanic, called at compiler/types/TyCoRep.hs:1944:28 in ghc:TyCoRep
  checkValidSubst, called at compiler/types/TyCoRep.hs:1980:17 in ghc:TyCoRep
  substTy, called at compiler/typecheck/TcHsType.hs:852:27 in ghc:TcHsType
  in_scope InScope {i_aKr}
  tenv [aDY :-> i_aKr[tau:1]]
  tenvFVs [aKf :-> t_aKf[tau:1], aKr :-> i_aKr[tau:1]]
  cenv []
  cenvFVs []
  tys [(i_aDY[sk] -> i_aDY[sk] -> *) -> i_aDY[sk] -> Constraint]
  cos []

Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug


*** unexpected failure for T11480b(normal)
=====> T11523(normal) 9 of 15 [0, 6, 0] 
cd ./polykinds &&  "/Users/matt/Documents/haskell/ghc/inplace/test   spaces/ghc-stage2" -c T11523.hs -fforce-recomp -dcore-lint -dcmm-lint -dno-debug-output -no-user-package-db -rtsopts -fno-warn-tabs -fno-warn-missed-specialisations -fno-ghci-history   > T11523.comp.stderr 2>&1
Compile failed (status 256) errors were:
ghc-stage2: panic! (the 'impossible' happened)
  (GHC version 8.1.20160225 for x86_64-apple-darwin):
	ASSERT failed!
  CallStack (from HasCallStack):
  assertPprPanic, called at compiler/types/TyCoRep.hs:1944:28 in ghc:TyCoRep
  checkValidSubst, called at compiler/types/TyCoRep.hs:1980:17 in ghc:TyCoRep
  substTy, called at compiler/typecheck/TcHsType.hs:852:27 in ghc:TcHsType
  in_scope InScope {j_aDu i_aDv}
  tenv [aBz :-> i_aDv[tau:1], aBA :-> j_aDu[tau:1]]
  tenvFVs [aDk :-> t_aDk[tau:1], aDu :-> j_aDu[tau:1],
           aDv :-> i_aDv[tau:1]]
  cenv []
  cenvFVs []
  tys [(i_aBz[sk] -> j_aBA[sk]) -> Cat i_aBz[sk]]
  cos []

Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug


*** unexpected failure for T11523(normal)
=====> rnfail047(normal) 10 of 15 [0, 7, 0] 
cd ./rename/should_fail &&  "/Users/matt/Documents/haskell/ghc/inplace/test   spaces/ghc-stage2" --make  rnfail047 -fforce-recomp -dcore-lint -dcmm-lint -dno-debug-output -no-user-package-db -rtsopts -fno-warn-tabs -fno-warn-missed-specialisations -fno-ghci-history  -v0 > rnfail047.comp.stderr 2>&1
Actual stderr output differs from expected:
--- ./rename/should_fail/rnfail047.stderr.normalised	2016-02-25 20:38:41.000000000 +0000
+++ ./rename/should_fail/rnfail047.comp.stderr.normalised	2016-02-25 20:38:41.000000000 +0000
@@ -1,3 +1,8 @@
 
 RnFail047_A.hs-boot:5:1:
     ‘RnFail047_A.y’ is exported by the hs-boot file, but not exported by the module
\ No newline at end of file
+ghc: panic! (the 'impossible' happened)
+  (GHC version 8.1.20160225 for x86_64-apple-darwin):
+	ASSERT failed! file compiler/main/GhcMake.hs, line 370
+
+Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug
\ No newline at end of file
*** unexpected failure for rnfail047(normal)
=====> rnfail055(normal) 11 of 15 [0, 8, 0] 
cd ./rename/should_fail &&  "/Users/matt/Documents/haskell/ghc/inplace/test   spaces/ghc-stage2" --make  RnFail055 -fforce-recomp -dcore-lint -dcmm-lint -dno-debug-output -no-user-package-db -rtsopts -fno-warn-tabs -fno-warn-missed-specialisations -fno-ghci-history  -v0 > rnfail055.comp.stderr 2>&1
Actual stderr output differs from expected:
--- ./rename/should_fail/rnfail055.stderr.normalised	2016-02-25 20:38:41.000000000 +0000
+++ ./rename/should_fail/rnfail055.comp.stderr.normalised	2016-02-25 20:38:41.000000000 +0000
@@ -99,3 +99,8 @@
     Main module: class (Eq a, Ord a) => C3 a
     Boot file:   class (Ord a, Eq a) => C3 a
     The class constraints do not match
\ No newline at end of file
+ghc: panic! (the 'impossible' happened)
+  (GHC version 8.1.20160225 for x86_64-apple-darwin):
+	ASSERT failed! file compiler/main/GhcMake.hs, line 370
+
+Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug
\ No newline at end of file
*** unexpected failure for rnfail055(normal)
=====> T11071(normal) 12 of 15 [0, 9, 0] 
cd ./rename/should_fail &&  "/Users/matt/Documents/haskell/ghc/inplace/test   spaces/ghc-stage2" -c T11071.hs -fforce-recomp -dcore-lint -dcmm-lint -dno-debug-output -no-user-package-db -rtsopts -fno-warn-tabs -fno-warn-missed-specialisations -fno-ghci-history   > T11071.comp.stderr 2>&1
Actual stderr output differs from expected:
--- ./rename/should_fail/T11071.stderr.normalised	2016-02-25 20:38:42.000000000 +0000
+++ ./rename/should_fail/T11071.comp.stderr.normalised	2016-02-25 20:38:42.000000000 +0000
@@ -9,11 +9,11 @@
 
 T11071.hs:21:12:
     Not in scope: ‘M.foobar’
-    Neither ‘Data.Map’ nor ‘Data.IntMap’ exports ‘foobar’.
+    Neither ‘Data.IntMap’ nor ‘Data.Map’ exports ‘foobar’.
 
 T11071.hs:22:12:
     Not in scope: ‘M'.foobar’
-    Neither ‘Data.Map’, ‘Data.IntMap’ nor ‘System.IO’ exports ‘foobar’.
+    Neither ‘Data.IntMap’, ‘Data.Map’ nor ‘System.IO’ exports ‘foobar’.
 
 T11071.hs:23:12:
     Not in scope: ‘Data.List.sort’
@@ -29,8 +29,8 @@
 T11071.hs:25:12:
     Not in scope: ‘M.size’
     Perhaps you want to add ‘size’ to one of these import lists:
-      ‘Data.Map’ (T11071.hs:4:1-33)
       ‘Data.IntMap’ (T11071.hs:5:1-36)
+      ‘Data.Map’ (T11071.hs:4:1-33)
 
 T11071.hs:26:12:
     Not in scope: ‘M.valid’
@@ -49,5 +49,5 @@
     Not in scope: ‘M'.size’
     Perhaps you want to remove ‘size’ from the hiding clauses
     in one of these imports:
-      ‘Data.Map’ (T11071.hs:10:1-53)
       ‘Data.IntMap’ (T11071.hs:12:1-48)
\ No newline at end of file
+      ‘Data.Map’ (T11071.hs:10:1-53)
\ No newline at end of file
*** unexpected failure for T11071(normal)
=====> T6132(normal) 13 of 15 [0, 10, 0] 
cd ./runghc &&  "/Users/matt/Documents/haskell/ghc/inplace/test   spaces/ghc-stage2" -c T6132.hs -fforce-recomp -dcore-lint -dcmm-lint -dno-debug-output -no-user-package-db -rtsopts -fno-warn-tabs -fno-warn-missed-specialisations -fno-ghci-history   > T6132.comp.stderr 2>&1
Compile failed (status 256) errors were:

T6132.hs:1:2: error: parse error on input ‘#!/’

*** unexpected failure for T6132(normal)
=====> T3468(normal) 14 of 15 [0, 11, 0] 
cd ./typecheck/should_fail &&  "/Users/matt/Documents/haskell/ghc/inplace/test   spaces/ghc-stage2" --make  T3468 -fforce-recomp -dcore-lint -dcmm-lint -dno-debug-output -no-user-package-db -rtsopts -fno-warn-tabs -fno-warn-missed-specialisations -fno-ghci-history  -v0 > T3468.comp.stderr 2>&1
Actual stderr output differs from expected:
--- ./typecheck/should_fail/T3468.stderr.normalised	2016-02-25 20:38:42.000000000 +0000
+++ ./typecheck/should_fail/T3468.comp.stderr.normalised	2016-02-25 20:38:42.000000000 +0000
@@ -7,3 +7,8 @@
                    F :: a -> Tool d
     Boot file:   abstract Tool
     The types have different kinds
\ No newline at end of file
+ghc: panic! (the 'impossible' happened)
+  (GHC version 8.1.20160225 for x86_64-apple-darwin):
+	ASSERT failed! file compiler/main/GhcMake.hs, line 370
+
+Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug
\ No newline at end of file
*** unexpected failure for T3468(normal)
=====> TcCoercibleFail(normal) 15 of 15 [0, 12, 0] 
cd ./typecheck/should_fail &&  "/Users/matt/Documents/haskell/ghc/inplace/test   spaces/ghc-stage2" -c TcCoercibleFail.hs -fforce-recomp -dcore-lint -dcmm-lint -dno-debug-output -no-user-package-db -rtsopts -fno-warn-tabs -fno-warn-missed-specialisations -fno-ghci-history   > TcCoercibleFail.comp.stderr 2>&1
^C
Unexpected results from:
TEST="MoreEx T11224b T11071 rnfail047 rnfail055 T3468 T11523 MonoidsTF T11480b ClosedFam3 prog006 T6132"

OVERALL SUMMARY for test run started at Thu Feb 25 20:38:33 2016 GMT
 0:00:32 spent to go through
      15 total tests, which gave rise to
      37 test cases, of which
      22 were skipped

       0 had missing libraries
       2 expected passes
       0 expected failures

       0 caused framework failures
       0 unexpected passes
      12 unexpected failures
       0 unexpected stat failures

Unexpected failures:
   ghci/prog006               prog006 [bad stderr] (ghci)
   indexed-types/should_fail  ClosedFam3 [stderr mismatch] (normal)
   patsyn/should_compile      MoreEx [exit code non-0] (normal)
   patsyn/should_compile      T11224b [exit code non-0] (normal)
   polykinds                  MonoidsTF [exit code non-0] (normal)
   polykinds                  T11480b [exit code non-0] (normal)
   polykinds                  T11523 [exit code non-0] (normal)
   rename/should_fail         T11071 [stderr mismatch] (normal)
   rename/should_fail         rnfail047 [stderr mismatch] (normal)
   rename/should_fail         rnfail055 [stderr mismatch] (normal)
   runghc                     T6132 [exit code non-0] (normal)
   typecheck/should_fail      T3468 [stderr mismatch] (normal)

WARNING: Testsuite run was terminated early
make: *** [test] Interrupt: 2
jme updated this revision to Diff 6780.Feb 26 2016, 11:37 AM
jme edited edge metadata.

Fix bugs flagged by ./validate

  • Update ASSERT in GhcMake.hs to handle the presence of .hs-boot files as well as flags which disable linking
  • Reverse the order of Data.Map and Data.IntMap in the error messages for T11071
  • Update memory consumption stats for T4029
jme added a comment.Feb 26 2016, 11:47 AM

One comment about the order of Data.Map and Data.IntMap in the error message for T11071. In the message, the order of the modules is determined in part by the uniques for the associated ModuleName FastStrings. Since these uniques are allocated lazily, adding strictness to PackageState changed the order in which the ModuleNames are forced. Before, the order was based on that of the imports in the module; now, it is more similar to the ordering of the ExposedModules in the serialized PackageConfig. Given all this, it might make sense at some point to change the error message so the modules are simply listed in lexicographic order.

Also, I have fixes (more accurately, blunt hammers) for the remaining large leaks I found, but they all incur a performance penalty (which is why I didn't include them in the patch). If memory consumption becomes an issue for people, I'll go back and see if I can come up with fixes which are a little less crude.

bgamari accepted this revision.Feb 29 2016, 8:15 AM
bgamari edited edge metadata.

Thanks @jme! This looks great.

This revision is now accepted and ready to land.Feb 29 2016, 8:15 AM
This revision was automatically updated to reflect the committed changes.