Server Admin Log
Updated 1,065 Days AgoPublic

Date: 2016-12-17

Date: 2016-09-13

  • 13:07 CDT @davean: major hackage.h.o CDN config changes, gzip on docs, 404 forced non-cache, X-Forwarded-Proto header, backend over HTTPS, ...

Date: 2016-06-16

Date: 2016-06-02

Date: 2016-05-30

  • 18:26 CDT @davean: change builder to GHC 8

Date: 2016-05-26

  • 23:47 CDT @gershomb: created for our webmon live status

Date: 2016-05-03

  • 20:36 CDT @austin: upgraded Phab to latest HEAD

Date: 2016-04-29

Date: 2016-02-12

  • 14:43 CST @austin: Upgraded to latest HEAD, full steam away on new buildbot set up!

Date: 2016-02-11

  • 17:55 CST @davean: Builder fixed and catching up

Date: 2016-02-01

  • 20:07 CST @davean: Added mikhail to the cabal-site sftp system on www-origin

Date: 2016-01-21

  • 10:31 CST @austin: Finally upgraded Phabricator, so many changes omg

Date: 2015-11-20

  • 16:16 CST @davean: Update many Fastly backends to use SSL

Date: 2015-10-05

  • 11:09 CDT @davean: Turn off cloudflare's proxying of in the hopes the site would be more up

Date: 2015-09-27

  • 02:21 CDT @hvr: upgraded Trac on {ghc,prime}.h.o to v1.0.9

Date: 2015-09-14

  • 16:15 CDT @austin: update Phabricator to latest HEAD - Remarkup now supports Cowsay and Figlet?

Date: 2015-08-27

Date: 2015-08-25

  • 09:53 CDT @hvr: restarting apache2 on ghc.h.o due to strange high cpu load

Date: 2015-08-24

  • 20:25 CDT @austin: updated to latest phabricator HEAD and whatnot. also restart @phaskell because you stupidly died, you stupid bot

Date: 2015-08-21

  • 14:31 CDT @austin: updated Phabricator to latest HEAD

Date: 2015-08-19

  • 23:10 CDT @duncan: hackage back and appears to be operating normally
  • 22:56 CDT @duncan: restarting hackage now, with a migration so probably slightly longer downtime than normal...
  • 22:32 CDT @duncan: redeploying hackage server shortly (for the hackage-security beta test)

Date: 2015-08-18

  • 13:21 CDT @austin: updated Phabricator to latest HEAD, which un-prototype's the Ponder Q&A application and fixes Harbormaster builds

Date: 2015-08-06

  • 01:37 CDT @austin: updated phabricator to latest HEAD, and do an apt update as usual.

Date: 2015-07-27

Date: 2015-07-25

  • 16:36 CDT @austin: Updated Phabricator to latest HEAD; also, fixed H28 and H29 rules as they were broken due to an API change in Herald. Currently unused, so this is OK.
  • 12:23 CDT @austin: update to latest Phabricator. Now with more badges!

Date: 2015-07-24

  • 13:01 CDT @relrod: added hackage-mirror account on hackage, with same ssh keys as hackage account
  • 12:58 CDT @relrod: added hackage-origin dns record

Date: 2015-07-22

  • 09:54 CDT @duncan: given edsko access to the hackage user account on hackage.h.o to deploy during the hackage-security beta

Date: 2015-07-20

  • 12:35 CDT @austin: log reupload into new consolidated bucket completed; old buckets delted, aaand we're done.
  • 11:27 CDT @austin: Move Fastly away from ghc.h.o and CloudFlare back in front; seems like we broke authentication for some reason...
  • 08:55 CDT @austin: Roughly ~10% done reuploading the old hackage logs into the new bucket...
  • 01:49 CDT @austin: Transfer of old cvs/www archives to new bucket is complete. Still downloading old hackage logs for re-upload...

Date: 2015-07-19

  • 23:57 CDT @austin: Also, the old cvs/www Haskell archives have been moved to a new bucket and consolidated as well.
  • 23:56 CDT @austin: downloads.h.o Fastly logs have been synced to the new bucket successfully. Now, the hackage.h.o bucket is being moved - only 39,000 more log files to go
  • 21:45 CDT @austin: fix most of the macro thumbnails on /macro
  • 21:28 CDT @austin: tfw uploading 40,000 log files to the new DreamHost bucket
  • 14:27 CDT @austin: Aaaand is now also behind Fastly; the DNS should propogate soon.
  • 14:26 CDT @austin: Another new Fastly entry:
  • 14:21 CDT @austin: Fastly is now in front of; the DNS should propogate in a few minutes.
  • 14:20 CDT @austin: Old www.h.o fastly entry deleted; new wiki.h.o domain is active with a test URL:
  • 14:10 CDT @austin: {downloads,hackage}.h.o fastly logs have been moved to the new bucket. Re-upload of old logs will happen.
  • 02:09 CDT @hvr: hackage @ 8373851ee8421e40d2c4e279196dddde33919db0 (guess-work)
  • 01:08 CDT @austin: Point most of the new Fastly sites to use a new DreamHost bucket. Will switch downloads/hackage soon)
  • 00:20 CDT @austin: reconfiguring DreamHost bucket uploads to be more sensible

Date: 2015-07-18

  • 23:47 CDT @duncan: restarting hackage server, with https base uri (used in sitemap, rss, user emails and rel=canonical links)
  • 23:33 CDT @duncan: hackage now has a google sitemap.xml (thanks to our GSoC student) hopefully will improve google pointing to latest pages and docs

Date: 2015-07-17

  • 06:15 CDT @hvr: migrated /trac/ghc env to postgresql

Date: 2015-07-11

  • 22:14 CDT @austin: moved {blog,planet}.h.o to fastly
  • 01:44 CDT @austin: All existing Fastly CNAMEs have been moved to the new wildcard certificate.
  • 01:39 CDT @austin: Moving fastly CNAME records to - now with more SSL! Phabricator is already switched, with requests coming in - the global DNS should finish propogating soon.

Date: 2015-07-08

  • 17:25 CDT @gershomb: set haskell-cafe subscriptions to require approval for the time being, to deal with spam/nonsense posts

Date: 2015-07-06

  • 20:08 CDT @austin: added some new things in Passphrase, mucked around and renamed some projects, etc
  • 13:08 CDT @davean: Updated system packages on builder
  • 13:01 CDT @davean: Updated system packages on www
  • 12:45 CDT @davean: Restarted nginx with a config exposing dcoutts' hackage signing alpha test
  • 12:39 CDT @davean: updated system packages on
  • 01:04 CDT @austin: updated to HEAD with new redesign

Date: 2015-06-19

  • 11:34 CDT @austin: going down for an update to Phabricator HEAD

Date: 2015-06-15

  • 22:00 CDT @gershomb: added Cite and UserMerge extensions to our mediawiki install

Date: 2015-06-12

  • 00:21 CDT @gershomb: on a brighter note, the builders appear caught up! updating

Date: 2015-06-03

  • 02:06 CDT @austin: going down for an update to Phabricator HEAD.

Date: 2015-05-27

  • 19:16 CDT @austin: removed old deb.h.o A/AAAA DNS records
  • 10:28 CDT @austin: Added a tiny bit of swap to phabricator temporarily. RAM usage is at a consistent 90%; the InnoDB buffer cache size setting for MySQL may have been slightly overestimated.

Date: 2015-05-25

  • 22:04 CDT @austin: Make everything like 1000x or 1000000x faster by moving the MySQL server.
  • 21:36 CDT @austin: Soon moving the Phabricator DB to its own server. Dumping DBs now!
  • 16:27 CDT @austin: Switch's DNS records over to Fastly in an attempt to get some improved performance. More to come.
  • 13:03 CDT @austin: updated phab to latest HEAD, and added a new FQDN for access:

Date: 2015-05-21

  • 03:47 CDT @austin: finished restructuring the wiki, now with better URLs!
  • 03:39 CDT @austin: done
  • 03:36 CDT @austin: updating Phabricator to latest HEAD...

Date: 2015-05-14

Date: 2015-05-13

  • 08:54 CDT @austin: Updated phabricator to latest HEAD, and re-organized wiki structure: now with much more readable URLs! Required an update to libphutil-haskell

Date: 2015-05-07

  • 13:37 CDT @austin: updated Phabricator to latest HEAD

Date: 2015-04-13

  • 10:00 CDT @hvr: temporarily reject via 403 /trac/*/register requests on ghc.h.o

Date: 2015-04-12

  • 22:08 CDT @gershomb: added a special cloudflare page rule just for trac register in i'm under attack mode, hopefully that'll cool things off
  • 21:05 CDT @gershomb: enabled cloudflare browser integrity check on* to try to mitigate the bots hammering the ghc trac. time will tell.

Date: 2015-04-11

  • 11:10 CDT @hvr: enabled mod_qos w/ QS_LocRequestPerSecLimit /trac/ghc/register 1 && QS_LocRequestLimit /trac/ghc/register 1

Date: 2015-04-06

  • 23:17 CDT @davean: changed the redirect system for the old wiki links. They should now redirect to the pages they were supposed to
  • 21:36 CDT @davean: Added a redirect to preserve old /wikiupload URLs

Date: 2015-03-28

  • 18:34 CDT @austin: updating Phabricator to latest HEAD

Date: 2015-03-24

  • 13:07 CDT @austin: doing upgrades on phab/mysql01

Date: 2015-03-22

  • 09:34 CDT @hvr: some bot is currently bruteforcing /trac/ghc/register

Date: 2015-03-21

  • 14:31 CDT @hvr: fiddled around with ghc.h.o trying to repack ghc.git in the hopes to speed up Trac

Date: 2015-03-19

  • 15:08 CDT @austin: Try disabling Maniphest emails to curb incoming spam

Date: 2015-03-18

  • 18:54 CDT @austin: ./bin/remove destroy'd a bunch of now-trash maniphest tasks
  • 18:48 CDT @austin: after a lot of fiddling, inbound maniphest task creation now does properly work. Also, because these new emails are getting hit by spam, turn on Mailgun's inbound spam blocker
  • 07:00 CDT @austin: hoorah! Phabricator now properly adds view/edit policies to incoming email, and new emails work: {support,bugs,security}, and each assign custom edit/view policies with a new rPHUH extension
  • 03:16 CDT @austin: going down for upgrade to phabricator HEAD.

Date: 2015-03-13

  • 12:56 CDT @relrod: updated the ipv6 record for to something that makes sense
  • 12:56 CDT @relrod: changed /etc/hostname and ran hostname on the mail server.
  • 11:52 CDT @relrod: deployed latest hl
  • 11:27 CDT @relrod: Added www-origin record and updated www record to make use of it
  • 11:27 CDT @relrod: ran on phab mainly to get phaskell back here

Date: 2015-03-05

  • 10:18 CST @austin: phabricator going down for an update to HEAD, which may fix the issue jstolarek encountered w/ D202

Date: 2015-02-15

  • 14:50 CST @gershomb: redirected bare h.o dns records to point to old box. redirects from h.o -> www.h.o are in place
  • 14:20 CST @carter: can carter log stuff
  • 14:19 CST @gershomb: reminder that when we turn on a cdn to it fucks up dns reverse lookup and our mails all get flagged as spam!
  • 12:47 CST @relrod: deploying hl at 1ab05d6c809a61c7902ef77de428df5088a9d001
  • 12:17 CST @davean: Directions for adding subsite users is at
  • 12:07 CST @davean: Closed out the haskell-symposium issue
  • 12:07 CST @davean: added a DNS entry to act as the cononical location for actually getting to the actual server
  • 11:58 CST @davean: confirmed sftp access with haskell-symposium account
  • 11:57 CST @davean: had to change home directory to /home/web/ because sshd requires chroot directories, and every dir along the path to said, to be root owned, and only root writable.
  • 11:43 CST @davean: Creating a /etc/ssh/authorized_keys directory to to hold keys for sftp only users
  • 11:30 CST @davean: subsites sftp only group founded
  • 01:04 CST @austin: oh, and also, was compromised. :( rackspace is in touch and we're investigating
  • 01:03 CST @austin: {phabricator,ghc,mysql01,www,mail} are all patched and up to date
  • 00:52 CST @austin: upgrading phabricator.h.o to latest HEAD

Date: 2015-02-14

  • 23:46 CST @austin: Redeploy ruby server under a new Heroku account for, and make sure it stays alive using a consistent 'ping' application
  • 23:25 CST @relrod: deployed hl a bunch of times :P
  • 23:24 CST @austin: broke legacy page for Hugs @ for now
  • 22:53 CST @relrod: it is done. everyone freak out now.
  • 22:49 CST @austin: the new homepage is about to go live zomg
  • 22:19 CST @relrod: thoughtpolice fixed pipermail alias, I fixed /ghc so that .shtml is implied again and pulled latest hl code

Date: 2015-02-13

  • 11:34 CST @davean: Hand patch mediawiki because the ubuntu package is bugged

Date: 2015-02-09

  • 14:04 CST @austin: phabricator.h.o updated to latest HEAD

Date: 2015-02-06

  • 01:59 CST @relrod: updating everything on monitor

Date: 2015-02-04

  • 12:39 CST @davean: nginx 1.6.2 active now
  • 12:38 CST @davean: Packages all updated and stuff checks out
  • 12:29 CST @davean: Giving a go at package update on wiki.h.o
  • 12:22 CST @davean: Kept interoperability with all previous clients, support FS with all clients supporting said. SSL Labs now an "A"

Date: 2015-02-03

  • 15:42 CST @davean: Your right, that was a lie - I only updated www/
  • 15:42 CST @davean: We now send the intermediary cert for TLS
  • 14:33 CST @austin: upgrade complete.
  • 14:32 CST @austin: updating Phabricator to latest HEAD

Date: 2015-02-02

  • 12:42 CST @gershomb: cancelled account for Rock at hetzner yesterday. it will go away in 3 weeks

Date: 2015-01-26

  • 02:01 CST @davean: is now smarthosted through and wiki password reset email success has been confirmed

Date: 2015-01-20

  • 13:16 CST @gershomb: created server at ip -- waiting on ndm to set up credentials

Date: 2015-01-19

  • 16:41 CST @austin: restarted Phabricator to get IRC rejoin, and fix a seeming race condition in commit message parsing(?) updated Phabricator to latest HEAD as well

Date: 2015-01-08

  • 01:15 CST @austin: added new domain names: frances.ghc.h.o (new sweet POWER8 machine), lovelace.ghc.h.o & hopper.ghc.h.o (dual ARMv7 quad core cloud instances), and frege.ghc.h.o (OSUOSL POWER7 machine). todo: add entries for other GHC buildbots (like phab-07)

Date: 2015-01-07

  • 01:25 CST @austin: actually perform upgrade, because i'm a dummy and forgot to update our Phab mirrors

Date: 2015-01-06

  • 22:49 CST @austin: upgrade complete.
  • 22:48 CST @austin: is going down for an upgrade to HEAD.

Date: 2015-01-04

  • 14:55 CST @duncan: Expect a couple minutes of hackage downtime soon as we update to the latest code

Date: 2014-12-22

  • 19:29 CST @davean: Builder is approximately caught up

Date: 2014-12-20

  • 21:36 CST @davean: New standard builder (one modification), running on several Cabal versions (1.20 for build, 1.22 git branch for cabal-install) seems to checkout on building several packages and is now running catchup
  • 14:32 CST @austin: upgraded Phab and mysql01.h.o to mariadb 10.0.15 - a transient bug (fixed in MariaDB 10.0.14) caused the gtk2hs wordpress page to break

Date: 2014-12-17

  • 11:59 CST @austin: install ntpd on hackage.h.o; there was clock drift of about 40 seconds causing bizarro errors - manually ran ntpdate to update. this should no longer be an issue as it will start on boot.

Date: 2014-12-15

  • 23:03 CST @relrod: bounced apache on www. load 138/141/140
  • 17:29 CST @davean: Ban bots from /hoogle/ temporarily due to load issues because hoogle is served via cgi
  • 15:09 CST @davean: Hackage builder disabled untill I figure out the source issue
  • 02:14 CST @relrod: added 3GB of swap to www in hopes that it prevents it from crashing and burning again
  • 02:08 CST @relrod: bounced apache on www, load was at 23/21/17, also no free RAM

Date: 2014-12-14

  • 15:26 CST @davean: Significantly bump fastly timeouts
  • 13:49 CST @davean: Fastly is infront of www.h.o again, though briefly it passed through CF due to update non-atomicity, DNS is propigating
  • 13:46 CST @davean: DNS changed out
  • 13:45 CST @davean: About to update DNS, sadly I need to delete the A record before adding the CNAME

Date: 2014-12-13

  • 00:00 CST @sclv_: looks like really is blacklisted on some spam lists. following up on them now

Date: 2014-12-11

  • 17:07 CST @davean: New build bot running with git version of Cabal/cabal-install, has uploaded reports without getting a 404

Date: 2014-12-09

Date: 2014-12-04

  • 13:18 CST @austin: done, after fighting phd daemons a bit (they got into a bizarro state)
  • 13:13 CST @austin: upgrading phabricator to latest HEAD; restarting daemons for config change
  • 12:39 CST @davean: hackage now is a CNAME is an alias of
  • 12:37 CST @davean: hackage was A record points to

Date: 2014-12-03

  • 23:24 CST @duncan: Hackage update completed successfully (1:58 downtime)
  • 23:04 CST @duncan: Expect a short period of hackage downtime soon as we update to the latest code

Date: 2014-11-29

  • 10:55 CST @austin: change that to 500/20 instead...
  • 10:49 CST @austin: updated phabricator.h.o to reflect the same spiped changes
  • 10:45 CST @austin: updated mysql01.h.o to have spiped daemon up the connection limit to 256 max conns w/ a 15s timeout. the default 100 was probably too low.

Date: 2014-11-26

  • 10:21 CST @austin: (hopefully) fixed mail relay for hackage.h.o and updated the Postfix config on www.h.o with the help of @duncan
  • 10:15 CST @austin: updated some old CloudFlare DNS settings with incorrect A records; added AAAA records for www and

Date: 2014-11-25

  • 18:23 CST @austin: note: last night, modified Fastly configs to upload logs into DreamHost/S3 for the wiki and downloads.h.o

Date: 2014-11-21

  • 17:59 CST @austin: got rid of kibana and es01 for now, until we figure out exactly how we want to manage them
  • 04:40 CST @davean: For the record, version 6 of the VCL for www.h.o is pre-cookie-cutter edits
  • 04:33 CST @davean: Strip cookies if they're empty, after ignoring GA cookies
  • 03:20 CST @davean: Change apache2 log rotate to daily from weekly on www.h.o
  • 03:15 CST @davean: Enabled cache control headers for mediawiki. PURGE proxy is running under a tmux as the root user during initial testing.
  • 02:46 CST @davean: Updating PHP on www.h.o to new security release
  • 02:44 CST @davean: Adding the debian haskell repo to www.h.o because seriously.

Date: 2014-11-20

  • 21:16 CST @relrod: undid the @ change because everything hates me
  • 21:10 CST @relrod: dns changes: MX points to mail, mail points to the origin IP, and @ points to the same IP as www
  • 13:57 CST @davean: Now on fastly for www - next up purging and cache headers.

Date: 2014-11-19

  • 20:08 CST @austin: www.h.o now redirects all GHC documentation to downloads.h.o, yay!
  • 19:39 CST @austin: the upload of the www tarball to DreamHost (all 23GB) is complete.
  • 19:11 CST @relrod: try.h.o available over https now
  • 19:10 CST @austin: upload of rock.h.o snapshot of www.h.o to dreamhost about 65% complete
  • 18:41 CST @relrod: doing an hl deploy so the tryhaskell REPL works again
  • 18:41 CST @relrod: removed ssl -> nonssl redirect hack
  • 18:41 CST @relrod: new ssl certs in place on new-www
  • 18:34 CST @relrod: applying updates on new-www
  • 18:01 CST @austin: beginning sync of old www archive from rock to DreamHost for archival
  • 16:47 CST @austin: old cvs.h.o tarball archive uploaded into a bucket, will scavange on rock.h.o soon
  • 16:03 CST @austin: enable CloudFlare for try.h.o and new-www.h.o, and added AAAA record for try
  • 15:46 CST @austin: uploading the old cvs.h.o snapshot to Dreamhost archival buckets
  • 15:11 CST @austin: upgrade complete
  • 15:08 CST @austin: performing phabricator upgrade to latest HEAD
  • 13:24 CST @austin: all symlinks in place, I think downloads.h.o is ready for redirects from www.h.o
  • 13:14 CST @austin: all docs should now be in place on downloads.h.o
  • 13:02 CST @davean: Added PTR record for www.h.o server as on both IPv4 and IPv6. Hopefully this is the primary mail delivery issue
  • 12:54 CST @austin: GHC documentation sync to has been successful; most docs should be moved into place
  • 12:29 CST @austin: rsyncing the GHC documentation to downloads.h.o
  • 12:00 CST @austin: restarted apache2 on new www server, moved redirects for www.h.o downloads to use SSL (since we use HTTPS all the time anyway)
  • 11:04 CST @davean: added memcached to www.h.o and enabled it for mediawiki to slighly hide the hiddious mysql performance
  • 08:21 CST @austin: OK, the new www server has spiped managing a connection to mysql01.h.o, although there is no init script for it yet

Date: 2014-11-18

  • 23:16 CST @davean: apache2 running again. All checks out other then mediawiki, which needs spiped. Redirects for platform and ghc downloads to downloads.h.o now in place. Seems mail is coming through haskell-cafe so initial indications are that is working
  • 15:25 CST @austin: www.h.o is in surgery now, with data migrating as we chat.
  • 13:52 CST @davean: currently rsyncing /home from hetzner to rackspace at ~6MBps
  • 13:43 CST @austin: we're already beginning. apache2 is down, I'll pull down mailman soon
  • 13:41 CST @davean: rock has lost a disk in its RAID1, the other disk is acting unhealthy. We're emergency migrating www off rock.
  • 01:25 CST @relrod: ok virsh destroy finally worked, www should be booting back up

Date: 2014-11-17

  • 09:07 CST @austin: www.h.o is back up - for now
  • 09:01 CST @austin: rock with a load of 26.25, 14.63, 9.16
  • 09:00 CST @austin: www.h.o is going down to cool off for a minute and blow off some steam; load average: 102.49, 78.45, 44.72

Date: 2014-11-16

  • 22:06 CST @relrod: www was going insanely slow again and had ~60 load average. Just bounced apache.

Date: 2014-11-14

  • 23:19 CST @gershomb: disk wait usage was stupid high and grinding www.h.o to a halt. killed apache and then started after 10 secs, and things seem better now?

Date: 2014-11-13

  • 16:10 CST @bitemyapp: restart hackage-server with a nofile limit of 32000 in upstart config. (Hopefully this is honored)
  • 13:58 CST @hvr: kicked Apache on ghc.h.o (had high load again, causing Trac to hang)
  • 11:33 CST @austin: shut down the old hackage VM on Rock for the last time. good night sweet prince
  • 11:29 CST @austin: permissions for downloads.h.o platform binaries restored (forgot a flag to cp -R)
  • 11:00 CST @davean: Changed CNAME of in cloudflare to point to
  • 11:00 CST @austin: platform downloads are now in place on downloads.h.o

Date: 2014-11-12

  • 12:24 CST @austin: done. the system load was quite high but I had no time to diagnose the cause.
  • 12:21 CST @austin: www.h.o is being rebooted
  • 10:18 CST @hvr: restarting Apache2 on ghc.h.o in the hope to unbreak Trac

Date: 2014-11-10

  • 16:18 CST @austin: installed postfix on hackage.h.o to fix registration and password reset issues

Date: 2014-11-09

  • 12:02 CST @austin: mirroring platform downloads to downloads.h.o now as well /cc T45
  • 10:51 CST @austin: proper cache-control headers set for downloads.h.o
  • 10:12 CST @austin: downloads.h.o/~ghc/ is mostly in place; I'm wget'ing the site to populate the cache a bit and test it a bit /cc T45
  • 10:03 CST @austin: done
  • 10:03 CST @austin: actually upgrading this time
  • 10:02 CST @austin: whoops, forgot to update our git mirrors first...
  • 10:01 CST @austin: going down for an update right quick

Date: 2014-11-07

  • 14:22 CST @austin: the initial sync for ghc downloads is done /cc T45
  • 12:42 CST @austin: about ~80% complete /cc T24
  • 12:23 CST @austin: initial sync of GHC tarballs to the new downloads server is ~50% complete
  • 11:54 CST @austin: done
  • 11:53 CST @austin: doing another phab upgrade for a few bugfixes
  • 10:58 CST @austin: downloads.h.o bind mounts, fstab etc fully configured and ready to go, still need SFTP jails /cc T45
  • 10:49 CST @austin: soon going to start mirroring old GHC downloads to downloads.h.o
  • 08:28 CST @austin: Schema adjustments complete. Emojis may work now! 🍕
  • 08:16 CST @austin: Phabricator going down for storage adjustments in ~5min
  • 08:12 CST @austin: upgrade done
  • 08:11 CST @austin: upgrading phabricator

Date: 2014-11-03

  • 09:56 CST @hvr: manual GC via /etc/init.d/apache2 restart @ ghc.h.o

Date: 2014-11-01

  • 20:48 CDT @relrod: rebooting the (currently-unused) box that will become try.h.o for a kernel update, before I start playing with ansible on it

Date: 2014-10-31

  • 15:44 CDT @austin: the apocalypse was avoided like an hour ago

Date: 2014-10-30

  • 13:40 CDT @austin: changed www to use 301 redirects to 302 redirects for HTTPS
  • 00:27 CDT @austin: Rackspace is having some underlying hardware issues; planet.h.o may have interrupted availablity

Date: 2014-10-29

  • 22:45 CDT @gershomb: forced https redirect and enabled hsts for

Date: 2014-10-26

  • 05:04 CDT @austin: currently creating new ghc builder (phab-ghc07) for harbormaster
  • 02:12 CDT @austin: ghc and new-www upgraded as well
  • 02:11 CDT @austin: hackage/darcs/deb also upgraded
  • 02:09 CDT @austin: planet/www/phabricator/mysql01 upgraded
  • 02:07 CDT @austin: upgrading dd-agent on current infrastructure
  • 01:25 CDT @austin: Added a page rule to CloudFlare so always goes over HTTPS
  • 00:58 CDT @austin: updated rPHUH to allow empty GHC Trac Issues: fields for diffs

Date: 2014-10-25

  • 17:52 CDT @austin: Disabled a few WAF rules ('Specials') in CloudFlare to reduce a shitton of false positive noise, causing arc to fail for some users.
  • 05:08 CDT @austin: upgraded all services to spiped 1.4.2, fixed T50
  • 04:13 CDT @austin: nuked a bunch of old DNS entries; fixed T47
  • 03:39 CDT @austin: Anon/registered user email works via bugs@phab.h.o, but no bugs@h.o. still investigating
  • 03:35 CDT @austin: new bugs@h.o and security@h.o emails up, pointed to phab. postfix restarted on www.h.o
  • 03:33 CDT @austin: Phab restarted
  • 03:32 CDT @austin: All new Maniphest tasks are now private by default, to be triaged/have their policy edited after the fact
  • 02:52 CDT @austin: is now proxied by CloudFlare, too /cc @hvr
  • 02:52 CDT @austin: is now proxied by CloudFlare

Date: 2014-10-24

  • 17:26 CDT @austin: CloudFlare Pro is fully active; including non-SNI based certificates
  • 11:39 CDT @austin: btw, confirmed as fixed
  • 11:37 CDT @austin: done; new record should pop up pretty damn fast.
  • 11:36 CDT @austin: Moving record from CNAME -> IP
  • 11:26 CDT @austin: update complete
  • 11:25 CDT @austin: updating Phab to fix blog.h.o

Date: 2014-10-23

  • 22:49 CDT @austin: Turned on CloudFlare WAF to make things break extra quickly
  • 22:44 CDT @austin: Turned on CloudFlare for as a test
  • 21:37 CDT @austin: triaged and did a lot of cleanup in Maniphest
  • 19:58 CDT @austin: Upgrade complete
  • 19:57 CDT @austin: Upgrading phabricator
  • 19:56 CDT @austin: Done (was already enabled for blog and whatnot)
  • 19:55 CDT @austin: Enabling CloudFlare for phabricator.h.o

Date: 2014-10-21

  • 21:35 CDT @gershomb: now redirects to w00t!!!

Date: 2014-10-19

  • 15:22 CDT @austin: syncing new hackage server for final move with @duncan

Date: 2014-10-17

  • 18:54 CDT @austin: updated the infra wiki with some boilerplate

Date: 2014-10-15

  • 11:11 CDT @hvr: added rDPH repo to Phabricator w/ default settings
  • 00:34 CDT @relrod: deployed new wildcard on phabricator
  • 00:34 CDT @relrod: updated phabricator
  • 00:17 CDT @relrod: and hackage
  • 00:06 CDT @relrod: deployed new wildcard on ghc
  • 00:00 CDT @relrod: deployed new wildcard on new-www and www

Date: 2014-10-14

  • 21:22 CDT @relrod: and ghc
  • 21:20 CDT @relrod: Just confirmed www.h.o and h.o don't have SSLv3 enabled
  • 20:48 CDT @austin: phame theme updated to link to atom URL.
  • 20:42 CDT @austin: moved blog.h.o to use a custom Phabricator theme; available in rBLOGTHEME (and on github). fixed to update theme as well.
  • 17:05 CDT @austin: actually fixed; i was dumb and forgot an -R to chmod
  • 16:58 CDT @austin: attempt to fix permissions again re: T24 :(

Date: 2014-10-12

  • 16:36 CDT @austin: attempt to fix permissions for /var/lib/phabricator again.
  • 15:39 CDT @austin: i'm showing @davean how to use the admin log.
  • 14:46 CDT @austin: OK, perms should actually be fixed now.
  • 14:43 CDT @austin: fixed all lingering file permissions hopefully (re: T24). /cc @hvr
  • 03:46 CDT @austin: fixed T24
  • 03:42 CDT @austin: done
  • 03:40 CDT @austin: Phabricator update time

Date: 2014-10-09

  • 11:16 CDT @relrod: arc probably doesn't support SNI (based on complaint from dfeuer). Disabling cloudflare on phabricator for now until I can debug it in a few hours.

Date: 2014-10-08

  • 19:49 CDT @austin: also enabled for Phabricator now as well
  • 19:48 CDT @austin: enabled CloudFlare on a few more domains: monitor.h.o, and downloads.h.o /cc @relrod

Date: 2014-10-05

  • 19:37 CDT @relrod: installed/started fail2ban on monitor
  • 19:33 CDT @relrod: it's back
  • 19:31 CDT @relrod: monitor is down, no idea why
  • 00:54 CDT @austin: looks like CloudFlare finally issued our certificate. kibana is working properly.

Date: 2014-10-03

  • 22:05 CDT @austin: Phab upgrade completed
  • 22:05 CDT @austin: upgrading Phab to the latest HEAD
  • 17:09 CDT @austin: upgrade done on mysql01
  • 17:09 CDT @austin: upgrade done on phabricator
  • 17:09 CDT @austin: doing apt-get upgrade on mysql01.h.o
  • 17:08 CDT @austin: doing apt-get upgrade on phabricator.h.o
  • 17:05 CDT @austin: CDN/CF switch pending; still no certificates yet from CF

Date: 2014-10-02

  • 13:25 CDT @austin: still no cloudflare certs for kibana :(

Date: 2014-10-01

  • 19:14 CDT @austin: deploy finished
  • 19:13 CDT @austin: updating Phab now
  • 17:32 CDT @austin: Phab update soon; ETA ~2hrs after latest GHC builds.
  • 17:32 CDT @relrod: rebooting monitor
  • 17:30 CDT @relrod: applying updates on monitor
  • 17:29 CDT @relrod: triggering an hl deploy
  • 15:10 CDT @austin: re-enabled CF again for Kibana, but still no SSL cert. commence twitter whining to get to front of queue.
  • 15:08 CDT @austin: phab upgrade soon; waiting for B1088 to finish first

Date: 2014-09-30

  • 15:53 CDT @austin: YOU WILL ADD A TIMESTAMP TO THIS MESSAGE (i hope)
  • 00:00 UTC @austin: CF disabled for kibana for now.
  • 00:00 UTC @austin: cloudflare SSL certs delayed ~48hrs. we'll get an email when it's setup for our DNS records.
  • 00:00 UTC @austin: attempted CloudFlare deploy on Kibana. busted. SSL certificates with proper subjectAltName not yet deployed it seems
  • 00:00 UTC @austin: upgraded Phab. new highlights: emoji support might work.

Date: 2014-09-28

  • 00:00 UTC @relrod: hl deploy (with D258)
  • 00:00 UTC @relrod: also replaced ghc-7.8.2 with ghc-7.8.3 on new-www
  • 00:00 UTC @relrod: applied updates on new-www
  • 00:00 UTC @austin: Rackspace-wide reboots are starting now, due to Advisory XSA-108. And remember, the cloud is a secure place to keep our data forever.

Date: 2014-09-27

Last Author
edsko, austin, relrod and 7 others