An Update on Hymnary Downtime

As many of you know, we had some issues with Hymnary this past weekend. The problems began on Saturday morning around 9:30 am EST with a hardware failure. Our on-call team began investigating right away and determined the issue was the storage server. From approx. 11 am to 8 pm on Saturday, we worked to get the storage server to work without losing any data but could not. We then had to do a factory-reset, and at midnight on Saturday, two of our team members began to work through the night to rebuild servers. Hymnary was mostly functional again by 3 am and thus able to be used by people on Sunday morning (our busiest time as of late) while other impacted websites (CCEL.org, preachingandworship.org and more) were back online by 6 am. Then, almost all day yesterday, members of our team continued restoring missing servers and services, got backups copied back and performed other important tasks. We are mostly functional again, and the good news (and blessing!) is that we lost very little production data thanks to our backup system and policies.