2017.05.22 05:00 | Kurt Tomlinson
Bloopist suffered a ton of downtime over the past couple months. Downtime was around 1 or 2 full days per month due to server instability. The issue was very difficult to track down, but it seems to have been related to how Bloopist was acquiring TLS certificates for each blog that it hosts.
Bloopist provides a custom subdomain for every blog and custom domains for paying blogs. These domains and subdomains are protected by individual TLS certificates provided by Let's Encrypt. Due to the great number of blogs on Bloopist, it was necessary to automate the task of retrieving certificates from Let's Encrypt. For this, the OpenResty plugin lua-resty-auto-ssl was used. Bloopist was running into rate limits with Let's Encrypt from the start. This wasn't a problem initially as lua-resty-auto-ssl handled those errors correctly. The Bloopist servers tracked which blogs had been issued TLS certificates at which domains and automatically served blogs over http or https as appropriate.
However, Bloopist appeared to hit a second rate limit with Let's Encrypt that the lua-resty-auto-ssl plugin didn't handle as well. This resulted in the lua-resty-auto-ssl plugin causing the OpenResty to stop serving requests to the Bloopist application servers. The exact problem that occurred has not been determined yet, so in the mean time an interim solution has been implemented: only selected blogs will be served over https. The remaining blogs will be served over insecurely over http until the issue with lua-resty-auto-ssl is resolved.