/ Engineering

Radioca.st Outage 28/02/2021 - Analysis

tl;dr
All HTTPS radioca.st URL's were unavailable due to an issue with an SSL certificate.

Situation
Centova Cast does not support native SSL streaming, unlike modern control panels such as AzuraCast & MediaCP. We developed our own resolution to this issue by creating a Port 80 SSL proxy for all streaming clients including resellers. This is critical as Google Chrome and other browsers now block insecure content across websites, meaning listeners are not able to listen if the stream is from a non-https source.

What happened?
11:53 UTC: Initial reports stated that HTTPS secure URL's were no longer functional.
11:55 UTC: Internal monitoring was alerted & troubleshooting began.
12:00 UTC: It was noted that the certificate had expired and was ready for renewal, this is a standard procedure.
16:00 UTC: As troubleshooting continued, we issued emergency servers using MediaCP to critical streaming clients, free-of-charge, alongside a help guide to setup relaying.
12:16 UTC: Our internal server management system suffered a major issue, meaning we were not able to deploy the new certificate to our Centova Cast servers.

Around 23:50 UTC - all radioca.st URL's were accessible.

Some clients faced issues with the cache across various sites causing further listener problems. This self-resolved after a few hours.

What was the impact?
Our radioca.st infrastructure is a critical part of our service and many broadcasters rely on this. This being said, it is a very complex & advanced system that was developed some time back.

What will happen & what have we learnt?
Work is ongoing to improve stability of our Ansible deployment system, to ensure critical updates can be made across all servers quickly & easily. We are also working to redesign our radioca.st URL system and create a management interface that will allow clients to manage their own domains.

Radioca.st Outage 28/02/2021 - Analysis
Share this