In reviewing this outage, there were clear signs that pointed to a predictable issue or failure of the service. Both the level of dropped packets and error counts from our aggregate switch escalated on 12/13.
We have adjusted sensors in our network monitoring software to look for this activity and report it before it impacts client experience.