Wednesday, October 16, 2013

1100 stage alerts (one of my favorite bugs)

This is a post I have had ready for about 5 years now. A long time ago, I caused a partial meltdown of our software. This was the email thread that went out by QA, and the reply from Dev. FYI: Stage Alerts are just emails sent to everyone saying something bad has just happened.

Note: this is the first time I have looked at it in about 3 years and its still pretty funny... well to me. Maybe you had to be there.

From: QA Group
To: Dev
Sent: 1:40pm
Subject: 1100+ Stage Alerts in 3 hours

I've given up trying to keep track of the Stage Alerts. Since noon today we've had over 1,100.

I haven't checked, but I'm pretty sure we (in QA) haven't reported bugs for every alert that has come through.

Just thought I'd send out a quick note to check and see if anyone in Dev has an eye on these alerts too.
From: Developer
To: QA Group
Subject: RE: 1100+ Stage Alerts in 3 hours?

There alerts are caused by problems try to persist audit events to the database. I've discovered a couple bugs in our logging implementation, that were exposed by some unique data.

Specifically, there was a borrower user in the database with the following:

First Name: Jebediah Obadiah Zachariah Jedediah Springfield-KD
Last Name: Squeaky Voiced Teen né Jeremy Peterson/Freedman-KD

This extra long name ends up in an audit message, and eventually causes problems because it exceeds a length limit. I've updated the user to shorten the name so that no more alerts will be generated.

I'll attempt to cleanup any existing audits that are failing/retrying in the bowels of BizTalk.