The links to panic case studies that I included in my previous post are fascinating, and I encourage you to read them, even if they are long. The two from “asktog.com” were written by Bruce “Tog” Tognazzini, a guy who specializes in human-computer interaction. The cell-phone-sunrise one illustrates the very affecting level of panic that is often induced in the users of electronic contraptions. If this sort of thing can happen to people who are extraordinarily tech-savvy, imagine how your average end-user feels?

I’m not an average end user. I did top-tier tech support back in the Windows dial-up days when “work-flows” didn’t exist – I sat under my call-center desk with my eyes closed and my head firmly jammed into a corner to keep it from exploding, and I could tell what was wrong with a modem by the sound of it. I’ve worked in a NOC, troubleshooting big circuits and token-ring networks. I physically built part of the Internet (granted, a really small part). I work on the Internet, I socialize on the Internet, I watch TV on the Net. My laptop is never out of reach – I used to sleep with the wretched thing when I was on 24/7 call for a very unstable network. I’ve provided hands-on and remote/phone Windows tech support to my family and friends for an unthinkable array of problems. I’ve built my own computers and set up my own LAN and VoIP phone. I think one can safely say I’m generally quite comfortable with computers and Internet technology. I am relating these details to set up what follows…

Check out what happened few months ago one workday evening when I opened my laptop – the one I’m typing on right now, that I’ve had for a long time – and the Internets failed to work. My first reaction was “Eh, my cable barfed, it happens…” so I went to reboot the cable modem. It came back up fine. I rebooted the WAP and the laptop just in case, and tried again. Nothing. “Hm,” says I, “I wonder if something upstream from me is broken?” So I dug out the other laptop and fired it up. It connected just fine. “Uh-oh, ” said my inner voice “This Is Bad! BAD!!”

…and that is when the panic set in. I distinctly remember the feeling of an icy wash sweeping over my body, followed by a hot flash. I was sweating, my breathing was short, and my hands had a fine tremor. Now, this was patently absurd. I had another, functional laptop! But that wasn’t relevant to me right then: THIS one didn’t work!! I opened up a command window, used ipconfig to flush everything. Nothing – and no new IP assigned, either. Oh, what was happening?! I even used my other laptop to download the latest wireless card driver for this laptop, stuck it on a thumb drive, and updated the driver. Nothing. It obviously was my laptop at fault and not my network…but guess what I did next?

(to be continued…)

As I go through my ticket queue I am often confronted with some pretty wacky stuff. When people lose their grips, the results can range from funny to disastrous. One of the tickets I worked today reminded me of this remarkable story from a few years back:

The City Manager of Tuttle, Oklahoma loses the plot. (you only need to read the first few to get the idea, I promise)

This is a perfect case-study of how bluster and threats can backfire in the most unexpected ways. Mr. Taylor, I am certain, never expected to become the laughingstock of the Internet overnight. The site got 516,147 hits total, and I remember that it got most of them within 48 hours of the original post. It spread like wildfire, and the howls of derision rang from one end of the world to the other. Emails were sent to his superiors, to his City Hall, to anywhere people could think of to spread the word to the town’s inhabitants and the world that Mr. Taylor had made a serious tactical blunder. He himself was sent so much email that their mail server fell over, and his email address was removed from the city website, as was his photo shortly thereafter. The Tuttle City webserver made a small sad sound, and died under the load. The tale made it to several large online newspapers, including The Register, in the UK. In a town with less than 5000 inhabitants, it must have been a very uncomfortable time for the guy.

I feel a certain sympathy for him. His initial reaction was a classic panic response, which happens to everyone sooner or later. Where he made his mistake was getting into a land war in Asia…^H^H oh, wait – was in rapidly escalating the situation to threats of FBI involvement, instead of actually reading the emails being sent to him by the CentOS developer that detailed how to correct his issue. My sympathy largely evaporates right about there.

Panic is a natural human response to stress. Getting out of a panic-inducing situation generally involves taking a step back and using rational thought processes, even if you only have a split second to do it – say, your parachute fails to deploy at 3000 feet. Uh-oh! Now that’s cause for panic! Mr. Taylor wasn’t falling out of the sky at 120MPH, though. He was safely on the ground, sitting in front of a computer and freaking out about his town’s website. He had the time to stop and think, but he didn’t use it, and the end result was world-wide mockery that is archived for the public, in all its painful glory, to this day (and probably until the end of time).

I get this sort of thing a lot. People don’t pause to read and think, or do a quick online search, or follow links provided in the bounce they got – they make assumptions, draw faulty conclusions, lose their tempers and come in swinging with blood in their eyes – or worse, hysterically weeping. One of my particular skills is talking this kind of person down out of the trees. I am almost always successful, but it’s hard work, and I do wish people wouldn’t do it to themselves – or me. Often when the panic is over, they wind up thanking me for trying to help them out, and if applicable, I explain how to avoid $FOO next time, then hang up hoping that they will indeed stop and think the next time.

Here’s a handy (but non-exhaustive) guide to avoiding email-related freak-outs*:

• * Panic first. Think, second. Act, third or even fourth.
• * Follow links in error messages, and read what they have to tell you. If you don’t understand what they say…
• * Web-search is your friend.
• * Back up important emails. Better yet, print them. Your inbox is not a bulletproof storage place.
• * For really crucial contacts, have a phone number, or at least a secondary email address.
• * Always double-check the “to” before you hit “send” on an email.
• * Don’t put stuff in an email you would cringe to see published in the Washington Post.
• * ISPs do not have magic powers or time machines. If you deleted an email in 2006 and it’s now critically important to a legal case, you are out of luck. That email is gone, never to return.
• * Don’t ever believe an email that’s offering you money, or telling you “confirm your account details or bad things will happen”. If it sounds too good to be true, it is. No modern institution with a lick of sense will send you such a confirmation email either. If you do business with one that is stupid enough to send an account-details-or-else email of the sort that phishers love to use, fire them.
• * Always check the file extension of an attachment. If it’s a .exe, proceed with extreme caution. Like, don’t open it, pick up the phone and call that number I told you to have for your contacts, and ask them if they meant to send you an executable file.
• * If you do fall for an email with an .exe in it, DONT REBOOT. Back away from the machine and get help.
• * Abusing the person who is trying to help you is rarely going to have a good outcome for you.
• * Did I already mention backing up important emails and contacts? Yeah? Well, it’s worth repeating. Back-ups will save your hide.

* Yes, of course! Some of these lessons were indeed learned by painful first hand experience.

From the AOL Postmaster blog:

The Postmaster team has suffered significant staff reductions, and any tickets opened will see slower processing times. Thank you for your understanding and patience.

—-

And from me, thank you all so much for your kind words, job leads, and condolences. They were all passed along to my team.

*IPs
*Error messages or detailed description of symptoms
*Headers and bounce if you have it..and if you don’t:
*Sender & Recipient/Date/time/time zone for an example of problem/missing mail
*When the problem began.
*What you have done to solve the problem.
*What kind of mail the IP is sending – or should be sending!
*Do your troubleshooting. Don’t make me ask you to do the basics, or have to do it for you.
*If you can’t get mail *from* AOL, send a test from an AOL account and include the results, and give it enough time to time out and send you a bounce.
*If you’re an Exchange admin, don’t expect me to make sense of your insanely configured MTA by sending me screen shots of the admin console.

This is not pointed at ESPs in particular. In fact, what provoked this was a corporate IP. It is a huge waste of everyone’s time when I’m asked to lift a block on an IP, I look at the complaints and see that clearly it was compromised…and have to send an email asking if it was fixed. 90% of the time, it was fixed before the ticket was opened. Say so the first time! The less guesswork I have to do, the faster I can solve a problem and the faster I can move on to the next one.

Most of the US Postmaster team just got laid off, including Christine. I am still employed, but of the non-programmers, I’m all that’s left in the US. *

This is a totally devastating blow to everyone. If people reading this have job leads for any of my folks…ex-folks…please drop me a line, or post in the comments and I’ll pass it on.

Jobs needed: Sysadmin, spam fighting/abuse, postmaster-y stuff, deliverability, programming, mail gateways/MTA, and of course management. We lost a lot of people outside the immediate Postmaster team also: anti-spam programming, mail/MTA, database, etc.

These people are the best of the best…
…and I am going to miss them more than I can even begin to express. Eight years is a long time.

*yes, Madkins is safe.

The more I consider the common uses of this obfuscation service, the more it confounds me. I cannot see any reason for a legal business entity to use it. If Joe User sets up a domain, keeps a personal webpage on it and wants to keep his registration information private, more power to him. I’d certainly use it in that situation since I don’t particularly enjoy stalkers. But, a business? Really? Why?

Here’s a perfect example of the lack of awesome: non-technical family member A is attempting to exchange email with definitely-not-technical family member B who is in something of a sticky situation. The server that relays these crucial emails is timing out connections, inbound and outbound. I wanted to help, so I figured I’d look up the technical contact for the domain and give them a nudge. I tried it.

Yep, Dear Reader, you guessed it. Their contact information is hidden. The domain’s website provides no avenue for support either, although it is a government site. So this family – and how many more? – remains fractured, frustrated and angry, their email continues to be deferred, and no-one can do anything about it because the admins who could fix it remain in blissful ignorance of the problem.

Genius!

Tell me again why a non-criminal business needs to hide who they are and how to reach them?

In a previous post, I mentioned a nifty service that will scan any file you upload to it with multiple AV programs. It is both useful and socially conscious, since any problems found are reported to all the participating anti-virus makers so that they can continue to improve their products to the benefit of everyone.

Naturally, if there is light there must be darkness: a couple weeks later I found several articles including one by Brian Krebs, that refer to a new breed of such scanners which are based on the premise that they will NOT share their findings with the AV makers, and make no pretense of being created to serve anyone but malware authors. For $1 per file, they can test their nasty little products against the big guys and no-one will be the wiser. And of course, the currency used is virtual, so there’s no trail to follow there, either.

When I was a kid and I was reading cyberpunk books, I thought it was all amazingly cool. Now that I’m living in the future, I’m finding it deeply alarming. The more I learn about the shadow economy, the more it scares me.

I fondly imagine that the people who are reading this blog are savvy enough to have their computers locked down tight, know how to spot phish, and are generally security conscious. Please, share the knowledge. Go to your parents and sibs and friends and help them learn. The bad guys are ahead of the game, so why not make it a little harder for them?

I hear a lot of “Hey, ISPs! Tell us exactly what to do, and we will do it!” from ESPs and marketers. Then last week I read this excellent post by Jamie Tomasello over at Cloudmark, and it got me to mulling over the whole thing.

So, okay. Ask and ye shall receive. I think I can probably safely speak for every major ISP in business with the following list. In no particular order, we wish you would do the following things:

– Do a stellar job of adhering to best practices, keeping complaints low and engagement high.
– Vet prospective clients carefully. Consider the consequences to your reputation if you assume them as customers.
– Don’t obfuscate your identity, or allow your clients to do so.
– Keep a close eye on the quality of the lists they send to, how often they send, and what the response is to the mailing. Make adjustments accordingly.
– Shut a customer that is causing a problem down immediately, no matter who they are, and fix the issue before resuming the send, unless there is a legal reason why this should not happen (and such instances will take place once in a blue moon on a leap year on the planet Pluto).
– Send less mail, and content of higher quality, to people that want it and are expecting to get it, and ONLY to those people.
– Set recipient expectations clearly at the outset, and don’t change the rules mid-game.
– Never send to a suppression list by accident. Render such an accident impossible.
– Don’t buy mailing lists.
– Don’t let your clients use purchased mailing lists. Learn how to spot one.
– Co-reg is nearly impossible to do right. So is e-pending. Consider the implications.
– Don’t listwash, or waterfall. Have high quality lists to begin with.
– Don’t hit spam traps. This is not as hard as it may seem, especially at AOL.
– Don’t send a ton of seed emails with every single triggered email. This will hurt your IP reputation.
– Ensure a low percentage of unknown users in a given send.
– Learn what ISP rejection codes mean. Abide by them. If a user no longer exists today, he will not exist tomorrow either.
– Be sure you have the network and server capacity to accept all your complaints and bounces.
– Look at your logs regularly.
– Unsubscribe people immediately, and don’t make them wait, or have to ask more than once. CAN SPAM may say ten days, but human nature says Right Now. Little is more infuriating and will drive your brand’s reputation down faster than getting more mail after having been told that one has been unsubscribed. Consider what humans do when they get angry.
– Don’t send to people who unsubscribed, a couple years later. Permission, once revoked, remains revoked. Consider what humans do when they get angry.
– Realize that marketing mail is a lot more important to you than to the network and end-user you’re sending it to, 98% of the time. Really.
– Don’t try to game the systems. We will figure it out, and that window will get more foggy for everyone. And your specific network will find itself unable to send email to that ISP.
– Study what IP reputation means at each major ISP. Learn it, and live it.
– When you bring new IPs online, warm them up slowly. Be aware of what to expect from various ISPs in such a scenario, and work accordingly. New IPs get rate limited. It’s a reality.
– Use consistent domains and congruent IP ranges whenever possible. Snowshoeing is bad. – Looking like you’re snowshoeing is not great either.
– Sign with DKIM.
– Do your own investigation before you contact the ISPs. You should already have a good idea what client X did wrong.
– Be able to do basic SMTP troubleshooting, or have someone on your team that can. Involve that person before you go ask the ISP for help.
– If you do ask the ISP for help, include useful data like IPs, error messages, time/date stamps, log lines, etc.
– Use the appropriate channels when asking for help. Don’t bother Barry unless you really have to. He’s a busy guy, and his primary focus is not on your marketing mail emergency.
– If you need to, hire a deliverability specialist to help you. You could even hire me!

These are not rules: these are the things we want you to do, and that I personally believe will improve deliverability in most cases. What we will not do is tell you how to go about doing them, what the thresholds are, or what the secret sauce that allows us to measure your success is.