I'd successfully configured Exchange 2010 server and built it into my new environment, incoming e-mails were being successfully received by SharePoint but for some reason, outgoing - i.e. alerts - were failing and ending up in the mailroot\badmail directory.
When I opened up one of the .BAD e-mail files from the badmail directory in WordPad I discovered the error:
Final-Recipient: <recipientname>
Action: Failed
Status: 5.7.1
Diagnostic-Code: smtp;530 5.7.1 Client was not authenticated
The reason was due to the fact I hadn't check-boxed the anonymous users check-box in my Exchange 2010 server's hub transport server default receive connector (by default, it's unchecked). So of course, Exchange was rejecting anything sent via my MOSS server's smtp. In my case, my smtp's access and outbound security was set to anonymous (internal dev environment) and I was also working between multiple physical servers.
Remember, if you need to test smtp on Windows Server 2008 or Windows Server 2008 R2 one option is to use Telnet. Yep, it's been around forever and it's still great for debugging internal and isolated smtp issues ;-)