We’ve experience the following behavior with SharePoint 2010.
In SharePoint 2007, when creating an email alert on a SharePoint list, you can specify an Active Directory Security Group as the recipient of the alerts. (AD Distribution Groups do not work, and that is by design).
In SharePoint 2010, this capability does not seem to exist, or at least it does not work. You can create the alert, add an AD Security Group as the recipient of the alert, and save the alert. This will notify the group members that the alert has been created, and that they have been added. However, when performing an action that would generate an alert (a new SharePoint List item, for example) the email alerts are never sent.
We knew that SharePoint and Exchange are communicating, as the notification of the creation of the alert gets sent out.
The quick fix is to add users individually to the Alert. This will work as expected for small groups but will get tedious for groups with a lot of members.