Can I set up a custom SMTP server for email notifications?

No.  This feature has been removed and only available on older accounts (as of October 2014).

For those still with access, you can set up AceProject so that the application e-mail notifications are sent through your own SMTP server. This may be done under  Administration => Account > Edit Account , in the “SMTP Server” section.