Ah, I see. I can't really say that I have much (well, any) experience on that scale. I'm thinking that it can be done with mail servers, it's just a matter of installing enough hardware. However, the SMTP protocol (and the handling of mails by postfix internally) supports many more features that you're likely going to need for intra-site messaging (even between different instances of your service), which slows down processing (read: you'll spend more on hardware) and increases maintenance requirements and failure risk.
Then again, if you implement something of your own, you might end up with crappy software that is unreliable and limiting, or you might spend more on development than postfix's complexity would otherwise cost you.
One thing you might consider however is thinking of messages and their transport as separate. Define a message format that is suitable for eventual later transmission via SMTP without too much massaging, but as long as your installation is still small and limited to a single database instance, make the transport of messages as simple as you can (e.g., just write the message directly into the receiver's inbox upon creation/