Email server on a VPS: set up domain mail that stays out of spam

To run an email server on a VPS (for addresses like hello@yourcompany.com), you install a mail stack, typically Postfix to send and receive, Dovecot for IMAP/POP3 and a spam filter, then publish four DNS records: MX, SPF, DKIM and DMARC. The prerequisites are a VPS provider that allows outbound port 25 and lets you set the IP's reverse DNS (PTR) to your mail hostname; without either, mail will not leave the server or will land in spam.

This guide helps you decide whether to self-host email at all, check that your VPS qualifies, get DNS right, set up the mail server with ZoPanel or by hand, and fix the most common problems.

Should you host email on your own VPS?

Honestly, email is the hardest part of running a hosting server. Not because of the installation, but because of sender reputation: Gmail, Outlook and the other big mailbox providers judge your IP and domain constantly.

	Self-hosted on a VPS	Email service (Google Workspace, Microsoft 365, Zoho…)
Cost	Little or nothing extra, even for hundreds of mailboxes	Per user, per month
Data control	Full; mail stays on your server	With the provider
Operational work	High: DNS, spam, blocklists, backups	Low
Inbox placement	Depends on IP, PTR, configuration and sending habits	Usually good from day one
Best for	Hosting companies, agencies, many small domains	Businesses that need reliability with little admin time

If you sell hosting, customers almost always expect email with it, so a mail server on the VPS is hard to avoid. If you only need a few mailboxes for your own company, a dedicated email service is often less work.

What you need to run an email server on a VPS

1. Outbound port 25 must be open

Many cloud providers block port 25 to fight spam. For example, DigitalOcean blocks SMTP ports 25, 465 and 587 by default on Droplets, and AWS EC2 blocks port 25 to public addresses until you request removal. Ask your provider before you rent a VPS for mail. To test from the VPS:

timeout 5 bash -c '</dev/tcp/gmail-smtp-in.l.google.com/25' && echo "Port 25 open" || echo "Port 25 blocked"

If it is blocked you can still receive mail, but to send you need a relay (smarthost) such as Amazon SES, SendGrid, Mailgun or Brevo.

2. Reverse DNS (PTR) pointing to the mail hostname

A PTR record maps an IP back to a name. It is managed by your VPS provider, not your domain registrar. Set the IP's PTR to mail.yourcompany.com, and make sure mail.yourcompany.com has an A record pointing back to the same IP. Check:

dig -x 203.0.113.10 +short
dig +short mail.yourcompany.com A

3. A clean IP

Check whether the VPS's IP is on a blocklist (Spamhaus, SpamCop, Barracuda) before you use it. An IP a previous tenant used for spam can get your mail rejected however well you configure it.

The DNS records you need

Record	Name	Example value	Purpose
A	mail.yourcompany.com	the VPS IP	The mail server's hostname
MX	yourcompany.com	mail.yourcompany.com, priority 10	Where mail for the domain is delivered
TXT (SPF)	yourcompany.com	v=spf1 mx a ~all	Which servers may send for the domain
TXT (DKIM)	zp._domainkey.yourcompany.com	public key	Signature proving the message was not altered
TXT (DMARC)	_dmarc.yourcompany.com	v=DMARC1; p=quarantine; rua=mailto:postmaster@yourcompany.com	What to do when SPF/DKIM fail

The DKIM selector (zp above) and the public key come from your mail server; other software may use a different selector.

Gmail's requirements

According to Google's email sender guidelines, every sender to Gmail needs SPF or DKIM, valid forward and reverse DNS (PTR), TLS for transmission, and a spam rate in Postmaster Tools below 0.3%. Senders of more than 5,000 messages a day to Gmail also need both SPF and DKIM, DMARC (the policy can be none), a From domain aligned with SPF or DKIM, and one-click unsubscribe for marketing mail.

Setting up an email server on a VPS with ZoPanel

ZoPanel's mail server is built on Postfix, Dovecot and Rspamd, with DKIM signing, spam filtering and Sieve rules (email docs).

Step 1: Install the mail server

1. Create an A record for mail.yourcompany.com pointing to the VPS.
2. Open the Email page and set Mail hostname to mail.yourcompany.com.
3. Click Install mail server. You can add Webmail and Calendars & contacts (CalDAV/CardDAV) from Components.
4. Ask your VPS provider to set the IP's PTR to mail.yourcompany.com.

The installer opens the mail ports in the firewall. If your provider blocks outbound port 25, the Email page warns you and you can configure Outgoing mail relay (smarthost). Set Settings → Panel domain & SSL so mail clients see a trusted certificate.

Step 2: Enable email for the domain and publish DNS

1. Click Enable email for a domain, choose the website's domain and click Enable.
2. Publish the records from the DNS records card at your DNS provider. If the domain's DNS is hosted on this server (DNS), a zone created after enabling email already contains them.
3. Click Run check in Mail delivery check. It compares public DNS with the expected MX, SPF, DKIM and DMARC, checks PTR and outbound port 25, and looks the IP up on the Spamhaus, SpamCop and Barracuda blocklists, with a hint for each problem.

Step 3: Create mailboxes

Open the domain, click New mailbox, and enter the address, password and Quota (MB). Users can open webmail without typing a password from Open webmail (no password needed), or set up Outlook, Thunderbird or a phone:

Protocol	Port	Security
IMAP	993	SSL/TLS
POP3	995	SSL/TLS
SMTP (sending)	465	SSL/TLS
SMTP (sending)	587	STARTTLS

The username is always the full email address.

Step 4: Protect your IP's reputation

Hacked websites are the most common source of spam on a hosting server. ZoPanel counts the recipients each account sends to per hour (500 by default, set in Email → Sending limits or per package), and Block direct SMTP from accounts is on by default: website code cannot connect to other servers' port 25, so all mail goes through the mail server where it is counted, DKIM-signed and logged. ZoPanel also checks daily whether the IP is on a blocklist and alerts you when the mail queue builds up.

Everything above is available on ZoPanel's Free plan (3 hosting accounts, 10 websites, 10 databases), so you can try the whole process on a test VPS.

What about building the mail server by hand?

It is entirely possible, but expect a substantial amount of work. You will need to:

1. install and configure Postfix (SMTP), including TLS, SASL authentication through Dovecot and restrictions that prevent an open relay;
2. install Dovecot for IMAP/POP3, the mailbox format and quotas;
3. add a spam filter (Rspamd or SpamAssassin) and DKIM signing;
4. get an SSL certificate for the mail hostname and keep it renewed;
5. add webmail, Fail2ban jails for the mail services and mail backups;
6. watch the queue, logs and blocklists for as long as the server runs.

One mistake, such as leaving Postfix as an open relay, can get your IP blocklisted within hours. If you build it yourself, work from the official Postfix and Dovecot documentation and test thoroughly with an external mail tester before going live.

Common problems with a VPS mail server

Symptom	Usual cause	Fix
Mail arrives but cannot be sent	Provider blocks port 25	Ask for it to be opened, or use a smarthost
Mail to Gmail lands in spam	Missing PTR, wrong SPF/DKIM, new IP with no reputation	Fix PTR and DNS, send steadily, avoid bulk sends at first
No incoming mail	Wrong MX, inbound port 25 closed in the cloud firewall	Check dig MX, open inbound port 25
DKIM fails	TXT record truncated or wrong selector	Paste the full key again, check with dig TXT
Outlook shows a certificate error	Mail hostname has no valid certificate	Issue a certificate for the mail hostname

If mail still goes to spam with correct DNS, see emails going to spam: SPF, DKIM and DMARC. If the domain is not pointed yet, start with how to point a domain to a VPS.

Conclusion

An email server on a VPS is practical and economical when you host many domains, as long as you pick a provider that allows port 25 and PTR records, and publish correct MX, SPF, DKIM and DMARC records. The ongoing work is protecting your IP's reputation: sending limits, no direct SMTP from websites, and watching blocklists.

You can install ZoPanel for free to get a mail server, webmail, DNS checks and sending limits in a few clicks, then read the email docs for the details.

Frequently asked questions

Is running an email server on a VPS free?

The software (Postfix, Dovecot, Rspamd) is open source and free, so you only pay for the VPS. The real cost is operational time: DNS, spam handling, blocklist monitoring and mail backups.

Why do emails from my VPS go to spam?

The usual causes are missing reverse DNS (PTR), missing or wrong SPF, DKIM or DMARC, or a new IP with no reputation. Gmail requires every sender to have SPF or DKIM, valid PTR and TLS.

How do I know if my VPS blocks port 25?

Try connecting from the VPS to an external mail server on port 25, for example gmail-smtp-in.l.google.com. If it fails, ask your provider; large clouds such as DigitalOcean and AWS block it by default.

What if my provider blocks port 25?

You can still receive mail normally. To send, configure the mail server to relay through a smarthost such as Amazon SES, SendGrid, Mailgun or Brevo, on port 587 or another port the service supports.

How much RAM does a VPS mail server need?

It depends on the number of mailboxes and the volume. Spam and virus filtering use a noticeable amount of memory, so for a server running both websites and mail, allow at least 2 GB of RAM and watch usage after installing.