On your VPS page, enter a hostname under Reverse DNS (IPv4), e.g. mail.your-domain.com, and save with the tick. An empty field resets the entry to the default.
Requirement: the hostname must already point to your server's IPv4 via DNS (A record).
A matching reverse DNS entry is especially important if you run your own mail server – otherwise many providers will put your emails in spam.
You can't currently set reverse DNS for IPv6 yourself. If you need it, send us a ticket.