It could be even worse than that. If the attacker tries to connect to you, your server may try to reverse their IP for logging, and the attacker can control the PTR record. Or the attacker could send you an email that's guaranteed to bounce, and they control the return path that your mailer has to resolve.