Most people dislike pain and prefer pleasure or happiness.
The happiness of one is, in at least most cases, contingent on the happiness of others. We have already in this class established the fact that everyone, short of sociopaths and psychopaths, experiences sympathy. When others feel pain or pleasure, you feel pain or pleasure.
I should think it would be safe to make the assumption that any individual would say that if he himself were caused pain, the action taken against him would have been an immoral one.
There are few people (again, only sociopaths, psychopaths and masochists) that enjoy experiencing or inflicting pain without remorse. While we can argue that the immorality of pain as defined by others applies to these as well, we must also acknowledge that people who fall into these three categories may disagree with the definition.
Excluding the opinions of these abnormal groups, the majority of mankind would agree, given the support provided, that to cause pain or sorrow to another is immoral. If causing someone else pain or sorrow results in experiencing that same pain or sorrow yourself (sympathy), and having someone inflict pain or sorrow on you is immoral, it would only make sense that causing unhappiness to another is immoral.