There were a lot of butthurt people around because Albert was considered the most attractive prince in all of Europe and they couldn't have him, and his total devotion to Victoria and refusal to let himself be used as a tool to influence her made him a lot of enemies. He pretty much hated the court assholes and they hated him.
Dude was also a polymath, and did things like invent remote locks for their bedroom doors so they didn't have to waste precious seconds when it was time to get busy with his wife. He was an absolute chad, but a monogamous one.