Hummers are hated because they have become the poster child for vehicle inefficiency, just for the sake of image
If you truly go off road, fine, but the vast majority of the people buying them do not - ever. They buy them (and many other SUV's) only because they look tough, and that's just stupid (note I'm not calling you stupid, you indicated you do actually take yours off road).
So why do Hummers get hated more than the other SUV's you mentioned? Because they are amongst the least efficient of an inefficient breed. Take your H3, for example. For less money, you could have purchased a Nissan Xterra which would have provided similar off road capabilities but would have had significantly more power, better mileage, and more interior space, all for a lower price - and would probably be more reliable over the long haul too. You compare your H3 to a 4 Runner, but the 4 Runner is a roomier, more refined vehicle, your H3 has less room than a RAV4.
Ditto the H2 - It gets mileage in the same range as Suburbans and Sequoya's, but has less room than an Explorer.
As for whether the hate will stay with the Hybrids - probably, once you get an image it's hard to lose it.