I think the “we/they” distinction is a bit of semantics. It especially pisses me off that the religious have co-opted the "Golden Rule", particularly when they're typically the biggest violators of it.

As someone who was raised a Catholic, but now an enlightened atheist for almost a decade now, I think the “Golden Rule” at its core is really just a sound ethical principle to follow. I tend to apply it more in the spirit of “live and let live” or “don’t screw people over”. I think the world would be a much better place if everyone was following the spirit of the golden rule. I feel that humans, among all life on Earth, have this preciously unique ability to be able to evaluate the long term consequences of our actions, yet we continually govern our society and culture based on visceral short term needs just like any other thoughtless organism on our planet.