Tim Bray confesses about having two patents in the pipeline and goes on to talk about software patents. I also have a couple of security-related patents in the pipeline. To me, it's not the software patents but patents that violate public's right of passage.
GIF patent was legit but Unisys was standing there and charging toll on what most of us considered public road. What is public road and what is not? The distinction is simple. If your enforcement of the patent hurts your public relations more than it adds to your bottomline, then you are standing on a public road.
To be more precise, if your patent gives your solution advantages in quality of service, then it's legit. But if your patent leads to the only solution, then you are a troll. If your patented formula makes cars go faster, I am fine with that. If you patented the idea of automobiles, I am not all right with that and all for public's right to steamroll over such patents.
My justification is this. Since patent laws can be changed or even banished by the people, the people has the right to selectively change or banish any specific patent it chooses. Implementation is problematic, but the principle is sound IMHO.