Not trying to be helpful
"I am not trying to be helpful" is a useful way to position what you are about to say.
It is a permission statement: A statement that lets you say what is on your mind. One that lets the other raise their filters a bit knowing what is coming is definitely not meant to be helpful.
If sometimes it still is — then that is a happy coincidence.