Answer

Hello.
This is a broad topic, and I’m also not expert in all the details. I’ll write what seems to me, based on my understanding.
 
The claims that anyone who isn’t “one of us” is “absolute treif” are purely political. You can calmly ignore them (and the stories that accompany them). The proof is that everyone says this about everyone else, and obviously they can’t all be right (certification X can’t be both the very best and absolute treif).
 
As a rule, in any large institutional kashrut system there are always problems—whether it’s a Badatz, Rabbi Landau, or the Rabbinate. I’ve heard hair-raising stories and testimonies about all of them. Some of the stories are tall tales, and some are true, but those exist in all kashrut systems.
 
The Rabbinate—which I am the last person likely to serve as defense attorney for—is steadily improving a great deal on kashrut. My impression is that the main problems originally stemmed from Haredi control over Rabbinate kashrut. The Haredim, who in any case don’t eat from it themselves, didn’t care much about letting those who “aren’t careful about kashrut” (since in their view it’s treif anyway) eat whatever they would eat. As I said, my impression is that the situation is improving today, and the Rabbinate is becoming more and more professional in this area, although its general conduct is like the Rabbinate in other areas, and no elaboration is needed. I don’t think private bodies are run much better. There are a lot of interests involved here, and this is not the place to elaborate. As a public institution, the Rabbinate has mechanisms for improvement and oversight, which in my estimation exist less in private bodies. Although of course the Rabbinate is a public body run like a little shtibl, so it isn’t really like any other public institution.
 
In general, it is accepted that one may rely on the laws of credibility and on presumptions, and the presumption is that a kashrut body does not feed the public carrion or forbidden meat. Beyond the rule that one witness is trusted regarding prohibitions, there is also the presumption that a professional will not undermine his own credibility. Even if there are problems—and there certainly are—one is permitted to rely on that presumption and eat anyway.
It is true that with ordinary Rabbinate kashrut they sometimes follow standards that are not universally accepted (only accepted by some opinions), but in my opinion there is no actual non-kosher food there, and you are not stumbling in anything serious. All this unless proven otherwise. Therefore it is preferable to eat food with Rabbinate mehadrin certification, but that too is not really obligatory.
Nowadays, if you don’t rely on established kashrut systems, you simply can’t live. You’d have to produce everything yourself at home. That isn’t practical, and the Torah was not given to ministering angels.
 
The concern about eating forbidden foods comes from the taboo notion that it dulls the soul and that non-kosher food has metaphysical effects. I don’t see it that way. In my view it is a law like any other, and one can follow the rules of Jewish law here without fear. If you failed inadvertently, then it was under compulsion, and the Merciful One exempts a person under compulsion. Nothing happened. And even if it does dull the soul, that would only be if you violated Jewish law, not when you kept it.
There is an interesting responsum by the Ben Ish Hai on this matter; see Responsa Rav Pe'alim, vol. 4, at the end, in the section Sod Yesharim, siman 5—astonishing things:
https://1drv.ms/w/s!Ahr4iW7qjF2o0SrEOTfujow1-dU3
And there as well, in the main body of the responsa, Orach Chayim siman 2, he brings an incident involving his grandfather, who was the rabbi of Baghdad. One day a traveling collector who had arrived in the city revealed to him that all the tefillin worn by the city’s residents for generations had not been sufficiently square. Their conclusion was that everyone’s tefillin had been invalid. He discusses there whether it is possible that all these people had never in their lives fulfilled the commandment of tefillin, and that all of them fell into the category of “a skull that never wore tefillin.” He proved from several sources that this is not so, and even though tefillin is a positive commandment, and in practice we hold that compulsion is not considered as though one acted, here they are regarded as people who did fulfill the commandment of tefillin. This is a startling innovation, but certainly with regard to prohibitions—where compulsion is considered as though one did not act—one can say this, as he indeed wrote in the responsum cited above.