Could you give a source for the claim that someone inadvertent who is close to being under duress is not liable for a sin-offering? I haven’t found one.

Maimonides, Laws of Errors, chapter 2, halakha 6:
Anyone liable for a fixed sin-offering for his inadvertent transgression, who acted inadvertently and only afterward became aware that he had sinned—although he had no prior knowledge that this was a sin—is liable for a sin-offering. How so? A child taken captive among the gentiles who grew up among them and does not know who Israel are nor their religion, and he performed labor on the Sabbath and ate forbidden fat and blood and the like—when he becomes aware that he is an Israelite and commanded regarding all these things, he is obligated to bring a sin-offering for each and every transgression, and so too in all similar cases.

Good question. For some reason I had assumed that this distinction between types of inadvertence was not said only regarding exile to a city of refuge (where of course it is explicit).

When I looked into the books, I found a dispute among the medieval authorities regarding one who says “it is permitted” and a child taken captive: Maimonides, who was cited above, holds that he is liable for a sin-offering even if he never knew of the prohibition at all, but Rashi on Keritot 2a wrote that he is exempt because he is under duress. Even though this is obviously a case of inadvertence by the usual definitions, it is still close to duress. It seems clear that at least according to Rashi, someone inadvertent who is close to duress is exempt from a sin-offering. And even in Maimonides I am inclined to think he would agree to that; it is just that in his view the line between inadvertence and inadvertence close to duress is different. See Makkot 9a, the dispute about whether one who says “it is permitted” is close to duress or specifically close to intentional violation.

The law is also like this regarding an oath in Shevuot 26a, where inadvertence is called duress (though see Rashba there, who says that even here one who says “it is permitted” is inadvertent). But regarding an oath it seems that this is a special rule because “his heart compelled him” (Rashi on Shevuot there).

And regarding secular Jews in our day, most halakhic decisors wrote that they are like people under duress and like children taken captive, and it is clear that their state is one of inadvertence (we are talking about lack of knowledge). Still, this too is not a proof, because today there is no obligation of a sin-offering, and perhaps their intent is only to say that they are inadvertent and not intentional sinners.

In short, by reasoning and from Scripture it seems that someone inadvertent who is close to duress is exempt from a sin-offering—that is, there are situations of lack of knowledge that are considered duress. The question is when he is called close to duress, and on that the opinions differ.

A. Why do you resist the reasoning that a sin-offering has nothing at all to do with punishment, and there is no claim against the person whatsoever, but only that he performed an act of prohibition—that is, he was commanded and violated it—and a metaphysical blemish and displeasure before the Holy One, blessed be He, was created, and thus the world was created in such a way that the sacrifice repairs, cleanses, and atones for that blemish?
B. “And even in Maimonides I am inclined to think he would agree to this (that someone inadvertent who is close to duress is exempt from a sin-offering), except that in his view the line between inadvertence and inadvertence close to duress is different.” Meaning: a child taken captive is not considered close to duress and therefore is liable for a sin-offering, but one who says “it is permitted” is considered close to duress and is exempt from a sin-offering? I have trouble seeing the logic in that. If anything, the opposite: one who says “it is permitted” could have learned and did not learn (and therefore there is an opinion that he is close to intentional violation), and if he is close to duress then all the more so a child taken captive should be close to duress.
C. The Kesef Mishneh there refers to Shabbat 68b: a dispute among Amoraim whether a child taken captive is liable for a sin-offering. So it seems that in any case we need to organize the whole body of Torah according to the two approaches, even if the medieval authorities disagreed over which opinion the law follows.

A. The difference between inadvertence and duress is explained (for example by Nachmanides at the beginning of Leviticus) by saying that in inadvertence there is negligence, and therefore a measure of blame. In other words, a sin-offering is brought only when there is blame, not merely because of the outcome itself. See my article on the guilt-offering, where I explain in exactly this way the difference between a sin-offering and a guilt-offering. https://mikyab.net/%D7%9B%D7%AA%D7%91%D7%99%D7%9D/%D7%9E%D7%90%D7%9E%D7%A8%D7%99%D7%9D/%D7%9E%D7%94%D7%95%D7%AA%D7%95-%D7%A9%D7%9C-%D7%A7%D7%A8%D7%91%D7%9F-%D7%90%D7%A9%D7%9D
See also my article on causing a secular Jew to stumble in a transgression, where I explained the nature of the sin-offering—that it is brought for the forgetfulness/errors, not for the transgressions themselves (see the discussion there about Afikei Yam; I think it is in a note). https://mikyab.net/%D7%9B%D7%AA%D7%91%D7%99%D7%9D/%D7%9E%D7%90%D7%9E%D7%A8%D7%99%D7%9D/%D7%91%D7%A2%D7%A0%D7%99%D7%99%D7%9F-%D7%94%D7%9B%D7%A9%D7%9C%D7%AA-%D7%97%D7%99%D7%9C%D7%95%D7%A0%D7%99-%D7%91%D7%A2%D7%91%D7%99%D7%A8%D7%94

B. There are arguments both ways. For example, in the case of one who says “it is permitted,” he is under duress because he checked and that was his conclusion. As opposed to a child taken captive, who did not check, so there is a possibility that when he checks he will understand that he is mistaken. Reasonings of this sort come up in these passages in every direction.

C. It is possible that these views are disputing the question of someone inadvertent who is close to duress in relation to a sin-offering. By reasoning it seems he is exempt. Or perhaps the dispute is only about a child taken captive—whether he is close to duress or merely inadvertent.