In an interview designed to shore up the traditional core Tory vote, Mr Duncan Smith spoke out as he prepares to unveil an interim report as head of Mr Cameron's social justice policy group.
IDS tells the Sunday Telegraph that the family unit of "two parents bringing up a child" should be strengthened.
Asked if that only meant heterosexual couples, he says:
I don't think the gay stuff is anything to do with this because it's irrelevant.
We're looking at figures about the bringing up of children. When it comes to gay couples they don't even register on the Richter scale of how to bring up kids.
We are looking at the issue of who brings up kids and the answer is it's men and women that are the issue here.
Men and women are the ones who have the children. Gay couples have nothing to do with this at all.
If you think that something like half of a per cent of Britain are gay, you are dealing with tiny numbers here.
This latest outburst by IDS will be embarrasing for David Cameron, who used his party conference speech this year to back civil partnerships. Cameron said marriage meant something "whether you're a man and a woman, a woman and a woman, or a man and another man".
But Cameron's comments on marriage sparked a backlash from more traditional Tories.
Mr Duncan Smith himself triggered a damaging revolt when he was Tory leader by ordering Conservative MPs to vote against equal adoption rights for unmarried couples - both gay and straight.
UPDATE: Tory blogger Iain Dale says IDS is wrong.
UPDATE 2: IDS has told the BBC the Sunday Telegraph "distorted" what he said.
UPDATE 3 Full transcript of IDS interview with Jon Sopel on BBC Politics Show in which he dismisses the Sunday Telegraph story as "silly".
(Well, it seems you can't believe all you read in the papers - as my recent experience of suing the Mail on Sunday demonstrates!).