Maybe fathers don't matter after all

From Brandon Berg at Distributed Republic:

It's quite well-documented that children of single mothers tend to perform much worse than children of married couples on a variety of socioeconomic metrics. The conclusion generally drawn from this is that children benefit greatly from being raised by both of their biological parents.

However, there are two other facts which are also reasonably well-documented:

1. Women* with low IQs are dramatically more likely to have children out of wedlock than women with high IQs.

2. IQ has a strong heritable component.

Which leads to a question which seems obvious in retrospect, but which I've never seen raised before: To what extent is the underperformance of children of single mothers due to being raised without fathers, and to what extent is it due to the fact that children inherit low cognitive ability and/or poor impulse control from their mothers**? Do fathers really matter?

*This may also apply to men, though I'm not personally aware of any research on the topic.

**And possibly fathers—see prior footnote.

 Well, maybe it's entirely due to hereditary factors, and the structure of the family has no contribution.  But it seems to me, the people who are arguing that any family structure is as good as any other will hate this proposed alternative theory even more.