As suspected (and as explained by cartoons) brain size does play a small role; they said that brain size accounts for 6.7 percent of variance in intelligence. Recent research has placed the brain's prefrontal cortex, a region just behind the forehead, as providing for 5 percent of the variation in intelligence between people.
The research from Washington University targets the left prefrontal cortex, and the strength of neural connections that it has to the rest of the brain. They think that these differences account for 10 percent of differences in intelligence among people. The study is the first to connect those differences to intelligence in people."
Link to Original Source