The gaps in the wall are made by:

The poem (Robert Frost's 'Mending Wall') explicitly says: 'The work of hunters is another thing... Where they have left not one stone on a stone, But they would have the rabbit out of hiding, To please the yelping dogs.' This clearly attributes the gaps to hunters who break down the wall to drive rabbits out for their dogs. The rabbits are hidden, the dogs benefit but don't break walls, and these specific gaps are man-made by hunters.