(a) The nail has fallen through a total vertical distance of $(h + x).$ Hence loss in tis potential energy = $mg(h + x).$ If average retarding force exerted by sand on nail is $f$, then work done $W = - Fx$
For equilibrium,
$mg(h + x) = Fx$ or $F = mg\frac{h + x}{x} = mg\left( \frac{h}{x} + 1 \right)$