(b) $V_{A} =$( Potential at $A$ due to $A$) + (Potential at $A$ due to $B$)
$\Rightarrow \ \ V_{A} = - \frac{Gm_{1}}{R} - \frac{Gm_{2}}{\sqrt{2}R}$
Similarly,
$V_{B} =$(Potential at $B$ due to $A$) + (Potential at $B$ due to $B$)
$\Rightarrow \ \ V_{B} = - \frac{Gm_{2}}{R} - \frac{Gm_{1}}{\sqrt{2}R}$
Since, $W_{A \rightarrow B} = m(V_{B} - V_{A}) \Rightarrow W_{A \rightarrow B}$
$= \frac{Gm\left( m_{1} - m_{2} \right)(\sqrt{2} - 1)}{\sqrt{2}R}$