(b)
$f^{'}(x) = \left( ab - b^{2} - 2 \right) + \cos^{4}x + \sin^{4}x < 0$
$= ab - b^{2} - 2 + \left( \sin^{2}x + \cos^{2}x \right)^{2} - 2\sin^{2}x\cos^{2}x < 0$
$\Rightarrow \ \ \ ab - b^{2} - 1 < \left( \frac{1}{2} \right)\sin^{2}{2x} < \frac{1}{2}$
$\Rightarrow \ \ \ 2ab - 2b^{2} - 2 < 0$
$\Rightarrow \ \ 2b^{2} - 2ab + 3 > 0$
$\therefore\left( - 2a^{2} \right) - 4.2.3 < 0$
$\Rightarrow a^{2} < 6$
$\Rightarrow \ \ - \sqrt{6} < a < \sqrt{6}$