Explanation

(a)
Let, $X = \begin{pmatrix}
a & b \\
c & d
\end{pmatrix}$
$\Rightarrow X^{2} = \begin{pmatrix}
a^{2} + bc & ab + bd \\
ac + cd & bc + d^{2}
\end{pmatrix}$
$\Rightarrow a^{2} + bc = 1\ and\ ab + bd = 1 \Rightarrow b(a + d) = 1$
$ac + cd = 2 \Rightarrow c(a + d) = 2 \Rightarrow 2c = c$
Also,
$bc + d^{2} = 3 \Rightarrow d^{2} - a^{2} = 2$
$\Rightarrow (d - a)(a + d) = 2 \Rightarrow d - a = 2b$ (using $bc = 1 - a^{2}$)
$a + d = \frac{1}{b}$
$\Rightarrow 2d = 2b + \frac{1}{b},\ \ \ 2a = \frac{1}{b} - 2b$
$d = b + \frac{1}{b},\ \ \ \ \ \ \ \ \ \ \ a = \frac{1}{b} - 2b$
$c = 2b$
$\Rightarrow \left( b^{2} + \frac{1}{4b^{2}} + 1 \right) + 2b^{2} = 3$
$\Rightarrow 3b^{2} + \frac{1}{4b^{2}} = 2$
$\Rightarrow 3x + \frac{1}{4x} = 2$
$\Rightarrow b = \pm \frac{1}{\sqrt{6}}$ or $b = \pm \frac{1}{\sqrt{2}}$
Therefore, matrices are
$\begin{pmatrix}
0 & \frac{1}{\sqrt{2}} \\
\sqrt{2} & \sqrt{2}
\end{pmatrix}\begin{pmatrix}
0 & - \frac{1}{\sqrt{2}} \\
- \sqrt{2} & - \sqrt{2}
\end{pmatrix}\begin{pmatrix}
\frac{2}{\sqrt{6}} & - \frac{1}{\sqrt{6}} \\
\frac{2}{\sqrt{6}} & \frac{4}{\sqrt{6}}
\end{pmatrix}$