(b)
LHS = $\int_{}^{}\left( \log x \right)^{2}dx$
$= x\ \left( \log x \right)^{2} - \int_{}^{}x.2\log x\frac{1}{x}\ dx$
$= x\ \left( \log x \right)^{2} - 2\left\lbrack x\log x - \int_{}^{}x.\frac{1}{x}dx \right\rbrack + c$
$= x\ \left( \log x \right)^{2} - 2\lbrack x\log x - x\rbrack$
$= x\ \left( \log x \right)^{2} - 2x\left\lbrack \log x - 1 \right\rbrack + c$
But RHS is given by
$x\left\lbrack f(x) \right\rbrack^{2} + Ax\left\lbrack f(x) - 1 \right\rbrack + c$
$\therefore f(x) = \log x$ and $A = - 2$