Given that, $A=\left[\begin{array}{cc}a& -b\\ b& a\end{array}\right]$ and $ab=\frac{5}{2} ....\left(\mathrm{i}\right)$

$\Rightarrow A^T=\left[\begin{array}{cc}a& b\\ -b& a\end{array}\right]$

$\Rightarrow A{A}^T=\left[\begin{array}{cc}{a}^2+b^2& 0\\ 0& a^2+b^2\end{array}\right]$

Also given $A{A}^T=20I$

$\Rightarrow \left[\begin{array}{cc}{a}^2+b^2& 0\\ 0& a^2+b^2\end{array}\right]=\left[\begin{array}{cc}20& 0\\ 0& 20\end{array}\right]$

$\Rightarrow a^2+b^2=20$

$\Rightarrow {\left(a+b\right)}^2-2ab=20$

$\Rightarrow {\left(a+b\right)}^2-2\times \frac{5}{2}=20$ [from $\left(\mathrm{i}\right)$]

$\Rightarrow {\left(a+b\right)}^2=25$

$\Rightarrow a+b=\pm 5$

Now, we know that general form of a quadratic equation is $x^2-\left(\mathrm{sum} \mathrm{of} \mathrm{roots}\right)x+\left(\mathrm{product} \mathrm{of} \mathrm{roots}\right)=0$

$\Rightarrow x^2\pm 5x+\frac{5}{2}=0$

$\Rightarrow 2x^2\pm 10x+5=0$