Given,

The slope of one of the lines represented by $5x^2+\frac{40}{3}xy+k{y}^2=0$ is $3$

Now comparing with $a{x}^2+2hxy+b{y}^2=0$ we get,

$m_1+m_2=\frac{-2h}{b}=\frac{-40}{3\times k} \& m_1{m}_2=\frac{a}{b}=\frac{5}{k}$,

Now taking $m_1=3$ we get,

$3+m_2=\frac{-40}{3\times k} \& 3m_2=\frac{5}{k}$

$\Rightarrow 3+\frac{5}{3k}=\frac{-40}{3\times k}$

$\Rightarrow 3=\frac{-45}{3\times k}$

$\Rightarrow k=-5$

Now the angle between the pair of lines is given by $\tan \theta =\frac{2\sqrt{h^2-ab}}{a+b}$

So, here $a+b=5-5=0$

$\Rightarrow \theta =90° \text{or} \frac{\mathrm{\pi }}{2}$