Given, det.$\left(A^{T}BA\right)=27$

$\Rightarrow \left|A^T\right|\left|B\right|\left|A\right|=27$

$\Rightarrow {\left|A\right|}^2\left|B\right|=27 .....\left(i\right)$

Also, det$\left|A{B}^{-1}\right|=8$

$\Rightarrow \frac{\left|A\right|}{\left|B\right|}=8$

$\Rightarrow \left|B\right|=\frac{\left|A\right|}{8} .....\left(ii\right)$

From $\left(i\right)$

$\Rightarrow {\left|A\right|}^2\frac{\left|A\right|}{8}=27$

$\Rightarrow \left|A\right|=6$

From $\left(ii\right)$,

$\left|B\right|=\frac{\left|A\right|}{8}=\frac{6}{8}=\frac{3}{4}$

det$\left(B^T{A}^{-1}B\right)$

$=\left|B^T\right|\left|A^{-1}\right|\left|B\right|$

$=\frac{{\left|B\right|}^2}{\left|A\right|}=\frac{{\left({\displaystyle \frac{3}{4}}\right)}^2}{6}$

$=\frac{9}{\left(16\right)\left(6\right)}=\frac{3}{32}$