Given,

The tangent to the parabola $y^2=12x$ at the point $\left(3, \alpha \right)$ be perpendicular to the line $2x+2y=3$

So, Slope of tangent $=1=\frac{6}{\alpha }\Rightarrow \alpha =6$
So, equation of hyperbola will be,

$36x^2-9y^2=324$

$\Rightarrow \frac{x^2}{9}-\frac{y^2}{36}=1$
Now equation of tangent at $\left(5,8\right)$ will be,

$\frac{5x}{9}-\frac{8y}{36}=1$

$\Rightarrow 5x-2y=9$
So, slope of normal$=\frac{-2}{5}$
Now finding, equation of normal $y-8=\frac{-2}{5}\left(x-5\right)$
$\Rightarrow 5y-40=-2x+10$

$\Rightarrow 5y+2x=50$ 

Now finding,distance from $\left(6, -4\right)$, we get,
$=\left|\frac{12-20-50}{\sqrt{29}}\right|=\frac{58}{\sqrt{29}}$

$=2\sqrt{29}$

Hence, ${\left(2\sqrt{29}\right)}^2=116$