Since, point $P\left(\alpha ,\beta ,\gamma \right)$ lies on the plane therefore, it satisfy the equation of plane and therefore we have

$2\alpha +\beta +\gamma =1 ...\left(i\right)$

Now, we have,

$\left[\begin{array}{lll}\alpha & \beta & \gamma \end{array}\right]\left[\begin{array}{lll}1& 9& 1\\ 8& 2& 1\\ 7& 3& 1\end{array}\right]=\left[\begin{array}{lll}0& 0& 0\end{array}\right]$

$\Rightarrow \left[\begin{array}{ccc}\alpha +8\beta +7\gamma & 9\alpha +2\beta +3\gamma & \alpha +\beta +\gamma \end{array}\right]=\left[\begin{array}{ccc}0& 0& 0\end{array}\right]$

Hence,

$$\begin{gathered} \alpha +8\beta +7\gamma =0 ...\left(ii\right) \\ 9\alpha +2\beta +3\gamma =0 ...\left(iii\right) \\ \alpha +\beta +\gamma =0 ...\left(iv\right) \end{gathered}$$

Subtracting $\left(iv\right)$ from $\left(i\right)$, we get

$\alpha =1$

Putting in $\left(iii\right) \& \left(iv\right)$, we get

$2\beta +3\gamma =-9 ...\left(v\right)$

$\beta +\gamma =-1 ...\left(vi\right)$

On solving, we get $\gamma =-7 \& \beta =6$

Therefore,

${\alpha }^2+{\beta }^2+{\gamma }^2=86$