Let

$\overrightarrow{a}=a_1\hat{i}+a_2\hat{j}+a_3\hat{k}$

It is $\overrightarrow{a}$ be a vector in the plane containing vectors $\overrightarrow{b}=\hat{i}+2\hat{j}+\hat{k}$ and $\overrightarrow{c}=2\hat{i}-\hat{j}+\hat{k}$, therefore $\overrightarrow{a}, \overrightarrow{b}$ and $\overrightarrow{c}$ are coplanar, so

$\overrightarrow{a}·\left(\overrightarrow{b}\times \overrightarrow{c}\right)=0$

$\Rightarrow \left(a_1\hat{i}+a_2\hat{j}+a_3\hat{k}\right)·\left|\begin{array}{ccc}\hat{i}& \hat{j}& \hat{k}\\ 1& 2& 1\\ 2& -1& 1\end{array}\right|=0$

$\Rightarrow \left(a_1\hat{i}+a_2\hat{j}+a_3\hat{k}\right)·\left(3\hat{i}+\hat{j}-5\hat{k}\right)=0$

$\Rightarrow 3a_1+a_2-5a_3=0 ....\left(\mathrm{i}\right)$

Also, $\overrightarrow{a}$ is perpendicular to $\left(\hat{i}+\hat{j}+3\hat{k}\right)$, therefore

$\left(a_1\hat{i}+a_2\hat{j}+a_3\hat{k}\right)·\left(\hat{i}+\hat{j}+3\hat{k}\right)=0$

$\Rightarrow a_1+a_2+3a_3=0 ....\left(\mathrm{ii}\right)$

Projection of $\overrightarrow{a}$ on $\overrightarrow{b}$ is $3\sqrt{6}$, so

$\frac{\overrightarrow{a}·\overrightarrow{b}}{\left|\overrightarrow{b}\right|}=3\sqrt{6}$

$\Rightarrow \frac{\left(a_1\hat{i}+a_2\hat{j}+a_3\hat{k}\right)·\left(\hat{i}+2\hat{j}+\hat{k}\right)}{\sqrt{1+4+1}}=3\sqrt{6}$

$\Rightarrow a_1+2a_2+a_3=18 ....\left(\mathrm{iii}\right)$

Solving $\left(\mathrm{i}\right), \left(\mathrm{ii}\right) \& \left(\mathrm{iii}\right)$, we get

$a_1=-8, a_2=14, a_3=-2$

So,

$\overrightarrow{a}=-8\hat{i}+14\hat{j}-2\hat{k}$

$\Rightarrow {\left|\overrightarrow{a}\right|}^2=64+196+4=264$