$$
\begin{aligned}
& \mathbf{c}=(\mathbf{a} \times \mathbf{c})+\mathbf{b} \\
& \Rightarrow \quad|\mathbf{c}|^2=|\mathbf{a}+\mathbf{c}|^2+|\mathbf{b}|^2+2(\mathbf{a} \times \mathbf{c}) \cdot \mathbf{b} \\
& \Rightarrow \quad 2[\mathbf{a b c}]=|\mathbf{a} \times \mathbf{c}|^2+1-|\mathbf{c}|^2
\end{aligned}
$$
Let angle between vectors $\mathbf{a}$ and $\mathbf{c}$ is ' $\theta$ '.
So, $\quad\left[\begin{array}{lll}\mathbf{a} & \mathbf{b} & \mathbf{c}\end{array}\right]=\frac{1}{2}\left[|\mathbf{c}|^2 \sin ^2 \theta-|\mathbf{c}|^2+1\right]$


From Eq. (i) and (ii)
$$
\begin{aligned}
& \frac{1}{2}\left\{1-|\mathbf{c}|^2 \cos ^2 \theta\right\}=|\mathbf{c}|^2 \sin ^2 \theta \Rightarrow|\mathbf{c}|^2=\frac{1}{\cos ^2 \theta+2 \sin ^2 \theta} \\
& \because\left[\begin{array}{lll}
\mathbf{a} & \mathbf{b} & \mathbf{c}
\end{array}\right]=|\mathbf{c}|^2 \sin ^2 \theta=\frac{\sin ^2 \theta}{\cos ^2 \theta+2 \sin ^2 \theta}=\frac{\sin ^2 \theta}{1+\sin ^2 \theta} \\
& =\frac{1}{\operatorname{cosec}^2 \theta+1} \\
&
\end{aligned}
$$
For maximum value of $\left[\begin{array}{lll}\mathbf{a} & \mathbf{b} & \mathbf{c}\end{array}\right], \operatorname{cosec}^2 \theta=1$, so
$\left[\begin{array}{lll}\mathbf{a} & \mathbf{b} & \mathbf{c}\end{array}\right]_{\max }=\frac{1}{2}$