Let, $v=$ velocity of the rocket when it is fired from the earth.
$h=$ height at which its velocity becomes zero.
Total energy of the rocket at the surface of the earth
$=\mathrm{K} . \mathrm{E} .+\mathrm{P} . \mathrm{E} .=\frac{1}{2} m v^2+\left(-\frac{G M m}{R}\right)$
At the highest point $v=0$
$\therefore \quad$ K.E. $=0$ and P.E. $=\frac{-G M m}{R+h}$
$\therefore T . E=\frac{-G M m}{R+h}$
Using law of conservation of energy,
$$
\begin{aligned}
&\frac{1}{2} m v^2-\frac{G M m}{R}=-\frac{G M m}{R+h} \\
&\frac{1}{2} v^2=G M\left[\frac{1}{R}-\frac{1}{R+h}\right]=G M\left[\frac{R+h-R}{R(R+h)}\right]
\end{aligned}
$$
$$
\begin{aligned}
&\Rightarrow \frac{1}{2} v^2=\frac{g R^2 h}{R(R+h)} \quad\left[\because g R^2=G M\right] \\
&\frac{1}{2} v^2=\frac{g R h}{R+h} \Rightarrow v^2(R+h)=2 g R h \\
&\Rightarrow v^2 R+v^2 h=2 g R h \\
&\Rightarrow h\left(v^2-2 g R\right)=-v^2 R \\
&\Rightarrow h\left(2 g R-v^2\right)=v^2 R \\
&\Rightarrow h=\frac{v^2 R}{2 g R-v^2}
\end{aligned}
$$
$$
\begin{aligned}
&=\frac{\left(5 \times 10^3\right)^2 \times 6.4 \times 10^6}{2 \times 9.8 \times 6.4 \times 10^6-\left(5 \times 10^3\right)^2} \\
&=1.6 \times 10^6 \mathrm{~m}
\end{aligned}
$$