$\because$ Lines $x^2+k x y+y^2$ and $x+y-1=0$ make
the sides of equilateral triangle.(given)



$\because x^2+k x y+y^2=0$ represent a pair of straight lines passing through origin.
Let $m$ be the slope of one line whose equation is $y=m x$
According to the question,
$$
\begin{aligned}
& \tan \theta=\left|\frac{m_1-m_2}{1+m_1 m_2}\right|, \theta=\text { angle between lines } \\
& \Rightarrow \tan \quad 60^{\circ}=\left|\frac{m-(-1)}{1+m(-1)}\right|[\because \text { slope of } x+y=1 \text { is }-1] \\
& \Rightarrow \quad \sqrt{3}=\frac{m-1}{1-m} \\
& \Rightarrow \quad(1-m)^2=(m+1)^2 \\
& \Rightarrow \quad 2 m^2-8 m+2=0 \\
& \Rightarrow \quad m^2-4 m+1=0
\end{aligned}
$$
$$
\begin{aligned}
& \Rightarrow\left(\frac{y}{x}\right)^2-4\left(\frac{y}{x}\right)+1=0 \quad\left[\because y=m x \Rightarrow \frac{y}{x}=m\right] \\
& \Rightarrow \quad \frac{y^2}{x^2}-\frac{4 y}{x}+1=0 \Rightarrow y^2-4 x y+x^2=0
\end{aligned}
$$
Compare with $x^2+k x y+y^2=0$, we get $k=-4$
$$
\therefore \quad k^2=(-4)^2=16
$$