Ok, it was as simple as using raspi-config to boot to terminal (CLI) instead of desktop (GUI). I'm new to RPi so this was probably obvious to all.
BTW, fun fact, the desktop uses about 600 KB RAM at 1280x900. That leaves me with about 960 KB free which I hope is sufficient.
Bonus side effect was switching to CLI and back seems to have resolved my system instability. So I am happy.