For more information on the Windows VM's command-line switches, see VmCommandLineOptions.
Macintosh
On the Mac, you can simply edit the amount of memory by doing a "Get Info" on the VM, and increasing the memory allotment in the dialog.
Unix/Linux
Use the switch "-memory 64m", for 64 Megs. A complete startup command might look like:
SqueakVM-2.7-SGI-irix-6.x -memory 64m Squeak2.7.image
Also, you can get the same effect by setting the environment variable SQUEAK_MEMORY.