Me temo que para mac solo hay svga de 128 Mb, si virtualizas windows si que puedes aumentar .
Esto es del manual.
Virtual Machine Graphics and Keyboard Support
Virtual machines support certain graphics features and keyboards.
n VGA and SVGA are supported. n 104-key Windows 95/98 enhanced keyboards are supported.
n To use the GL_EXT_texture_compression_s3tc and GL_S3_s3tc Open Graphics Library (OpenGL) extensions in a Windows XP or Windows 7 guest operating system, you must install Microsoft DirectX End-User Runtime in the guest operating system. OpenGL is an application program interface that is used to define 2D and 3D computer graphics. You can download Microsoft DirectX End-User Runtime from the Microsoft Download Center Web site.