Using the 3D features of a graphics card requires more memory than when it's just used as a 2D device. Double buffering, depth buffering, stencil buffers, textures, etc. all require extra graphics memory. These features may require four times the memory used for a simple 2D display.
If your graphics card doesn't have a lot of memory (less than 16MB, for example), you may have to reduce your screen size and/or color depth in order to use 3D features. Reducing the screen resolution will also leave more space for texture images, possibly improving 3D performance. If, for example, you play Quake3 at 1024x768 but start your display at 1600x1200 you might consider restarting X at 1024x768 in order to maximize your texture memory space.
The documentation included with your card should have information about maximum screen size when using 3D.