<Cause>
Occurs when the memory capacity of the PC for compiling is insufficient.
 
<Workaround>
・If you have multiple windows open, close them.
・If there is software running in the background, stop it.
・If the recommended memory capacity of the target device is not satisfied, add memory.
For device-specific memory recommendations, run the Release Notes from the Quartus II Help menu.
You can check with (up to Ver8.x).
 
If your PC's physical memory is full and it still occurs, it could be an OS limitation.
In that case, please refer to the following URL link of Altera Corporation for related information and how to change the settings.
 
  https://www.altera.co.jp/support/support-resources/software/quartus2/memory.html?GSA_pos=3&WT.oss_r=1&WT.oss=out%2520of%2520memory