The JVM is by design an insecure environment and it is generally difficult to run untrusted code in a sandboxed environment.
However, it seems that is relatively easy to sandbox JavaScript code running in Oracle Nashorn. The instructions are here. Strangely, this was not easy to find through a Google search.
Below I have listed some further sources on Sandboxing JavaScript and Java code. Although there is plenty of material on Rhino, I would not recommend using this engine. I think Nashorn has been designed with support for Sandboxed code in mind from the very beginning while in Rhino the functionality feels kind of bolted on.
UPDATE I have implemented two little libraries which takes care of the grunt work of sandboxing Nashorn and Rhino code in Java:
Nashorn Sandbox (on GitHub)
Rhino Sandbox (on Github)
Sandboxing JavaScript
Nashorn
Restricting Script Access to Specified Java Classes: From the Oracle Nashorn docs. Shows how to restrict access to specific Java classes.
Rhino
Class ContextFactory: Useful for monitoring and setting restrictions on Rhino code.
Method initSafeStandardObjects: Useful for creating sandboxed Rhino code.
Rhino Sandbox: A small library for sandboxing JavaScript code running in Rhino.
Sandboxing Rhino in Java: Blog post
Securing Rhino in Java6: Blog post
DynJS
Sandboxing JavaScript Execution in Java: Blog post
Sandboxing Java
Example Code Monitoring Threads: Example code how thread CPU usage can be monitored.
The Java Sandbox: A library for sandboxing any Java code. Might be useful to sandbox the Java code with runs the script.
The Full Stack Blog 
Unfortunately stopping a thread with Thread.stop() is not effective against malicious scripts: JS can catch java.lang.ThreadDeath just by doing catch(e) and ignore it.
Thank you for pointing this out. I did a quick test for this but couldn’t escape the sandbox (since it also injects custom js code to interrupt the script).
TestScriptInterruptionAndCatch.xtend
I ended up using Rhino after all. I had to load a 1 MB JavaScript library and it took > 3 GB of memory on Nashorn but worked fine with Rhino. For the sandbox I developed for Rhino, I also added a test case with catch which passes:
TestCPUViolationAndCatch.xtend
Or is there a smart way around this? Thanks!
I was trying to achieve sandbox JS execution with both Nashorn and Rhino (but not using your code). Following script cannot be stopped using Thread#stop in my Nashorn sandbox:
while(true) {
try {
while(true) {
var j=0;
for(var i=0;i<1000000;i++) j++;
print('JS cont '+j);
}
} catch (e) { print('JS got exception:'+e); };
}
My not yet complete Rhino sandbox can terminate it by raising RuntimeException in my ContextFactory#observeInstructionCount and it works in both compiled and interpreted mode.
I have not yet done any memory measurements – your comment about memory consumption is helpful. It looks like I will go with Rhino.
Thanks for your reply and for sharing the example.
I also terminate scripts by raising a RuntimeException in ContextFactory#observeInstructionCount for the Rhino Sandbox. Please be welcome to try it out – it’s on GitHub and I’ll be happy to patch it if you find any issues.
I do use Thread.stop for the Nashorn Sandbox I implemented – but only as a measure of last resort. The usual means of quitting a script is a custom function I call after EVERY line of JavaScript code from the original script. This function then will raise a RuntimeException if the script is to be terminated. If this doesn’t work (it should!), I call Thread.stop.