[DiSL-user] JVM Crash

Wed May 3 10:55:48 CEST 2017

Hi Alessio,

what we usually do when we manage to crash the JVM is introduce global
exclusion list to limit the scope of instrumentation. For example, in ShadowVM
version of Elephant Tracks, I had an initial exclusion like this:

java.lang.*.*
java.security.*.*
java.util.*.*

sun.misc.*.*
sun.reflect.*.*

Later I refined refined java.lang.*.* into

java.lang.Class.*
java.lang.ClassLoader.*
java.lang.Thread.*
java.lang.String.*
java.lang.System.*

java.lang.instrument.*.*
java.lang.invoke.*.*
java.lang.reflect.*.*
java.lang.ref.*.*

I didn't get to the other packages yet :-)

The global exclusion list is given to DiSL as a path to file using the
disl.exclusionList property.

BTW, when running ShadowVM (or plain DiSL, for that matter) I rarely use the
disl.py launcher. Instead, I'm using a script like the one I'm attaching,
because I want to be able to see what is going on.

It may use some of the features from the trunk version to determine when a
server has started. Also, it will not be directly portable to Mac, because it
for example expects dynamic libraries to have an .so suffix, but you'll get
the idea and maybe adapt it your needs.

You will also notice that I'm running the instrumentation without bypass,
which requires that the instrumentation does not really use the class library
and only calls into the REDispatch class.

Best regards,
Lubomir

On 03/05/17 08:07, Alessio Gambi wrote:
> Hi Lubomir,
> 
> I split the original message in two threads
> 
>> Regarding the crash -- are you instrumenting anything in the Java Class Library?
> 
> Yes, I do. 
> 
> Is there any additional thing that I can try out on my own to rule out possible root causes?
> 
> For example, some deeper logging/debugging, disabling/enabling DiSL options, 
> restrict the scope of instrumentation, define additional guards, etc…
> 
> Best
> 
> — Alessio
> 
> 
>>
>> But let's fix the compilation problem first and re-test on the trunk.
>>
>>
>> Best regards,
>> Lubomir
>>
>>
>>
>> On 28/04/17 20:43, Alessio Gambi wrote:
>>> Dear DiSL-ers,
>>>
>>> while running a basic dynamic analysis using DiSL 2.1 (release) I stumbled upon a bad crash of the JVM [1].
>>>
>>> I have no idea of what’s going on, and how I can fix this.
>>>
>>> I can share the logs/dumps upon request.
>>>
>>> Here my configuration [2] (taken from the  hs_err_pid file)
>>>
>>> Any help is appreciated …
>>>
>>> Thanks
>>>
>>> Best
>>>
>>> — Alessio
>>>
>>> PS: I tried but failed to build DiSL from trunk ...
>>> 	
>>>
>>> [1]
>>>     [exec] # A fatal error has been detected by the Java Runtime Environment:
>>>     [exec] #
>>>     [exec] #  SIGSEGV (0xb) at pc=0x00007fff9af03f72, pid=1802, tid=4867
>>>     [exec] #
>>>     [exec] # JRE version: Java(TM) SE Runtime Environment (8.0_60-b27) (build 1.8.0_60-b27)
>>>     [exec] # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.60-b23 mixed mode bsd-amd64 compressed oops)
>>>     [exec] # Problematic frame:
>>>     [exec] # C  [libsystem_c.dylib+0xf72]  strlen+0x12
>>>     [exec] #
>>>     [exec] # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
>>>     [exec] #
>>>
>>>
>>> [2] ---------------  S Y S T E M  ---------------
>>>
>>> OS:Bsduname:Darwin 14.5.0 Darwin Kernel Version 14.5.0: Fri Feb 17 10:33:20 PST 2017; root:xnu-2782.50.9.1.1~1/RELEASE_X86_64 x86_64
>>> rlimit: STACK 8192k, CORE 0k, NPROC 709, NOFILE 10240, AS infinity
>>> load average:1.38 1.22 0.68
>>>
>>> CPU:total 8 (4 cores per cpu, 2 threads per core) family 6 model 70 stepping 1, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, lzcnt, ht, tsc, tscinvbit, bmi1, bmi2
>>>
>>> Memory: 4k page, physical 16777216k(3479728k free)
>>>
>>> /proc/meminfo:
>>>
>>>
>>> vm_info: Java HotSpot(TM) 64-Bit Server VM (25.60-b23) for bsd-amd64 JRE (1.8.0_60-b27), built on Aug  4 2015 10:47:24 by "java_re" with gcc 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)
>>>
>>> time: Fri Apr 28 20:40:58 2017
>>> elapsed time: 2 seconds (0d 0h 0m 2s)
>>>
-------------- next part --------------
#!/bin/bash

stop_server () {
	if [ -f "$1" ]; then
		local PID=$(< "$1")

		[ -n "$PID" ] && {
			kill -TERM $PID 2> /dev/null
			sleep 1
			[ -d "/proc/$PID" ] && kill -KILL $PID 2> /dev/null
		}

		rm -f "$1"
	fi
}

await_server () {
	local TIMEOUT=3
	while [ $TIMEOUT -gt 0 ]; do
		if [ ! -d "/proc/$1" ] ; then
			echo " died"
			return 1
		fi

		if [ ! -e "$2" ] ; then
			echo " done"
			return 0
		fi

		sleep 1
		echo -n "."

		TIMEOUT=$[TIMEOUT - 1]
	done
}

relative_path () {
	local COMMON TARGET UP

	COMMON=${1%/} TARGET=${2%/}/
	while [ "${TARGET#"$COMMON"/}" = "$TARGET" ]; do
		COMMON=${COMMON%/*} UP=../$UP
	done

	TARGET=$UP${TARGET#"$COMMON"/}
	TARGET=${TARGET%/}
	printf %s "${TARGET:-.}"
}

require_file () {
	[ -f "$1" ] || {
		echo "error: $1 is missing"
		exit 1
	}

	eval $2=$(relative_path $PWD $1)
}

###############################################################################

BASE_DIR=$(dirname $0)/..
[ -f "$BASE_DIR/.etrc" ] && source $BASE_DIR/.etrc

ET_LIB_DIR=${ET_HOME:-$BASE_DIR}/lib
DISL_LIB_DIR=${DISL_HOME:-$BASE_DIR}/lib
JAVA_CMD=${JAVA_HOME:+$JAVA_HOME/jre/bin/}java

[ -n "$DEBUG" ] && $JAVA_CMD -version

if [ $# -lt 1 ]; then
	echo "Usage: `basename $0` java-params"
	exit
fi

#
# Make sure we have everything we need
#

require_file $ET_LIB_DIR/et-remote.jar ET_REMOTE
require_file $ET_LIB_DIR/et-local.jar ET_LOCAL
require_file $ET_LIB_DIR/et-inst.jar ET_INST

require_file $DISL_LIB_DIR/libdislagent.so DISL_AGENT
require_file $DISL_LIB_DIR/disl-bypass.jar DISL_BYPASS
require_file $DISL_LIB_DIR/disl-server.jar DISL_SERVER

require_file $DISL_LIB_DIR/libdislreagent.so SHVM_AGENT
require_file $DISL_LIB_DIR/dislre-dispatch.jar SHVM_DISPATCH
require_file $DISL_LIB_DIR/dislre-server.jar SHVM_SERVER

#
# Prepare server status and PID files
#

TEMP_DIR=$(mktemp -d) || {
	echo "error: failed to create temporary directory"
	exit 1
}

SERVER_PID_FILE=$TEMP_DIR/server.pid
SERVER_STATUS_FILE=$TEMP_DIR/server.status
SHADOW_PID_FILE=$TEMP_DIR/shadow.pid
SHADOW_STATUS_FILE=$TEMP_DIR/shadow.status

XTRACE_STATUS=$(set +o|grep xtrace)

#
# Start DiSL server and wait for it to initialize
#

EXCLUDE_LIST=$BASE_DIR/exclude.lst
[ -f "$EXCLUDE_LIST" ] && EXCLUDE_OPT="-Ddisl.exclusionList=$EXCLUDE_LIST"

echo -n "Starting DiSL server... "
touch $SERVER_STATUS_FILE

[ -n "$DEBUG" ] && set -o xtrace

$JAVA_CMD \
	-cp $DISL_SERVER:$ET_INST \
	-Dserver.pid.file=$SERVER_PID_FILE \
	-Dserver.status.file=$SERVER_STATUS_FILE \
	-Ddisl.noexcepthandler=true \
	-Ddislserver.disablebypass=true \
	-Ddislserver.instrumented=$BASE_DIR/output/disl/inst \
	-Ddislserver.uninstrumented=$BASE_DIR/output/disl/orig \
	${DEBUG_SERVER:+-Ddebug=true} \
	$EXCLUDE_OPT \
	ch.usi.dag.dislserver.DiSLServer \
	2> server.err 1> server.out &

$XTRACE_STATUS

SERVER_PID=$!
echo $SERVER_PID > $SERVER_PID_FILE

await_server $SERVER_PID $SERVER_STATUS_FILE || {
	echo "error: failed to start DiSL server"
	rm -rf $TEMP_DIR
	exit 1
}

#
# Start Shadow VM server and wait for it to initialize
#

echo -n "Starting Shadow VM server... "
touch $SHADOW_STATUS_FILE

[ -n "$DEBUG" ] && set -o xtrace

$JAVA_CMD \
	-Xms1G -Xmx2G \
	-cp $SHVM_SERVER:$ET_REMOTE \
	-Dserver.pid.file=$SHADOW_PID_FILE \
	-Dserver.status.file=$SHADOW_STATUS_FILE \
	${DEBUG_SHADOW:+-Ddebug=true} \
	ch.usi.dag.dislreserver.DiSLREServer \
	2> shadow.err 1> shadow.out &

$XTRACE_STATUS

SHADOW_PID=$!
echo $SHADOW_PID > $SHADOW_PID_FILE

await_server $SHADOW_PID $SHADOW_STATUS_FILE || {
	echo "error: failed to start Shadow VM server"
	stop_server $SERVER_PID_FILE
	rm -rf $TEMP_DIR
	exit 1
}

#
# Start the client VM.
#

echo "Running client JVM..."

[ -n "$DEBUG" ] && set -o xtrace

$JAVA_CMD \
	-agentpath:${DISL_AGENT} -agentpath:${SHVM_AGENT} \
	-Xbootclasspath/a:$DISL_BYPASS:$SHVM_DISPATCH:$ET_LOCAL \
	-Ddisl.bypass=disabled \
	${DEBUG_CLIENT:+-Ddebug=true} \
	"$@"

$XTRACE_STATUS

#
# Stop the DiSL and Shadow VM servers
#

echo "Stopping servers and cleaning up..."

stop_server $SERVER_PID_FILE
stop_server $SHADOW_PID_FILE

#
# Delete empty output files.
#
for f in server.{out,err} shadow.{out,err}; do
	[ ! -s $f ] && rm $f
done

rm -rf $TEMP_DIR