IncQuery MoDisco Benchmark

Introduction

The benchmark presented on this page is provided by the developers of the Epsilon tool and it applies model queries to find class properties in existing Java classes. A class property is a variable with name X that has two corresponding methods in the same class, with name getX/setX or isX/setX.

class C{
 int X;
 
 int getX(){return X;}
 
 void isX(int newX){X = newX;}
}

The input models for the benchmark were created by the MoDisco tool from existing components of the Epsilon tool and the Eclipse framework. The Instance models used for the benchmark are publicly available.

Benchmark setup

The following pattern definitions were provided by the benchmark authors:

pattern nameOfElement(Element, Name) = {
 NamedElement(Element);
 NamedElement.name(EAttribute, Element, Name);
 EString(Name);
}
	
pattern classes(Element) = {
 ClassDeclaration(Element);
}	
	
pattern fieldDeclaration(Element) = {
 FieldDeclaration(Element);
}
	
pattern fieldAndFrag(A, B) = {
 FieldDeclaration(A);
 VariableDeclarationFragment(B);
 AbstractVariablesContainer.fragments(EReference, A, B);
}
	
pattern classWithBody(A, B) = {
 ClassDeclaration(A);
 BodyDeclaration(B);
 AbstractTypeDeclaration.bodyDeclarations(EReference, A, B);
}
	
pattern classWithVariables(Class, Variable) = {
 ClassDeclaration(Class);
 find classWithBody(Class, Body);
 find fieldAndFrag(Body, Variable);
}

pattern classWithMethods(Class, Method) = {
 ClassDeclaration(Class);
 MethodDeclaration(Method);
 AbstractTypeDeclaration.bodyDeclarations(EReference, Class, Method);
}

pattern matchedClass(Class, Variable) = {
 ClassDeclaration(Class);
 find classWithVariables(Class, Variable);
 find nameOfElement(Variable, VarName);
 find classWithMethods(Class, Method1);
 find classWithMethods(Class, Method2);
 find nameOfElement(Method1, Method1Name);
 find nameOfElement(Method2, Method2Name);
 check((toLower(value(Method1Name)) == "get" + toLower(value(VarName))) || 
  (toLower(value(Method1Name)) == "is" + toLower(value(VarName))));
 check(toLower(value(Method2Name)) == "set" + toLower(value(VarName)));
}

Results

We set up the benchmark using the provided java.ecore and the four EMF models with both EMF-IncQuery 0.4 and the current version of VIATRA2, based on the guidelines written in the Performance page.

EMF-IncQuery 0.4

The following table lists for each input model the size of the original model, the total memory footprint (see Performance). We have used several VM argument configurations ranging from a high of 2,5GB memory to the lowest of 700MB memory. For each configuration, we listed the time it takes for EMF to load the XMI file and the initialization time for EMF-IncQuery. Note that the collection of matches from the Rete network is not listed, but it was around 10-30 ms in each case.

Note, that we also measured the optimized code (by Gábor Bergmann) from the VIATRA2 solution to see how it affects performance.

VIATRA2

We measured the pattern matching off VIATRA2 by adding the @incremental annotation to the machine, so that it uses the Incremental Pattern matcher. We also removed the println() calls from the code, since writing to the output takes considerable time that should not be measured. For the results, we used the built-in time measurement of VIATRA that outputs the time of imports and transfromations on the system output.

rule main() = let Buf = getBuffer("core://out") in seq{
 forall C, A with find matchedClass(C, A) do
  println(Buf,name(C) + "." + name(A));
}

The following VM arguments were used for these measurements: -Xms200m -Xmx6000m -XX:MaxPermSize=3536M -XX:+UseParallelGC -XX:PermSize=1528M. Note that we have been able to load the largest model with these settings but we were unable to run the transformation.

Optimized VIATRA2

Next, we went on to modify the pattern definitions themselves to create the optimized, but equivalent versions based on our experience with the Rete network and the incremental pattern matcher.

pattern fieldAndFrag(A, B) = {
 FieldDeclaration(A);
 VariableDeclarationFragment(B);
 AbstractVariablesContainer.fragments(EReference, A, B);
}

pattern classWithBody(A, B) = {
 ClassDeclaration(A);
 BodyDeclaration(B);
 AbstractTypeDeclaration.bodyDeclarations(EReference, A, B);
}

pattern classWithVariableName(Class, VariableName) = {
 ClassDeclaration(Class);
 find classWithBody(Class, Body);
 find fieldAndFrag(Body, Variable);
 NamedElement.name(EAttribute, Variable, VariableName);
 EString(VariableName);
}

pattern classWithMethodName(Class, MethodName) = {
 ClassDeclaration(Class);
 MethodDeclaration(Method);
 AbstractTypeDeclaration.bodyDeclarations(EReference, Class, Method);
 NamedElement.name(EAttribute, Method, MethodName);
 EString(MethodName);
}

pattern matchedClass(Class, VarName) = {
 ClassDeclaration(Class);
 find classWithVariableName(Class, VarName);
 find classWithMethodName(Class, Method1Name);
 find classWithMethodName(Class, Method2Name);
 check(str.equalsIgnoreCase(value(Method1Name), "get" + value(VarName)) ||
  str.equalsIgnoreCase(value(Method1Name), "is" + value(VarName)));
 check(str.equalsIgnoreCase(value(Method2Name), "set" + value(VarName)));
}

Finally, we asked our fellow researcher Gábor Bergmann, the original developer of the incremental pattern matcher (and its underlying Rete implementation) to see if he can do a bit more optimization. These results are given in the last column. Note that the most noticable performance gain is from the decomposition of the main pattern to the two subpatterns for finding getters and setters and the main pattern connects these patterns.

pattern nameOfElement(Element, Name) = 
{
 NamedElement(Element);
 NamedElement.name(EAttribute, Element, Name);
 EString(Name);
}

pattern fieldAndFrag(A, B) =
{
 FieldDeclaration(A);
 VariableDeclarationFragment(B);
 AbstractVariablesContainer.fragments(EReference, A, B);
}

pattern classWithBody(A, B) =
{
 ClassDeclaration(A);
 BodyDeclaration(B);
 AbstractTypeDeclaration.bodyDeclarations(EReference, A, B);
}

pattern classWithVariables(Class, Variable) =
{
 ClassDeclaration(Class);
 find classWithBody(Class, Body);
 find fieldAndFrag(Body, Variable);
}

pattern classWithVarName(Class, Variable, VarName) =
{
 find classWithVariables(Class, Variable);
 find nameOfElement(Variable, VarName);
}

pattern classWithMethods(Class, Method) =
{
 ClassDeclaration(Class);
 MethodDeclaration(Method);
 AbstractTypeDeclaration.bodyDeclarations(EReference, Class, Method);
}

pattern classWithMethName(Class, Method, MethodName) =
{
 find classWithMethods(Class, Method);
 find nameOfElement(Method, MethodName);
}

pattern hasGetter(Class, Variable) = {
 find classWithVarName(Class, Variable, VarName);
 find classWithMethName(Class, Method, MethodName);
 check(str.equalsIgnoreCase(value(MethodName), "get" + value(VarName)) ||
  str.equalsIgnoreCase(value(MethodName), "is" + value(VarName)));		
}
pattern hasSetter(Class, Variable) = {
 find classWithVarName(Class, Variable, VarName);
 find classWithMethName(Class, Method, MethodName);
 check(str.equalsIgnoreCase(value(MethodName), "set" + value(VarName)));
}

pattern matchedClass(Class, Variable) =
{
 find hasGetter(Class, Variable);
 find hasSetter(Class, Variable);
}

The following VM arguments were used for all measurements Xmx6000,PermGen=3568.

Note that using the pattern definitions created by Gábor, we were able to finally run the transformation for the largest model as well!

EMF-IncQuery 0.6

The new generation of EMF-IncQuery uses a new Xtext-based query definition editor and provides a nice Query Explorer for testing queries dynamically at the same time as they are defined, to remove the development loop of code generation and using a runtime Eclipse.

package modisco

import "http://www.eclipse.org/MoDisco/Java/0.2.incubation/java"
import "http://www.eclipse.org/emf/2002/Ecore"

pattern fieldAndFrag(A, B)= {
 FieldDeclaration(A);
 VariableDeclarationFragment(B);
 AbstractVariablesContainer.fragments(A, B);
}

pattern classWithBody(A, B)= {
 ClassDeclaration(A);
 BodyDeclaration(B);
 AbstractTypeDeclaration.bodyDeclarations(A, B);
}

pattern classWithVariableName(Clss, VariableName)= {
 ClassDeclaration(Clss);
 find classWithBody(Clss, Body);
 find fieldAndFrag(Body, Variable);
 NamedElement.name(Variable, VariableName);
 EString(VariableName);
}

pattern classWithMethodName(Clss, MethodName)= {
 ClassDeclaration(Clss);
 MethodDeclaration(Method);
 AbstractTypeDeclaration.bodyDeclarations(Clss, Method);
 NamedElement.name(Method, MethodName);
 EString(MethodName);
}

pattern matchedClass(Clss, VarName)= {
 ClassDeclaration(Clss);
 find classWithVariableName(Clss, VarName);
 find classWithMethodName(Clss, Method1Name);
 find classWithMethodName(Clss, Method2Name);
 check(Method1Name.toString.equalsIgnoreCase("get".concat(VarName.toString)) ||
   Method1Name.toString.equalsIgnoreCase("is".concat(VarName.toString)));
 check(Method2Name.toString.equalsIgnoreCase("set".concat(VarName.toString)));
}

In the current version, the above definition can only be checked in the Query Explorer due to a minor bug caused by Xtext and Xbase type providers clashing. The results of the measurements with the same settings as for EMF-IncQuery 0.4 will be posted soon!