Program Query Evaluation

Authors: Zoltán Ujhelyi, Ákos Horváth, Dániel Varró, Norbert István Csiszár, Gábor Szőke, László Vidács, Rudolf Ferenc

In the CSMR-WCRE 2014 paper "Anti-pattern Detection with Model Queries: A Comparison of Approaches" we compare three different approaches for evaluating program queries. This page details all measurements done for the paper.

 

Model sizes

The approaches were evaluated on a test set of 30 modules of 17 open source projects. The projects are sized between 1kLOC and 500kLOC, are are used in various scenarios, thus sharing different characteristics. The list of projects include the ArgoUML UML editor, JTransforms, a Fourier transformation library, the SVNKit Subversion client, the online homework system WeBWorK, the Weka data mining software and many more.

The next table describes the sizes of these projects in lines of code together with the size of the ASG and EMF models in terms of nodes and edges.
 

Project Module LOC ASG EMF
Nodes Edges Nodes Edges
argouml argouml.jar

307069

700459

1468495

711121

1479044

argouml-diagrams-sequence.jar

3683

44684

85650

46964

87928

argouml-mdr.jar

22476

115018

234585

118637

238151

argouml-model.jar

13834

19903

35557

20623

36277

javadjvu

djvuframe.jar

1398

32909

60067

35459

62616

javadjvu.jar

36223

124185

250372

127520

253696

jml

jml-1.0b3.jar

17182

70403

146020

72754

148365

jsr80

jsr80.jar

4251

11288

21636

12172

22520

jtransforms

jtransforms

43118

294067

651947

294910

652790

log4j

log4j-1.2.15.jar

31428

135444

270076

140760

275392

physhun

physhun.jar

7368

40974

85412

42773

87073

projectlibre

openproj-contrib-tmp.jar

2624

29275

55640

31563

57925

openproj-reports-tmp.jar

1241

24753

46092

26986

48325

proteinshader

ProteinShader.jar

50775

158358

309201

164121

314956

qwicap-guess

guess. war

525

10302

19686

11290

20674

robocode

cachecleaner.jar

73

2667

4891

2956

5180

extract.jar

40624

195413

399603

201409

405284

robocode-setup-1.5.4.jar

41027

197959

406191

203975

411517

robocode.jar

37092

180656

367825

186537

373698

roborumble.jar

2347

19128

38366

20047

39284

sdedit

sdedit.jar

22710

145380

291375

151837

297823

stendhal

stendhal-maps-0.75.1.jar

33345

185450

407602

188189

410338

stendhal-server-0.75.1.jar

52059

216194

458040

221627

463465

svnkit

svnkit-cli.jar

13524

108188

232597

114206

238588

svnkit-javahl.jar

34799

63101

120762

67744

125284

svnkit.jar

121276

554165

1225837

566166

1237308

org.tmatesoft.svnkit-1.3.0-bundle.jar

130298

566401

1250727

578493

1262167

webwork

webwork

48782

217616

446727

224544

453641

weka

weka

468018

1529023

3286541

1542205

3299530

xins

logdoc.jar

570

4815

8916

5173

9274

Measurement Description

All measurements were executed on a dedicated Linux-based server with 6 GB RAM running Java 6. On the server the Java ASG of the Columbus Framework was installed together with the EMF-INCQUERY model query framework.

All program queries were implemented as both visitors for the ASG (by Columbus experts) and as graph patterns (by EMF-INCQUERY experts). The visitor implementation was executed on both model representations, while the graph patterns were used to initialize local search and Rete network based pattern matchers for the EMF representation. In all cases, the time to load the model from its serialized form and the time to execute the program query was measured together with the maximum heap size usage.

Every program query was executed ten times, and the time and memory results were averaged. In order to minimize the interference between different runs, for each execution a new JVM was created and each query was run in isolation. The time to start up and shut down the JVM were not included in the measurement results.

Memory and Search Time Results

    Load Time (s) Search Time (s) Maximum Heap Size (MB)
Model Case ASG EMF LS IQ ASG EMF LS IQ ASG EMF LS IQ
argouml catch 5.34 14.41 19.68 35.17 2.09 1.33 0.02 0.00 116.93 317.56 363.78 1571.18
concatenate 5.33 14.58 19.72 35.20 2.04 1.30 0.67 0.00 116.93 317.56 449.84 1571.05
constantcompare 5.33 14.62 19.67 35.20 2.12 1.32 0.12 0.00 116.93 317.56 363.78 1571.13
nodefaultswitch 5.33 14.54 19.67 35.26 2.02 1.30 0.01 0.00 116.93 317.56 362.29 1571.16
stringcompare 5.34 14.37 19.69 35.22 2.03 1.18 0.14 0.00 116.94 317.56 372.35 1571.05
unusedparameter 5.32 14.58 19.70 35.27 2.28 1.48 0.20 0.00 116.93 323.49 369.74 1571.07
argouml-
diagrams-
sequence
catch 0.70 2.35 4.77 5.48 0.85 0.38 0.01 0.00 20.48 44.82 49.98 118.21
concatenate 0.70 2.35 4.81 5.49 0.85 0.38 0.18 0.00 20.54 44.82 49.85 118.21
constantcompare 0.70 2.34 4.78 5.51 0.86 0.38 0.03 0.00 20.54 45.26 50.40 118.21
nodefaultswitch 0.70 2.35 4.73 5.48 0.85 0.38 0.01 0.00 20.52 44.82 49.98 118.21
stringcompare 0.71 2.36 4.80 5.48 0.72 0.36 0.02 0.00 20.33 45.25 50.38 118.21
unusedparameter 0.70 2.34 4.77 5.47 1.05 0.45 0.05 0.00 21.26 41.06 50.56 118.21
argouml-mdr catch 1.30 3.45 5.77 8.76 0.75 0.59 0.02 0.00 24.91 53.78 80.65 238.01
concatenate 1.29 3.38 5.78 8.75 0.73 0.56 0.27 0.00 24.62 53.78 80.38 238.02
constantcompare 1.29 3.35 5.77 8.73 0.74 0.57 0.03 0.00 24.61 53.78 81.19 238.01
nodefaultswitch 1.30 3.45 5.80 8.73 0.73 0.56 0.01 0.00 24.91 53.78 80.65 238.02
stringcompare 1.30 3.44 5.78 8.75 0.70 0.56 0.05 0.00 24.62 53.39 82.26 238.01
unusedparameter 1.30 3.34 5.78 8.77 0.82 0.61 0.07 0.00 26.18 54.86 82.29 238.01
argouml-model catch 0.40 2.13 3.66 4.15 0.29 0.18 0.01 0.00 24.08 26.71 38.79 64.56
concatenate 0.40 2.14 3.64 4.15 0.29 0.18 0.08 0.00 24.08 26.71 40.92 64.56
constantcompare 0.40 2.16 3.68 4.15 0.29 0.18 0.02 0.00 24.09 26.71 38.79 64.55
nodefaultswitch 0.40 2.13 3.65 4.16 0.29 0.18 0.01 0.00 24.08 26.71 38.79 64.56
stringcompare 0.40 2.14 3.66 4.15 0.25 0.15 0.01 0.00 24.16 26.72 38.79 64.56
unusedparameter 0.40 2.14 3.63 4.16 0.35 0.20 0.02 0.00 24.40 27.23 39.32 64.56
cachecleaner catch 0.12 0.65 1.03 1.46 0.03 0.04 0.01 0.01 9.18 13.24 19.23 24.39
concatenate 0.12 0.65 1.03 1.46 0.03 0.04 0.04 0.01 9.56 13.24 19.73 24.50
constantcompare 0.12 0.65 1.03 1.46 0.03 0.04 0.02 0.00 9.18 13.24 19.73 24.50
nodefaultswitch 0.12 0.65 1.03 1.46 0.03 0.04 0.01 0.00 9.18 13.24 19.23 24.50
stringcompare 0.12 0.65 1.03 1.46 0.02 0.03 0.02 0.00 9.18 13.26 19.73 24.39
unusedparameter 0.12 0.65 1.03 1.46 0.03 0.04 0.02 0.01 9.56 13.74 19.73 24.44
djvuframe catch 0.56 2.06 3.81 4.95 0.69 0.28 0.01 0.00 31.90 19.98 35.77 82.32
concatenate 0.56 2.05 3.80 4.94 0.69 0.28 0.11 0.00 31.90 19.98 38.97 82.32
constantcompare 0.56 2.05 3.81 4.94 0.69 0.28 0.04 0.00 31.90 19.98 36.31 82.32
nodefaultswitch 0.56 2.05 3.81 4.94 0.69 0.28 0.01 0.00 31.90 19.98 35.77 82.32
stringcompare 0.56 2.06 3.80 4.96 0.60 0.26 0.02 0.00 31.91 19.76 36.31 82.32
unusedparameter 0.56 2.05 3.78 4.91 0.81 0.31 0.04 0.00 31.90 20.80 36.31 82.32
extract catch 2.10 4.78 7.67 12.49 1.36 0.71 0.02 0.01 36.39 96.04 143.87 373.13
concatenate 2.10 4.73 7.63 12.49 1.28 0.70 0.38 0.01 36.22 96.04 143.87 373.13
constantcompare 2.09 4.73 7.66 12.60 1.43 0.71 0.13 0.01 36.54 96.04 143.86 373.13
nodefaultswitch 2.11 4.73 7.64 12.52 1.32 0.70 0.02 0.01 36.41 96.04 143.86 373.13
stringcompare 2.11 4.74 7.63 12.49 1.52 0.63 0.13 0.00 36.61 96.05 143.86 373.13
unusedparameter 2.08 4.75 7.65 12.50 1.41 0.78 0.15 0.01 38.68 98.68 143.86 373.13
guess catch 0.25 1.51 2.56 3.10 0.15 0.16 0.03 0.01 15.62 26.77 22.14 31.58
concatenate 0.25 1.52 2.55 3.10 0.15 0.16 0.11 0.01 15.63 26.77 23.67 31.56
constantcompare 0.25 1.53 2.56 3.09 0.16 0.16 0.04 0.01 15.63 26.77 22.14 31.58
nodefaultswitch 0.25 1.53 2.55 3.09 0.15 0.16 0.02 0.01 15.63 26.77 22.14 31.58
stringcompare 0.25 1.52 2.55 3.08 0.13 0.15 0.05 0.01 16.15 26.60 22.14 31.58
unusedparameter 0.25 1.52 2.54 3.09 0.18 0.17 0.05 0.01 16.11 26.52 22.14 31.56
javadjvu catch 1.43 3.64 5.83 9.78 1.37 0.54 0.03 0.01 34.44 73.75 70.21 281.04
concatenate 1.43 3.60 5.84 9.75 1.36 0.54 0.30 0.01 34.41 73.75 84.89 280.71
constantcompare 1.42 3.57 5.87 9.81 1.39 0.55 0.06 0.01 34.19 73.75 70.41 280.69
nodefaultswitch 1.43 3.57 5.83 9.78 1.36 0.55 0.03 0.01 34.21 73.75 70.47 280.73
stringcompare 1.43 3.62 5.84 9.70 1.36 0.52 0.06 0.01 34.33 65.65 71.75 280.69
unusedparameter 1.43 3.61 5.85 9.79 1.45 0.59 0.10 0.01 35.22 65.64 71.55 280.69
jml-1.0b3 catch 0.87 2.72 4.73 6.93 1.11 0.45 0.01 0.00 36.18 46.66 49.35 134.47
concatenate 0.87 2.71 4.73 6.94 1.10 0.45 0.21 0.00 36.18 46.65 57.59 134.47
constantcompare 0.87 2.70 4.73 6.95 1.13 0.46 0.03 0.00 36.13 46.66 49.35 134.47
nodefaultswitch 0.87 2.71 4.71 6.94 1.11 0.44 0.01 0.00 36.05 46.66 49.35 134.47
stringcompare 0.88 2.71 4.72 6.94 1.06 0.40 0.04 0.00 35.89 46.66 50.43 134.47
unusedparameter 0.87 2.72 4.71 6.92 1.23 0.48 0.05 0.00 36.18 47.77 50.43 134.47
jsr80 catch 0.27 1.59 2.56 3.29 0.17 0.22 0.02 0.00 16.53 28.60 24.07 36.72
concatenate 0.27 1.59 2.56 3.28 0.17 0.21 0.12 0.01 16.52 28.61 25.61 36.72
constantcompare 0.27 1.59 2.56 3.28 0.17 0.22 0.04 0.01 16.52 28.61 24.07 36.71
nodefaultswitch 0.27 1.59 2.56 3.28 0.17 0.22 0.03 0.01 16.52 28.61 24.07 36.74
stringcompare 0.27 1.60 2.56 3.28 0.15 0.18 0.04 0.01 16.97 28.61 24.07 36.73
unusedparameter 0.27 1.59 2.56 3.28 0.20 0.24 0.05 0.01 17.03 29.10 24.59 36.74
jtransforms-1.7 catch 2.14 5.01 7.95 16.63 1.07 0.56 0.03 0.00 50.16 131.76 173.83 668.14
concatenate 2.14 5.01 7.95 16.51 1.08 0.57 0.51 0.00 50.16 131.76 173.83 668.19
constantcompare 2.14 5.01 7.98 16.60 1.09 0.56 0.03 0.00 50.16 131.76 173.83 668.18
nodefaultswitch 2.14 5.02 7.97 16.52 1.06 0.56 0.02 0.00 50.16 131.76 173.83 668.20
stringcompare 2.15 5.03 7.97 16.51 0.91 0.50 0.07 0.00 50.18 131.77 173.83 668.19
unusedparameter 2.14 5.02 7.95 16.52 1.19 0.58 0.09 0.00 50.16 131.76 173.83 668.19
log4j-1.2.15 catch 1.62 3.84 6.52 10.52 1.32 0.61 0.03 0.01 44.54 69.77 80.79 268.51
concatenate 1.61 3.82 6.56 10.50 1.29 0.58 0.33 0.01 44.54 69.77 80.06 268.51
constantcompare 1.61 3.80 6.54 10.50 1.31 0.58 0.10 0.01 44.50 69.77 81.32 268.52
nodefaultswitch 1.61 3.82 6.56 10.51 1.25 0.58 0.02 0.01 44.53 69.77 80.70 268.51
stringcompare 1.62 3.81 6.54 10.52 1.52 0.53 0.13 0.01 44.51 69.77 83.00 268.51
unusedparameter 1.62 3.82 6.54 10.51 1.48 0.68 0.17 0.01 39.29 71.90 82.90 268.51
logdoc catch 0.14 1.00 1.53 2.12 0.05 0.05 0.01 0.01 11.01 17.09 26.71 18.63
concatenate 0.13 1.00 1.53 2.13 0.05 0.05 0.04 0.01 10.97 17.08 27.45 18.63
constantcompare 0.13 1.00 1.53 2.13 0.05 0.05 0.02 0.00 10.97 17.08 26.71 18.63
nodefaultswitch 0.13 1.00 1.53 2.12 0.05 0.05 0.01 0.00 10.97 17.09 26.71 18.63
stringcompare 0.14 1.01 1.53 2.12 0.05 0.04 0.02 0.00 11.08 17.11 26.71 18.63
unusedparameter 0.13 1.00 1.53 2.12 0.06 0.05 0.02 0.01 11.48 17.57 26.71 18.63
openproj-
contrib
catch 0.48 2.05 3.30 4.32 0.47 0.25 0.01 0.01 30.53 37.28 38.24 72.94
concatenate 0.48 2.04 3.31 4.33 0.47 0.24 0.13 0.01 30.62 37.28 41.31 72.94
constantcompare 0.48 2.04 3.34 4.31 0.48 0.24 0.04 0.00 30.62 37.28 38.64 72.94
nodefaultswitch 0.48 2.04 3.33 4.31 0.47 0.26 0.02 0.00 30.54 37.28 38.31 72.94
stringcompare 0.49 2.06 3.34 4.33 0.38 0.23 0.04 0.00 30.59 37.35 38.64 72.94
unusedparameter 0.48 2.05 3.33 4.32 0.56 0.26 0.05 0.01 31.10 37.70 38.64 72.94
openproj-
reports
catch 0.45 1.93 3.21 4.16 0.40 0.22 0.01 0.00 27.03 29.92 26.14 65.95
concatenate 0.44 1.93 3.20 4.15 0.40 0.22 0.09 0.00 27.04 29.92 28.79 65.95
constantcompare 0.45 1.93 3.20 4.16 0.40 0.22 0.03 0.00 27.04 29.92 26.67 65.95
nodefaultswitch 0.44 1.94 3.20 4.16 0.40 0.22 0.01 0.00 27.04 29.92 26.14 65.95
stringcompare 0.45 1.94 3.20 4.14 0.34 0.20 0.02 0.00 27.15 29.95 26.67 65.95
unusedparameter 0.45 1.93 3.20 4.14 0.46 0.24 0.03 0.00 27.54 30.44 26.67 65.95
org.tmatesoft.svnkit-
1.3.0-
bundle
catch 4.45 10.07 15.20 30.23 1.21 1.01 0.02 0.00 91.90 276.20 297.22 1264.26
concatenate 4.48 10.38 15.24 30.13 1.17 1.00 0.66 0.00 91.90 276.30 376.63 1263.94
constantcompare 4.47 10.20 15.27 30.08 1.24 1.02 0.07 0.00 92.79 276.26 297.25 1263.69
nodefaultswitch 4.45 10.25 15.19 30.17 1.16 1.00 0.01 0.00 91.90 276.20 297.22 1264.27
stringcompare 4.47 10.15 15.21 30.25 1.09 0.91 0.15 0.00 91.90 276.23 310.00 1264.27
unusedparameter 4.48 10.23 15.39 30.33 1.31 1.11 0.22 0.00 94.57 277.91 304.95 1263.70

physhun

catch 0.56 2.21 3.92 5.06 0.49 0.29 0.01 0.00 16.29 38.68 36.38 110.42
concatenate 0.56 2.20 3.93 5.07 0.49 0.29 0.16 0.00 16.29 38.68 41.46 110.42
constantcompare 0.56 2.21 3.95 5.07 0.50 0.29 0.03 0.00 16.29 38.68 36.38 110.42
nodefaultswitch 0.56 2.20 3.95 5.07 0.49 0.29 0.01 0.00 16.29 38.68 36.38 110.42
stringcompare 0.56 2.22 3.94 5.10 0.44 0.27 0.04 0.00 16.33 39.14 37.45 110.42
unusedparameter 0.56 2.21 3.94 5.09 0.55 0.32 0.04 0.00 16.80 39.20 36.91 110.42
ProteinShader catch 1.85 4.41 7.14 11.46 0.91 0.62 0.03 0.01 44.01 86.96 108.40 317.25
concatenate 1.84 4.42 7.14 11.48 0.92 0.63 0.33 0.01 44.10 86.96 107.67 317.21
constantcompare 1.84 4.43 7.17 11.48 0.93 0.64 0.08 0.01 44.08 86.96 108.50 317.25
nodefaultswitch 1.84 4.43 7.16 11.49 0.92 0.63 0.03 0.01 44.15 86.96 108.30 317.18
stringcompare 1.85 4.39 7.17 11.51 0.91 0.59 0.09 0.01 44.14 86.97 110.66 317.19
unusedparameter 1.85 4.45 7.15 11.46 1.03 0.71 0.12 0.01 46.20 86.96 110.14 317.23
robocode catch 2.04 4.83 7.55 12.48 0.96 0.71 0.02 0.01 49.05 60.39 100.76 382.80
concatenate 2.04 4.84 7.60 12.41 0.92 0.67 0.31 0.01 49.05 60.34 121.55 382.82
constantcompare 2.04 4.84 7.55 12.39 0.99 0.68 0.10 0.01 49.05 60.39 101.70 382.78
nodefaultswitch 2.04 4.84 7.56 12.43 0.93 0.68 0.02 0.01 49.05 60.39 100.76 382.82
stringcompare 2.05 4.83 7.53 12.39 0.86 0.61 0.10 0.00 49.23 60.29 103.49 382.80
unusedparameter 2.05 4.84 7.53 12.44 1.06 0.75 0.12 0.01 49.05 62.68 102.59 382.84
robocode-
setup-
1.5.4
catch 2.22 4.93 7.91 14.41 1.12 0.71 0.02 0.01 39.26 104.07 107.91 396.22
concatenate 2.21 5.03 7.89 14.43 1.11 0.70 0.33 0.01 39.26 104.07 130.06 396.31
constantcompare 2.20 5.00 7.96 14.43 1.17 0.71 0.09 0.01 39.26 104.08 108.25 396.35
nodefaultswitch 2.21 4.94 7.93 14.49 1.10 0.71 0.02 0.01 39.26 104.07 107.35 396.22
stringcompare 2.21 4.95 7.86 14.45 1.05 0.63 0.12 0.01 39.26 104.09 110.94 396.18
unusedparameter 2.21 4.95 7.90 14.46 1.23 0.78 0.12 0.00 39.26 106.69 110.05 396.35
roborumble catch 0.36 1.78 2.99 3.64 0.29 0.19 0.02 0.01 22.02 20.96 20.16 49.18
concatenate 0.36 1.76 3.01 3.64 0.29 0.20 0.13 0.01 22.03 20.96 22.78 49.18
constantcompare 0.36 1.76 3.00 3.64 0.30 0.20 0.03 0.01 22.04 20.92 20.16 49.17
nodefaultswitch 0.36 1.76 3.01 3.62 0.29 0.19 0.02 0.01 22.15 20.96 20.16 49.18
stringcompare 0.37 1.77 3.01 3.63 0.24 0.16 0.05 0.00 22.14 20.93 20.68 49.18
unusedparameter 0.36 1.76 3.00 3.64 0.33 0.21 0.03 0.00 22.44 21.45 20.16 49.17
sdedit catch 1.68 3.91 6.70 10.94 1.46 0.62 0.02 0.00 46.06 81.55 88.95 330.14
concatenate 1.68 3.93 6.69 10.83 1.46 0.62 0.37 0.01 46.06 81.55 88.28 331.29
constantcompare 1.68 3.92 6.68 10.84 1.54 0.62 0.08 0.01 46.06 81.55 89.62 331.16
nodefaultswitch 1.68 3.91 6.70 10.81 1.47 0.62 0.02 0.01 46.06 81.55 88.47 331.29
stringcompare 1.69 3.92 6.71 10.85 1.59 0.56 0.08 0.01 46.06 81.56 91.15 331.00
unusedparameter 1.67 3.92 6.72 10.80 1.60 0.73 0.12 0.01 46.06 81.55 90.57 331.30
stendhal-
maps-
0.75.1
catch 1.87 4.72 7.37 12.01 0.75 0.61 0.01 0.00 34.48 106.20 122.12 410.93
concatenate 1.87 4.71 7.41 12.09 0.77 0.61 0.36 0.00 34.66 106.14 146.34 410.93
constantcompare 1.86 4.72 7.38 12.02 0.82 0.62 0.03 0.00 34.61 106.20 123.11 410.90
nodefaultswitch 1.88 4.71 7.38 12.02 0.74 0.61 0.01 0.00 34.61 106.20 122.13 410.93
stringcompare 1.87 4.70 7.39 12.00 0.82 0.55 0.10 0.00 28.26 106.14 127.08 410.93
unusedparameter 1.87 4.75 7.40 12.05 0.85 0.68 0.10 0.00 35.67 93.48 125.13 410.93
stendhal-
server-
0.75.1
catch 2.11 5.12 7.87 13.17 1.86 0.72 0.02 0.01 48.70 114.59 122.47 491.27
concatenate 2.10 5.14 7.94 13.21 1.84 0.72 0.38 0.01 48.70 114.59 148.96 491.57
constantcompare 2.11 5.13 7.85 13.16 1.91 0.74 0.08 0.01 48.70 114.59 123.44 491.61
nodefaultswitch 2.10 5.13 7.94 13.16 1.87 0.72 0.02 0.01 48.70 114.59 122.47 491.46
stringcompare 2.11 5.11 7.92 13.16 2.16 0.64 0.15 0.01 48.70 114.67 128.38 491.53
unusedparameter 2.11 5.14 7.87 13.15 1.93 0.83 0.14 0.01 48.70 116.38 125.39 491.51
svnkit catch 3.94 9.91 15.16 29.80 1.91 1.02 0.02 0.00 83.09 238.49 290.88 1205.60
concatenate 3.92 10.14 15.13 29.94 2.25 1.00 0.63 0.00 78.71 238.49 350.04 1205.60
constantcompare 3.93 10.05 15.16 29.70 1.94 1.01 0.07 0.00 79.75 238.49 277.57 1205.59
nodefaultswitch 3.92 10.12 15.14 29.80 1.88 0.99 0.01 0.00 78.87 238.49 277.55 1205.59
stringcompare 3.95 9.95 15.15 29.78 1.77 0.90 0.15 0.00 83.97 238.49 283.35 1205.59
unusedparameter 3.93 10.07 15.19 29.75 2.23 1.10 0.21 0.00 81.73 241.61 285.04 1205.60
svnkit-cli catch 1.29 3.27 5.54 8.30 0.93 0.52 0.01 0.00 38.30 64.54 61.11 249.87
concatenate 1.29 3.28 5.52 8.29 0.91 0.51 0.25 0.00 38.30 64.54 74.94 249.81
constantcompare 1.29 3.28 5.55 8.32 0.92 0.51 0.03 0.00 38.30 64.54 61.71 249.81
nodefaultswitch 1.29 3.27 5.58 8.27 0.91 0.51 0.01 0.00 38.30 64.54 61.11 249.81
stringcompare 1.29 3.27 5.55 8.30 0.95 0.51 0.06 0.00 38.66 64.55 63.31 249.96
unusedparameter 1.29 3.29 5.53 8.28 1.00 0.54 0.05 0.00 38.30 64.54 61.70 249.80
svnkit-javahl catch 0.87 2.95 4.99 7.17 0.85 0.43 0.01 0.00 36.10 50.72 59.20 130.31
concatenate 0.87 2.95 4.98 7.15 0.85 0.43 0.20 0.00 36.19 50.72 58.66 130.31
constantcompare 0.87 2.94 4.96 7.21 0.78 0.43 0.03 0.00 35.76 50.72 59.20 130.31
nodefaultswitch 0.87 2.95 4.98 7.17 0.87 0.43 0.01 0.00 35.70 50.72 59.20 130.31
stringcompare 0.88 2.94 4.98 7.16 0.86 0.40 0.02 0.00 35.93 50.72 60.27 130.31
unusedparameter 0.87 2.95 4.95 7.16 0.88 0.48 0.06 0.00 37.20 50.72 60.27 130.31
webwork
-2.2.7
catch 2.28 5.23 7.95 15.36 0.86 0.68 0.02 0.00 54.50 99.75 121.22 403.71
concatenate 2.28 5.23 7.93 15.41 0.84 0.66 0.34 0.00 54.50 99.66 145.84 403.71
constantcompare 2.28 5.24 7.96 15.37 0.92 0.67 0.09 0.00 54.50 99.66 121.92 403.71
nodefaultswitch 2.29 5.26 7.93 15.41 0.84 0.66 0.02 0.00 54.50 99.46 121.22 403.71
stringcompare 2.29 5.26 7.99 15.46 0.76 0.58 0.14 0.01 54.51 99.69 127.04 403.71
unusedparameter 2.28 5.28 8.00 15.37 0.98 0.78 0.13 0.00 54.50 102.34 124.19 403.71
weka catch 9.22 23.13 34.56 71.02 2.37 1.83 0.03 0.00 234.03 740.76 701.42 3306.91
concatenate 9.33 23.19 34.38 71.23 2.30 1.78 1.43 0.00 234.03 740.75 908.25 3306.62
constantcompare 9.18 23.03 34.37 70.95 2.50 1.78 0.12 0.00 234.03 740.75 705.90 3306.74
nodefaultswitch 9.05 23.17 34.44 71.43 2.45 1.76 0.01 0.00 234.03 740.75 701.43 3316.87
stringcompare 9.27 22.91 34.39 71.47 2.20 1.61 0.26 0.00 234.03 740.75 737.18 3317.30
unusedparameter 9.08 22.89 34.32 71.29 2.80 1.98 0.32 0.00 234.67 745.70 715.73 3316.49

The measurements showed that model load time is largely independent from the query selection. It can be seen that the load time doubles when using an EMF-based implementation over the manual Java ASG, and further increases can be seen when initializing the pattern matchers for local search and incremental queries.

The required two-phase load algorithm for the EMF model (EMF case), and the time to set up the indexes (local search) and partial matches (Rete) can account for these increases. A similar increase can be seen on the memory usage: the EMF representation uses more than twice as much memory, while the incremental engine uses 10 times more memory to store its partial result caches compared to the ASG. The smaller memory footprint increase of Java ASG representation are probably caused by model-specific optimizations not applicable in generic EMF models. The additional increase for local search and Rete-based pattern matchers mainly represent the index and partial match set sizes, respectively.

For each model and each program query the average search time is listed at first. The visitor implementations perform similarly, as they traverse the entire model to find the results. The time differences between the ASG and EMF visitor implementations are mainly the result of memory optimizations, as traversing the model requires more indirection then in the EMF case.

The local search and Rete based solutions provide a two or three orders of magnitude faster query execution, achieved by replacing model traversal by calls to a pre-populated (and incrementally updated) index.

 

Additionally, the runtime of visitor implementation increases linearly with the code size. This is in line with our expectation, as visitors have to traverse the entire model during search. On the other hand, the search time for incremental queries are roughly the same for all queries, as the search means only returning the results. Surprisingly, in most of our patterns, local search time behaves similarly to incremental queries. However, the concatenation pattern behaves more like the visitors in this regard. Based on our earlier experience with different pattern matching strategies the execution performance for local search based approaches depends on the query complexity and the model structure.

Usage Profiles

In order to compare the approaches, we calculated the total time required to execute program queries for three different usage profiles: one-time, commit-time and save-time analysis.

  • One-time analysis consists of loading the model and executing each program query in a batch mode. In case the analysis needs to be repeated, the model is reloaded. In our measurements, this mode is represented by a load operation followed by a single query evaluation.
  • Commit-time analysis can be used in a program analysis server that keeps the model in-memory, and on each commit, it updates this model as opposed to reloading it entirely, and then it re-executes all queries of the analysis. In our measurements, this mode is represented by a load operation followed by 10 query evaluation.
  • Save-time analysis is executed whenever the programmer saves a file in the IDE, and then the IDE either executes the analysis itself, or notifies the analysis server. This mode is similar to commit-time analysis, just it is executed more often (with all queries executed as well). In our measurements, this mode is represented by a load operation followed by 100 query evaluation.

Diagrams

One-time analysis

one-time analysis results

Commit-time analysis results

one-time analysis results

On-save analysis results

one-time analysis results

Box-plot showing the distribution of measurement results

Box plot of results

Raw, tabular data

Model Case One-time Analysis

Commit-time Analysis

Save-time Analysis
ASG EMF LS IQ ASG EMF LS IQ ASG EMF LS IQP
argouml catch 7.42 15.75 19.69 35.17 55.44 145.48 196.77 351.67 535.60 1442.76 1967.52 3516.65
concatenate 7.37 15.88 20.39 35.20 55.30 147.06 197.88 352.01 534.69 1458.89 1972.77 3520.03
constantcompare 7.45 15.95 19.78 35.20 55.38 147.57 196.79 352.00 534.66 1463.81 1966.88 3519.94
nodefaultswitch 7.35 15.83 19.68 35.27 55.28 146.65 196.67 352.63 534.57 1454.82 1966.61 3526.28
stringcompare 7.37 15.55 19.83 35.23 55.42 144.89 197.07 352.25 535.99 1438.27 1969.44 3522.44
unusedparameter 7.60 16.06 19.90 35.27 55.48 147.29 197.20 352.69 534.23 1459.55 1970.15 3526.84

argouml-
diagrams-
sequence

catch 1.55 2.73 4.78 5.48 7.84 23.89 47.68 54.77 70.71 235.47 476.66 547.69
concatenate 1.55 2.73 4.99 5.50 7.87 23.88 48.32 54.93 71.06 235.38 481.58 549.28
constantcompare 1.57 2.72 4.81 5.51 7.90 23.82 47.78 55.06 71.20 234.81 477.53 550.55
nodefaultswitch 1.56 2.73 4.75 5.49 7.87 23.87 47.36 54.84 71.03 235.32 473.49 548.39
stringcompare 1.43 2.72 4.82 5.48 7.80 23.94 47.99 54.83 71.58 236.11 479.78 548.24
unusedparameter 1.75 2.79 4.81 5.47 8.06 23.87 47.72 54.72 71.17 234.65 476.75 547.20
argouml-mdr catch 2.05 4.04 5.79 8.77 13.72 35.05 57.73 87.62 130.43 345.21 577.17 876.22
concatenate 2.03 3.95 6.05 8.75 13.68 34.37 58.04 87.46 130.18 338.61 577.92 874.53
constantcompare 2.03 3.92 5.80 8.73 13.68 34.07 57.69 87.32 130.17 335.49 576.63 873.19
nodefaultswitch 2.02 4.01 5.81 8.73 13.68 35.02 58.00 87.27 130.29 345.14 579.96 872.71
stringcompare 2.00 4.00 5.83 8.75 13.72 34.93 57.87 87.50 130.92 344.19 578.25 874.98
unusedparameter 2.11 3.95 5.86 8.77 13.77 34.01 57.90 87.71 130.33 334.62 578.35 877.05
argouml-model catch 0.69 2.31 3.67 4.15 4.26 21.48 36.60 41.47 39.95 213.24 365.91 414.69
concatenate 0.69 2.31 3.72 4.15 4.27 21.56 36.44 41.47 40.07 213.99 363.67 414.69
constantcompare 0.69 2.33 3.70 4.16 4.26 21.75 36.82 41.53 39.97 215.91 368.01 415.32
nodefaultswitch 0.69 2.30 3.66 4.16 4.27 21.44 36.52 41.56 40.04 212.80 365.15 415.59
stringcompare 0.65 2.29 3.67 4.16 4.28 21.55 36.58 41.53 40.55 214.22 365.67 415.31
unusedparameter 0.75 2.34 3.66 4.16 4.31 21.61 36.33 41.59 39.99 214.30 363.12 415.90
cachecleaner catch 0.15 0.69 1.04 1.47 1.22 6.53 10.30 14.62 11.91 64.94 102.90 146.14
concatenate 0.15 0.69 1.07 1.47 1.22 6.55 10.32 14.64 11.94 65.15 102.78 146.39
constantcompare 0.15 0.69 1.05 1.47 1.21 6.55 10.30 14.64 11.90 65.14 102.82 146.39
nodefaultswitch 0.15 0.69 1.05 1.47 1.22 6.57 10.35 14.64 11.91 65.39 103.40 146.38
stringcompare 0.15 0.69 1.05 1.46 1.26 6.57 10.30 14.59 12.34 65.41 102.80 145.85
unusedparameter 0.15 0.69 1.05 1.47 1.22 6.56 10.31 14.65 11.91 65.20 102.89 146.49
djvuframe catch 1.25 2.34 3.82 4.95 6.27 20.86 38.13 49.50 56.48 206.08 381.24 495.00
concatenate 1.25 2.32 3.91 4.94 6.27 20.74 38.15 49.37 56.50 204.91 380.57 493.67
constantcompare 1.25 2.33 3.85 4.94 6.25 20.79 38.12 49.36 56.28 205.33 380.88 493.59
nodefaultswitch 1.25 2.33 3.82 4.94 6.28 20.77 38.09 49.39 56.58 205.17 380.77 493.92
stringcompare 1.16 2.32 3.83 4.96 6.22 20.85 38.05 49.59 56.86 206.16 380.30 495.83
unusedparameter 1.37 2.36 3.83 4.92 6.38 20.81 37.89 49.15 56.46 205.32 378.50 491.49
extract catch 3.47 5.49 7.69 12.50 22.37 48.54 76.73 124.91 211.39 479.01 767.13 1249.01
concatenate 3.38 5.43 8.00 12.50 22.29 48.00 76.66 124.92 211.40 473.63 763.21 1249.13
constantcompare 3.52 5.45 7.78 12.61 22.32 48.04 76.69 126.01 210.35 473.96 765.69 1260.01
nodefaultswitch 3.43 5.43 7.65 12.53 22.40 48.00 76.38 125.20 212.05 473.70 763.67 1251.98
stringcompare 3.62 5.37 7.76 12.50 22.60 47.98 76.47 124.91 212.33 474.13 763.55 1249.05
unusedparameter 3.49 5.53 7.80 12.51 22.25 48.26 76.65 125.04 209.84 475.63 765.16 1250.34

guess

catch 0.40 1.67 2.59 3.10 2.64 15.31 25.66 30.97 24.97 151.65 256.35 309.62
concatenate 0.40 1.68 2.66 3.10 2.63 15.37 25.61 30.96 24.97 152.31 255.09 309.51
constantcompare 0.40 1.68 2.59 3.10 2.63 15.42 25.60 30.93 24.88 152.80 255.66 309.22
nodefaultswitch 0.40 1.69 2.57 3.10 2.64 15.43 25.48 30.90 24.98 152.86 254.60 308.98
stringcompare 0.39 1.67 2.60 3.09 2.66 15.39 25.54 30.84 25.41 152.56 255.01 308.38
unusedparameter 0.43 1.69 2.59 3.10 2.67 15.42 25.45 30.90 25.00 152.64 254.13 308.90
javadjvu catch 2.80 4.18 5.86 9.78 15.65 36.93 58.36 97.78 144.10 364.43 583.39 977.79
concatenate 2.79 4.13 6.14 9.75 15.65 36.50 58.71 97.46 144.19 360.14 584.41 974.52
constantcompare 2.80 4.12 5.93 9.81 15.56 36.27 58.76 98.08 143.14 357.82 586.98 980.74
nodefaultswitch 2.79 4.12 5.86 9.79 15.63 36.21 58.36 97.79 144.05 357.13 583.33 977.88
stringcompare 2.79 4.14 5.91 9.71 15.62 36.72 58.50 97.05 143.91 362.50 584.46 970.47
unusedparameter 2.88 4.21 5.95 9.79 15.76 36.74 58.64 97.88 144.54 362.07 585.50 978.74
jml-1.0b3 catch 1.99 3.17 4.74 6.94 9.84 27.68 47.34 69.34 88.33 272.83 473.31 693.34
concatenate 1.97 3.15 4.94 6.95 9.80 27.52 47.48 69.43 88.11 271.16 472.84 694.27
constantcompare 2.00 3.16 4.75 6.95 9.86 27.48 47.30 69.51 88.47 270.73 472.70 695.09
nodefaultswitch 1.98 3.15 4.72 6.94 9.80 27.49 47.08 69.36 88.04 270.94 470.72 693.53
stringcompare 1.95 3.12 4.76 6.94 9.88 27.53 47.24 69.39 89.20 271.72 472.08 693.89
unusedparameter 2.10 3.20 4.76 6.93 9.94 27.67 47.19 69.23 88.33 272.40 471.44 692.28
jsr80 catch 0.44 1.81 2.58 3.29 2.88 16.15 25.63 32.86 27.23 159.59 256.04 328.52
concatenate 0.44 1.80 2.68 3.29 2.87 16.12 25.69 32.80 27.17 159.23 255.84 327.94
constantcompare 0.44 1.81 2.60 3.29 2.86 16.17 25.66 32.82 27.09 159.70 256.27 328.17
nodefaultswitch 0.44 1.80 2.59 3.29 2.87 16.09 25.65 32.81 27.16 158.93 256.24 328.03
stringcompare 0.43 1.78 2.60 3.28 2.89 16.16 25.64 32.76 27.54 159.96 255.98 327.52
unusedparameter 0.47 1.83 2.62 3.28 2.90 16.14 25.68 32.77 27.20 159.31 256.32 327.67
jtransforms-1.7 catch 3.21 5.57 7.98 16.64 22.47 50.63 79.50 166.33 215.03 501.27 794.69 1663.27
concatenate 3.22 5.58 8.47 16.52 22.47 50.68 80.04 165.12 215.03 501.66 795.80 1651.18
constantcompare 3.24 5.56 8.01 16.60 22.54 50.63 79.85 166.01 215.56 501.26 798.23 1660.08
nodefaultswitch 3.21 5.58 7.99 16.52 22.48 50.72 79.68 165.20 215.28 502.17 796.55 1651.95
stringcompare 3.06 5.52 8.03 16.51 22.39 50.76 79.75 165.08 215.74 503.12 796.89 1650.74
unusedparameter 3.33 5.61 8.04 16.53 22.58 50.82 79.62 165.25 215.13 502.90 795.46 1652.43
log4j-1.2.15 catch 2.94 4.45 6.55 10.53 17.48 38.97 65.20 105.24 162.93 384.15 651.72 1052.36
concatenate 2.90 4.40 6.89 10.51 17.41 38.81 65.95 105.01 162.49 382.87 656.51 1050.03
constantcompare 2.92 4.39 6.64 10.51 17.45 38.62 65.48 105.05 162.78 380.96 653.83 1050.40
nodefaultswitch 2.86 4.40 6.58 10.52 17.37 38.76 65.58 105.14 162.47 382.41 655.61 1051.28
stringcompare 3.14 4.34 6.66 10.53 17.69 38.67 65.49 105.19 163.16 381.93 653.73 1051.83
unusedparameter 3.10 4.50 6.71 10.52 17.68 38.89 65.61 105.13 163.50 382.75 654.62 1051.25
logdoc catch 0.19 1.05 1.54 2.13 1.41 10.09 15.32 21.25 13.66 100.42 153.10 212.47
concatenate 0.18 1.05 1.57 2.13 1.40 10.07 15.31 21.27 13.51 100.31 152.79 212.62
constantcompare 0.18 1.05 1.55 2.13 1.39 10.08 15.32 21.31 13.48 100.40 153.06 213.02
nodefaultswitch 0.18 1.05 1.54 2.13 1.40 10.05 15.27 21.23 13.54 100.10 152.62 212.25
stringcompare 0.19 1.05 1.55 2.13 1.44 10.10 15.35 21.22 13.95 100.59 153.28 212.18
unusedparameter 0.20 1.05 1.55 2.12 1.41 10.05 15.35 21.18 13.50 100.02 153.36 211.71
openproj-
contrib
catch 0.95 2.29 3.31 4.33 5.29 20.71 33.01 43.24 48.66 204.82 329.99 432.31
concatenate 0.95 2.28 3.43 4.33 5.27 20.65 33.19 43.27 48.46 204.35 330.79 432.63
constantcompare 0.96 2.28 3.38 4.32 5.28 20.65 33.43 43.15 48.51 204.38 333.89 431.48
nodefaultswitch 0.95 2.30 3.35 4.32 5.27 20.69 33.32 43.13 48.42 204.57 333.09 431.28
stringcompare 0.87 2.29 3.37 4.33 5.25 20.78 33.42 43.28 49.05 205.76 333.84 432.79
unusedparameter 1.04 2.31 3.37 4.32 5.37 20.72 33.32 43.17 48.61 204.79 332.80 431.67

openproj-
reports

catch 0.84 2.15 3.22 4.16 4.87 19.56 32.10 41.60 45.10 193.63 320.93 415.97
concatenate 0.84 2.14 3.29 4.15 4.84 19.48 32.10 41.46 44.85 192.84 320.13 414.54
constantcompare 0.84 2.15 3.23 4.16 4.86 19.50 31.99 41.59 45.00 193.08 319.64 415.83
nodefaultswitch 0.84 2.15 3.22 4.17 4.84 19.57 32.06 41.62 44.80 193.74 320.49 416.20
stringcompare 0.79 2.14 3.22 4.14 4.87 19.63 32.02 41.42 45.70 194.51 320.02 414.17
unusedparameter 0.90 2.17 3.23 4.15 4.92 19.52 32.04 41.43 45.12 193.00 320.08 414.27
org.tmatesoft.svnkit-
1.3.0-
bundle
catch 5.66 11.08 15.22 30.23 45.71 101.72 151.99 302.27 446.21 1008.14 1519.67 3022.67
concatenate 5.66 11.38 15.89 30.14 46.00 104.79 153.05 301.35 449.41 1038.85 1524.59 3013.44
constantcompare 5.71 11.21 15.35 30.08 45.94 102.99 152.81 300.78 448.29 1020.75 1527.41 3007.74
nodefaultswitch 5.61 11.25 15.20 30.18 45.70 103.45 151.95 301.74 446.58 1025.51 1519.46 3017.34
stringcompare 5.56 11.06 15.35 30.25 45.79 102.45 152.23 302.46 448.05 1016.26 1521.01 3024.54
unusedparameter 5.79 11.34 15.60 30.34 46.09 103.43 154.07 303.34 449.12 1024.36 1538.79 3033.42
physhun catch 1.05 2.51 3.94 5.07 6.06 22.41 39.24 50.63 56.23 221.42 392.30 506.29
concatenate 1.05 2.49 4.09 5.07 6.09 22.31 39.46 50.72 56.49 220.43 393.08 507.18
constantcompare 1.06 2.51 3.98 5.07 6.07 22.42 39.53 50.72 56.21 221.53 395.04 507.19
nodefaultswitch 1.04 2.49 3.96 5.07 6.06 22.29 39.47 50.66 56.16 220.24 394.61 506.63
stringcompare 1.00 2.48 3.98 5.10 6.07 22.43 39.46 50.99 56.76 221.84 394.32 509.85
unusedparameter 1.11 2.53 3.98 5.09 6.11 22.45 39.44 50.88 56.12 221.70 394.07 508.73
ProteinShader catch 2.76 5.03 7.17 11.47 19.41 44.72 71.43 114.63 185.92 441.60 714.10 1146.24
concatenate 2.76 5.05 7.48 11.48 19.28 44.86 71.76 114.79 184.48 442.90 714.62 1147.81
constantcompare 2.77 5.06 7.25 11.49 19.33 44.89 71.75 114.79 184.92 443.19 716.74 1147.82
nodefaultswitch 2.75 5.05 7.19 11.50 19.29 44.88 71.63 114.89 184.64 443.15 716.08 1148.87
stringcompare 2.77 4.99 7.26 11.51 19.45 44.53 71.78 115.06 186.31 440.02 716.98 1150.51
unusedparameter 2.87 5.16 7.27 11.47 19.49 45.22 71.64 114.64 185.63 445.82 715.31 1146.33
robocode catch 3.00 5.54 7.57 12.49 21.38 49.03 75.52 124.81 205.20 483.97 755.01 1248.02
concatenate 2.97 5.51 7.92 12.42 21.35 49.06 76.33 124.14 205.22 484.50 760.50 1241.39
constantcompare 3.03 5.53 7.65 12.39 21.39 49.13 75.61 123.86 204.93 485.17 755.19 1238.54
nodefaultswitch 2.97 5.52 7.58 12.44 21.33 49.08 75.61 124.35 204.95 484.68 755.98 1243.46
stringcompare 2.91 5.44 7.63 12.39 21.39 48.91 75.39 123.86 206.16 483.55 752.93 1238.54
unusedparameter 3.10 5.59 7.65 12.45 21.51 49.16 75.42 124.44 205.57 484.86 753.09 1244.33
robocode-
setup-
1.5.4
catch 3.34 5.64 7.93 14.42 23.29 50.04 79.12 144.14 222.79 494.02 791.02 1441.32
concatenate 3.32 5.72 8.22 14.44 23.21 50.98 79.20 144.31 222.13 503.51 789.08 1443.02
constantcompare 3.37 5.71 8.05 14.44 23.15 50.69 79.72 144.33 220.97 500.52 796.38 1443.22
nodefaultswitch 3.31 5.64 7.95 14.49 23.17 50.09 79.36 144.86 221.80 494.58 793.43 1448.57
stringcompare 3.26 5.58 7.98 14.45 23.13 50.10 78.74 144.47 221.92 495.33 786.34 1444.61
unusedparameter 3.44 5.74 8.02 14.47 23.30 50.32 79.12 144.62 221.88 496.15 790.14 1446.18
roborumble catch 0.65 1.97 3.02 3.64 3.91 17.96 29.97 36.39 36.44 177.89 299.45 363.81
concatenate 0.65 1.96 3.14 3.64 3.92 17.83 30.20 36.37 36.63 176.56 300.78 363.62
constantcompare 0.66 1.95 3.03 3.65 3.93 17.75 30.05 36.42 36.62 175.77 300.28 364.13
nodefaultswitch 0.65 1.95 3.03 3.63 3.92 17.77 30.11 36.25 36.64 175.96 300.94 362.49
stringcompare 0.61 1.93 3.06 3.64 3.91 17.82 30.12 36.35 36.98 176.69 300.75 363.42
unusedparameter 0.70 1.97 3.03 3.64 3.98 17.80 30.04 36.37 36.76 176.14 300.07 363.62

sdedit

catch 3.14 4.53 6.72 10.94 18.26 39.72 67.00 109.36 169.53 391.62 669.82 1093.61
concatenate 3.14 4.56 7.06 10.83 18.26 39.95 67.26 108.27 169.47 393.87 669.27 1082.62
constantcompare 3.22 4.54 6.76 10.85 18.32 39.81 66.84 108.43 169.31 392.51 667.66 1084.23
nodefaultswitch 3.14 4.53 6.72 10.81 18.24 39.72 67.02 108.09 169.24 391.57 670.05 1080.87
stringcompare 3.28 4.48 6.79 10.86 18.45 39.72 67.18 108.51 170.15 392.16 671.08 1085.09
unusedparameter 3.27 4.65 6.84 10.81 18.33 39.97 67.28 108.05 168.90 393.11 671.71 1080.46
stendhal-
maps-
0.75.1
catch 2.62 5.32 7.38 12.01 19.46 47.76 73.71 120.12 187.85 472.12 737.00 1201.18
concatenate 2.63 5.32 7.77 12.09 19.43 47.74 74.48 120.89 187.39 471.96 741.62 1208.84
constantcompare 2.68 5.35 7.42 12.02 19.45 47.83 73.87 120.19 187.13 472.72 738.37 1201.90
nodefaultswitch 2.63 5.31 7.38 12.02 19.58 47.66 73.76 120.18 189.13 471.14 737.56 1201.79
stringcompare 2.69 5.25 7.49 12.01 19.54 47.54 74.00 120.04 188.08 470.45 739.04 1200.39
unusedparameter 2.72 5.43 7.50 12.05 19.58 48.21 74.12 120.47 188.16 475.96 740.31 1204.63
stendhal-
server-
0.75.1
catch 3.97 5.84 7.89 13.17 22.96 51.91 78.70 131.70 212.79 512.58 786.77 1316.93
concatenate 3.95 5.85 8.31 13.22 22.87 52.09 79.73 132.14 212.13 514.47 793.94 1321.31
constantcompare 4.02 5.86 7.93 13.17 23.01 51.99 78.56 131.58 212.95 513.30 784.87 1315.75
nodefaultswitch 3.98 5.85 7.96 13.17 22.91 52.03 79.39 131.64 212.20 513.79 793.72 1316.32
stringcompare 4.27 5.75 8.07 13.17 23.30 51.78 79.35 131.58 213.63 512.08 792.21 1315.75
unusedparameter 4.04 5.97 8.01 13.16 23.02 52.22 78.87 131.54 212.84 514.78 787.42 1315.32
svnkit catch 5.84 10.93 15.18 29.81 41.27 100.09 151.60 298.04 395.54 991.72 1515.77 2980.38
concatenate 6.17 11.14 15.77 29.95 41.43 102.40 151.95 299.42 394.09 1015.01 1513.77 2994.19
constantcompare 5.87 11.06 15.23 29.71 41.27 101.49 151.67 297.05 395.27 1005.83 1516.01 2970.50
nodefaultswitch 5.80 11.11 15.14 29.81 41.04 102.19 151.37 298.04 393.51 1013.04 1513.65 2980.35
stringcompare 5.72 10.85 15.30 29.79 41.26 100.41 151.67 297.83 396.67 995.92 1515.39 2978.27
unusedparameter 6.16 11.17 15.40 29.76 41.52 101.78 152.15 297.53 395.19 1007.85 1519.65 2975.27
svnkit-cli catch 2.22 3.79 5.55 8.30 13.80 33.22 55.37 82.97 129.64 327.47 553.64 829.65
concatenate 2.20 3.79 5.77 8.29 13.79 33.36 55.48 82.90 129.70 328.98 552.62 828.94
constantcompare 2.21 3.78 5.58 8.32 13.80 33.26 55.49 83.18 129.67 328.07 554.68 831.73
nodefaultswitch 2.20 3.78 5.59 8.28 13.77 33.25 55.77 82.74 129.43 327.96 557.66 827.39
stringcompare 2.25 3.78 5.61 8.30 13.86 33.17 55.59 82.97 130.04 327.10 555.43 829.67
unusedparameter 2.29 3.84 5.58 8.28 13.86 33.49 55.35 82.78 129.62 330.02 553.11 827.73
svnkit-javahl catch 1.72 3.38 5.00 7.17 9.55 29.94 49.94 71.68 87.93 295.60 499.31 716.75
concatenate 1.72 3.37 5.18 7.15 9.58 29.92 49.99 71.50 88.17 295.40 498.03 714.98
constantcompare 1.65 3.37 4.99 7.22 9.49 29.84 49.66 72.14 87.90 294.56 496.28 721.42
nodefaultswitch 1.73 3.38 4.99 7.17 9.56 29.95 49.81 71.66 87.83 295.65 498.00 716.59
stringcompare 1.74 3.35 5.01 7.17 9.63 29.83 49.85 71.63 88.49 294.72 498.26 716.31
unusedparameter 1.75 3.43 5.01 7.16 9.60 29.96 49.55 71.56 88.07 295.25 494.97 715.59

webwork

-2.2.7

catch 3.13 5.91 7.97 15.36 23.65 53.02 79.49 153.59 228.76 524.08 794.73 1535.82
concatenate 3.12 5.89 8.28 15.41 23.68 52.95 79.68 154.08 229.27 523.58 793.76 1540.80
constantcompare 3.20 5.92 8.05 15.37 23.74 53.11 79.70 153.66 229.14 525.01 796.21 1536.60
nodefaultswitch 3.13 5.92 7.95 15.41 23.71 53.30 79.30 154.06 229.51 527.07 792.88 1540.59
stringcompare 3.05 5.84 8.13 15.47 23.67 53.15 80.05 154.65 229.81 526.27 799.31 1546.48
unusedparameter 3.26 6.06 8.13 15.37 23.81 53.57 80.12 153.69 229.28 528.60 800.02 1536.85
weka catch 11.59 24.96 34.58 71.02 94.56 233.13 345.61 710.20 924.28 2314.87 3455.85 7101.94
concatenate 11.62 24.98 35.82 71.24 95.55 233.72 345.28 712.35 934.86 2321.18 3439.87 7123.43
constantcompare 11.68 24.80 34.48 70.95 94.25 232.03 343.77 709.49 920.00 2304.32 3436.62 7094.89
nodefaultswitch 11.50 24.94 34.45 71.44 92.98 233.47 344.43 714.35 907.75 2318.84 3444.22 7143.45
stringcompare 11.47 24.52 34.65 71.48 94.89 230.73 344.13 714.73 929.09 2292.76 3438.95 7147.24
unusedparameter 11.88 24.88 34.64 71.30 93.64 230.93 343.57 712.92 911.18 2291.40 3432.79 7129.19

 

Lessons learned

 

  1.  Generic model implementations, such as EMF, can substitute manually optimized, specific model implementations, as their increased memory consumption is less then a factor of 2.
  2. Generic solutions require additional memory to execute up to a factor of 5 and an additional increased initialization time. However, this additional resource consumption seems acceptable for a large set of problems based on the evaluation.
  3. The runtime of the analysis itself using generic model query implementations is orders of magnitude faster than manually optimized visitor queries.
Attachment: