Program Query Evaluation

Authors: Zoltán Ujhelyi, Ákos Horváth, Dániel Varró, Norbert István Csiszár, Gábor Szőke, László Vidács, Rudolf Ferenc

In the CSMR-WCRE 2014 paper "Anti-pattern Detection with Model Queries: A Comparison of Approaches" we compare three different approaches for evaluating program queries. This page details all measurements done for the paper.

Model sizes

The approaches were evaluated on a test set of 30 modules of 17 open source projects. The projects are sized between 1kLOC and 500kLOC, are are used in various scenarios, thus sharing different characteristics. The list of projects include the ArgoUML UML editor, JTransforms, a Fourier transformation library, the SVNKit Subversion client, the online homework system WeBWorK, the Weka data mining software and many more.

The next table describes the sizes of these projects in lines of code together with the size of the ASG and EMF models in terms of nodes and edges.

Project	Module	LOC	ASG		EMF
Project	Module	LOC	Nodes	Edges	Nodes	Edges
argouml	argouml.jar	307069	700459	1468495	711121	1479044
	argouml-diagrams-sequence.jar	3683	44684	85650	46964	87928
	argouml-mdr.jar	22476	115018	234585	118637	238151
	argouml-model.jar	13834	19903	35557	20623	36277
javadjvu	djvuframe.jar	1398	32909	60067	35459	62616
javadjvu	javadjvu.jar	36223	124185	250372	127520	253696
jml	jml-1.0b3.jar	17182	70403	146020	72754	148365
jsr80	jsr80.jar	4251	11288	21636	12172	22520
jtransforms	jtransforms	43118	294067	651947	294910	652790
log4j	log4j-1.2.15.jar	31428	135444	270076	140760	275392
physhun	physhun.jar	7368	40974	85412	42773	87073
projectlibre	openproj-contrib-tmp.jar	2624	29275	55640	31563	57925
projectlibre	openproj-reports-tmp.jar	1241	24753	46092	26986	48325
proteinshader	ProteinShader.jar	50775	158358	309201	164121	314956
qwicap-guess	guess. war	525	10302	19686	11290	20674
robocode	cachecleaner.jar	73	2667	4891	2956	5180
	extract.jar	40624	195413	399603	201409	405284
	robocode-setup-1.5.4.jar	41027	197959	406191	203975	411517
	robocode.jar	37092	180656	367825	186537	373698
	roborumble.jar	2347	19128	38366	20047	39284
sdedit	sdedit.jar	22710	145380	291375	151837	297823
stendhal	stendhal-maps-0.75.1.jar	33345	185450	407602	188189	410338
stendhal	stendhal-server-0.75.1.jar	52059	216194	458040	221627	463465
svnkit	svnkit-cli.jar	13524	108188	232597	114206	238588
	svnkit-javahl.jar	34799	63101	120762	67744	125284
	svnkit.jar	121276	554165	1225837	566166	1237308
	org.tmatesoft.svnkit-1.3.0-bundle.jar	130298	566401	1250727	578493	1262167
webwork	webwork	48782	217616	446727	224544	453641
weka	weka	468018	1529023	3286541	1542205	3299530
xins	logdoc.jar	570	4815	8916	5173	9274

Measurement Description

All measurements were executed on a dedicated Linux-based server with 6 GB RAM running Java 6. On the server the Java ASG of the Columbus Framework was installed together with the EMF-INCQUERY model query framework.

All program queries were implemented as both visitors for the ASG (by Columbus experts) and as graph patterns (by EMF-INCQUERY experts). The visitor implementation was executed on both model representations, while the graph patterns were used to initialize local search and Rete network based pattern matchers for the EMF representation. In all cases, the time to load the model from its serialized form and the time to execute the program query was measured together with the maximum heap size usage.

Every program query was executed ten times, and the time and memory results were averaged. In order to minimize the interference between different runs, for each execution a new JVM was created and each query was run in isolation. The time to start up and shut down the JVM were not included in the measurement results.

Memory and Search Time Results

		Load Time (s)				Search Time (s)				Maximum Heap Size (MB)
Model	Case	ASG	EMF	LS	IQ	ASG	EMF	LS	IQ	ASG	EMF	LS	IQ
argouml	catch	5.34	14.41	19.68	35.17	2.09	1.33	0.02	0.00	116.93	317.56	363.78	1571.18
	concatenate	5.33	14.58	19.72	35.20	2.04	1.30	0.67	0.00	116.93	317.56	449.84	1571.05
	constantcompare	5.33	14.62	19.67	35.20	2.12	1.32	0.12	0.00	116.93	317.56	363.78	1571.13
	nodefaultswitch	5.33	14.54	19.67	35.26	2.02	1.30	0.01	0.00	116.93	317.56	362.29	1571.16
	stringcompare	5.34	14.37	19.69	35.22	2.03	1.18	0.14	0.00	116.94	317.56	372.35	1571.05
	unusedparameter	5.32	14.58	19.70	35.27	2.28	1.48	0.20	0.00	116.93	323.49	369.74	1571.07
argouml- diagrams- sequence	catch	0.70	2.35	4.77	5.48	0.85	0.38	0.01	0.00	20.48	44.82	49.98	118.21
	concatenate	0.70	2.35	4.81	5.49	0.85	0.38	0.18	0.00	20.54	44.82	49.85	118.21
	constantcompare	0.70	2.34	4.78	5.51	0.86	0.38	0.03	0.00	20.54	45.26	50.40	118.21
	nodefaultswitch	0.70	2.35	4.73	5.48	0.85	0.38	0.01	0.00	20.52	44.82	49.98	118.21
	stringcompare	0.71	2.36	4.80	5.48	0.72	0.36	0.02	0.00	20.33	45.25	50.38	118.21
	unusedparameter	0.70	2.34	4.77	5.47	1.05	0.45	0.05	0.00	21.26	41.06	50.56	118.21
argouml-mdr	catch	1.30	3.45	5.77	8.76	0.75	0.59	0.02	0.00	24.91	53.78	80.65	238.01
	concatenate	1.29	3.38	5.78	8.75	0.73	0.56	0.27	0.00	24.62	53.78	80.38	238.02
	constantcompare	1.29	3.35	5.77	8.73	0.74	0.57	0.03	0.00	24.61	53.78	81.19	238.01
	nodefaultswitch	1.30	3.45	5.80	8.73	0.73	0.56	0.01	0.00	24.91	53.78	80.65	238.02
	stringcompare	1.30	3.44	5.78	8.75	0.70	0.56	0.05	0.00	24.62	53.39	82.26	238.01
	unusedparameter	1.30	3.34	5.78	8.77	0.82	0.61	0.07	0.00	26.18	54.86	82.29	238.01
argouml-model	catch	0.40	2.13	3.66	4.15	0.29	0.18	0.01	0.00	24.08	26.71	38.79	64.56
	concatenate	0.40	2.14	3.64	4.15	0.29	0.18	0.08	0.00	24.08	26.71	40.92	64.56
	constantcompare	0.40	2.16	3.68	4.15	0.29	0.18	0.02	0.00	24.09	26.71	38.79	64.55
	nodefaultswitch	0.40	2.13	3.65	4.16	0.29	0.18	0.01	0.00	24.08	26.71	38.79	64.56
	stringcompare	0.40	2.14	3.66	4.15	0.25	0.15	0.01	0.00	24.16	26.72	38.79	64.56
	unusedparameter	0.40	2.14	3.63	4.16	0.35	0.20	0.02	0.00	24.40	27.23	39.32	64.56
cachecleaner	catch	0.12	0.65	1.03	1.46	0.03	0.04	0.01	0.01	9.18	13.24	19.23	24.39
	concatenate	0.12	0.65	1.03	1.46	0.03	0.04	0.04	0.01	9.56	13.24	19.73	24.50
	constantcompare	0.12	0.65	1.03	1.46	0.03	0.04	0.02	0.00	9.18	13.24	19.73	24.50
	nodefaultswitch	0.12	0.65	1.03	1.46	0.03	0.04	0.01	0.00	9.18	13.24	19.23	24.50
	stringcompare	0.12	0.65	1.03	1.46	0.02	0.03	0.02	0.00	9.18	13.26	19.73	24.39
	unusedparameter	0.12	0.65	1.03	1.46	0.03	0.04	0.02	0.01	9.56	13.74	19.73	24.44
djvuframe	catch	0.56	2.06	3.81	4.95	0.69	0.28	0.01	0.00	31.90	19.98	35.77	82.32
	concatenate	0.56	2.05	3.80	4.94	0.69	0.28	0.11	0.00	31.90	19.98	38.97	82.32
	constantcompare	0.56	2.05	3.81	4.94	0.69	0.28	0.04	0.00	31.90	19.98	36.31	82.32
	nodefaultswitch	0.56	2.05	3.81	4.94	0.69	0.28	0.01	0.00	31.90	19.98	35.77	82.32
	stringcompare	0.56	2.06	3.80	4.96	0.60	0.26	0.02	0.00	31.91	19.76	36.31	82.32
	unusedparameter	0.56	2.05	3.78	4.91	0.81	0.31	0.04	0.00	31.90	20.80	36.31	82.32
extract	catch	2.10	4.78	7.67	12.49	1.36	0.71	0.02	0.01	36.39	96.04	143.87	373.13
	concatenate	2.10	4.73	7.63	12.49	1.28	0.70	0.38	0.01	36.22	96.04	143.87	373.13
	constantcompare	2.09	4.73	7.66	12.60	1.43	0.71	0.13	0.01	36.54	96.04	143.86	373.13
	nodefaultswitch	2.11	4.73	7.64	12.52	1.32	0.70	0.02	0.01	36.41	96.04	143.86	373.13
	stringcompare	2.11	4.74	7.63	12.49	1.52	0.63	0.13	0.00	36.61	96.05	143.86	373.13
	unusedparameter	2.08	4.75	7.65	12.50	1.41	0.78	0.15	0.01	38.68	98.68	143.86	373.13
guess	catch	0.25	1.51	2.56	3.10	0.15	0.16	0.03	0.01	15.62	26.77	22.14	31.58
	concatenate	0.25	1.52	2.55	3.10	0.15	0.16	0.11	0.01	15.63	26.77	23.67	31.56
	constantcompare	0.25	1.53	2.56	3.09	0.16	0.16	0.04	0.01	15.63	26.77	22.14	31.58
	nodefaultswitch	0.25	1.53	2.55	3.09	0.15	0.16	0.02	0.01	15.63	26.77	22.14	31.58
	stringcompare	0.25	1.52	2.55	3.08	0.13	0.15	0.05	0.01	16.15	26.60	22.14	31.58
	unusedparameter	0.25	1.52	2.54	3.09	0.18	0.17	0.05	0.01	16.11	26.52	22.14	31.56
javadjvu	catch	1.43	3.64	5.83	9.78	1.37	0.54	0.03	0.01	34.44	73.75	70.21	281.04
	concatenate	1.43	3.60	5.84	9.75	1.36	0.54	0.30	0.01	34.41	73.75	84.89	280.71
	constantcompare	1.42	3.57	5.87	9.81	1.39	0.55	0.06	0.01	34.19	73.75	70.41	280.69
	nodefaultswitch	1.43	3.57	5.83	9.78	1.36	0.55	0.03	0.01	34.21	73.75	70.47	280.73
	stringcompare	1.43	3.62	5.84	9.70	1.36	0.52	0.06	0.01	34.33	65.65	71.75	280.69
	unusedparameter	1.43	3.61	5.85	9.79	1.45	0.59	0.10	0.01	35.22	65.64	71.55	280.69
jml-1.0b3	catch	0.87	2.72	4.73	6.93	1.11	0.45	0.01	0.00	36.18	46.66	49.35	134.47
	concatenate	0.87	2.71	4.73	6.94	1.10	0.45	0.21	0.00	36.18	46.65	57.59	134.47
	constantcompare	0.87	2.70	4.73	6.95	1.13	0.46	0.03	0.00	36.13	46.66	49.35	134.47
	nodefaultswitch	0.87	2.71	4.71	6.94	1.11	0.44	0.01	0.00	36.05	46.66	49.35	134.47
	stringcompare	0.88	2.71	4.72	6.94	1.06	0.40	0.04	0.00	35.89	46.66	50.43	134.47
	unusedparameter	0.87	2.72	4.71	6.92	1.23	0.48	0.05	0.00	36.18	47.77	50.43	134.47
jsr80	catch	0.27	1.59	2.56	3.29	0.17	0.22	0.02	0.00	16.53	28.60	24.07	36.72
	concatenate	0.27	1.59	2.56	3.28	0.17	0.21	0.12	0.01	16.52	28.61	25.61	36.72
	constantcompare	0.27	1.59	2.56	3.28	0.17	0.22	0.04	0.01	16.52	28.61	24.07	36.71
	nodefaultswitch	0.27	1.59	2.56	3.28	0.17	0.22	0.03	0.01	16.52	28.61	24.07	36.74
	stringcompare	0.27	1.60	2.56	3.28	0.15	0.18	0.04	0.01	16.97	28.61	24.07	36.73
	unusedparameter	0.27	1.59	2.56	3.28	0.20	0.24	0.05	0.01	17.03	29.10	24.59	36.74
jtransforms-1.7	catch	2.14	5.01	7.95	16.63	1.07	0.56	0.03	0.00	50.16	131.76	173.83	668.14
	concatenate	2.14	5.01	7.95	16.51	1.08	0.57	0.51	0.00	50.16	131.76	173.83	668.19
	constantcompare	2.14	5.01	7.98	16.60	1.09	0.56	0.03	0.00	50.16	131.76	173.83	668.18
	nodefaultswitch	2.14	5.02	7.97	16.52	1.06	0.56	0.02	0.00	50.16	131.76	173.83	668.20
	stringcompare	2.15	5.03	7.97	16.51	0.91	0.50	0.07	0.00	50.18	131.77	173.83	668.19
	unusedparameter	2.14	5.02	7.95	16.52	1.19	0.58	0.09	0.00	50.16	131.76	173.83	668.19
log4j-1.2.15	catch	1.62	3.84	6.52	10.52	1.32	0.61	0.03	0.01	44.54	69.77	80.79	268.51
	concatenate	1.61	3.82	6.56	10.50	1.29	0.58	0.33	0.01	44.54	69.77	80.06	268.51
	constantcompare	1.61	3.80	6.54	10.50	1.31	0.58	0.10	0.01	44.50	69.77	81.32	268.52
	nodefaultswitch	1.61	3.82	6.56	10.51	1.25	0.58	0.02	0.01	44.53	69.77	80.70	268.51
	stringcompare	1.62	3.81	6.54	10.52	1.52	0.53	0.13	0.01	44.51	69.77	83.00	268.51
	unusedparameter	1.62	3.82	6.54	10.51	1.48	0.68	0.17	0.01	39.29	71.90	82.90	268.51
logdoc	catch	0.14	1.00	1.53	2.12	0.05	0.05	0.01	0.01	11.01	17.09	26.71	18.63
	concatenate	0.13	1.00	1.53	2.13	0.05	0.05	0.04	0.01	10.97	17.08	27.45	18.63
	constantcompare	0.13	1.00	1.53	2.13	0.05	0.05	0.02	0.00	10.97	17.08	26.71	18.63
	nodefaultswitch	0.13	1.00	1.53	2.12	0.05	0.05	0.01	0.00	10.97	17.09	26.71	18.63
	stringcompare	0.14	1.01	1.53	2.12	0.05	0.04	0.02	0.00	11.08	17.11	26.71	18.63
	unusedparameter	0.13	1.00	1.53	2.12	0.06	0.05	0.02	0.01	11.48	17.57	26.71	18.63
openproj- contrib	catch	0.48	2.05	3.30	4.32	0.47	0.25	0.01	0.01	30.53	37.28	38.24	72.94
	concatenate	0.48	2.04	3.31	4.33	0.47	0.24	0.13	0.01	30.62	37.28	41.31	72.94
	constantcompare	0.48	2.04	3.34	4.31	0.48	0.24	0.04	0.00	30.62	37.28	38.64	72.94
	nodefaultswitch	0.48	2.04	3.33	4.31	0.47	0.26	0.02	0.00	30.54	37.28	38.31	72.94
	stringcompare	0.49	2.06	3.34	4.33	0.38	0.23	0.04	0.00	30.59	37.35	38.64	72.94
	unusedparameter	0.48	2.05	3.33	4.32	0.56	0.26	0.05	0.01	31.10	37.70	38.64	72.94
openproj- reports	catch	0.45	1.93	3.21	4.16	0.40	0.22	0.01	0.00	27.03	29.92	26.14	65.95
	concatenate	0.44	1.93	3.20	4.15	0.40	0.22	0.09	0.00	27.04	29.92	28.79	65.95
	constantcompare	0.45	1.93	3.20	4.16	0.40	0.22	0.03	0.00	27.04	29.92	26.67	65.95
	nodefaultswitch	0.44	1.94	3.20	4.16	0.40	0.22	0.01	0.00	27.04	29.92	26.14	65.95
	stringcompare	0.45	1.94	3.20	4.14	0.34	0.20	0.02	0.00	27.15	29.95	26.67	65.95
	unusedparameter	0.45	1.93	3.20	4.14	0.46	0.24	0.03	0.00	27.54	30.44	26.67	65.95
org.tmatesoft.svnkit- 1.3.0- bundle	catch	4.45	10.07	15.20	30.23	1.21	1.01	0.02	0.00	91.90	276.20	297.22	1264.26
	concatenate	4.48	10.38	15.24	30.13	1.17	1.00	0.66	0.00	91.90	276.30	376.63	1263.94
	constantcompare	4.47	10.20	15.27	30.08	1.24	1.02	0.07	0.00	92.79	276.26	297.25	1263.69
	nodefaultswitch	4.45	10.25	15.19	30.17	1.16	1.00	0.01	0.00	91.90	276.20	297.22	1264.27
	stringcompare	4.47	10.15	15.21	30.25	1.09	0.91	0.15	0.00	91.90	276.23	310.00	1264.27
	unusedparameter	4.48	10.23	15.39	30.33	1.31	1.11	0.22	0.00	94.57	277.91	304.95	1263.70
physhun	catch	0.56	2.21	3.92	5.06	0.49	0.29	0.01	0.00	16.29	38.68	36.38	110.42
	concatenate	0.56	2.20	3.93	5.07	0.49	0.29	0.16	0.00	16.29	38.68	41.46	110.42
	constantcompare	0.56	2.21	3.95	5.07	0.50	0.29	0.03	0.00	16.29	38.68	36.38	110.42
	nodefaultswitch	0.56	2.20	3.95	5.07	0.49	0.29	0.01	0.00	16.29	38.68	36.38	110.42
	stringcompare	0.56	2.22	3.94	5.10	0.44	0.27	0.04	0.00	16.33	39.14	37.45	110.42
	unusedparameter	0.56	2.21	3.94	5.09	0.55	0.32	0.04	0.00	16.80	39.20	36.91	110.42
ProteinShader	catch	1.85	4.41	7.14	11.46	0.91	0.62	0.03	0.01	44.01	86.96	108.40	317.25
	concatenate	1.84	4.42	7.14	11.48	0.92	0.63	0.33	0.01	44.10	86.96	107.67	317.21
	constantcompare	1.84	4.43	7.17	11.48	0.93	0.64	0.08	0.01	44.08	86.96	108.50	317.25
	nodefaultswitch	1.84	4.43	7.16	11.49	0.92	0.63	0.03	0.01	44.15	86.96	108.30	317.18
	stringcompare	1.85	4.39	7.17	11.51	0.91	0.59	0.09	0.01	44.14	86.97	110.66	317.19
	unusedparameter	1.85	4.45	7.15	11.46	1.03	0.71	0.12	0.01	46.20	86.96	110.14	317.23
robocode	catch	2.04	4.83	7.55	12.48	0.96	0.71	0.02	0.01	49.05	60.39	100.76	382.80
	concatenate	2.04	4.84	7.60	12.41	0.92	0.67	0.31	0.01	49.05	60.34	121.55	382.82
	constantcompare	2.04	4.84	7.55	12.39	0.99	0.68	0.10	0.01	49.05	60.39	101.70	382.78
	nodefaultswitch	2.04	4.84	7.56	12.43	0.93	0.68	0.02	0.01	49.05	60.39	100.76	382.82
	stringcompare	2.05	4.83	7.53	12.39	0.86	0.61	0.10	0.00	49.23	60.29	103.49	382.80
	unusedparameter	2.05	4.84	7.53	12.44	1.06	0.75	0.12	0.01	49.05	62.68	102.59	382.84
robocode- setup- 1.5.4	catch	2.22	4.93	7.91	14.41	1.12	0.71	0.02	0.01	39.26	104.07	107.91	396.22
	concatenate	2.21	5.03	7.89	14.43	1.11	0.70	0.33	0.01	39.26	104.07	130.06	396.31
	constantcompare	2.20	5.00	7.96	14.43	1.17	0.71	0.09	0.01	39.26	104.08	108.25	396.35
	nodefaultswitch	2.21	4.94	7.93	14.49	1.10	0.71	0.02	0.01	39.26	104.07	107.35	396.22
	stringcompare	2.21	4.95	7.86	14.45	1.05	0.63	0.12	0.01	39.26	104.09	110.94	396.18
	unusedparameter	2.21	4.95	7.90	14.46	1.23	0.78	0.12	0.00	39.26	106.69	110.05	396.35
roborumble	catch	0.36	1.78	2.99	3.64	0.29	0.19	0.02	0.01	22.02	20.96	20.16	49.18
	concatenate	0.36	1.76	3.01	3.64	0.29	0.20	0.13	0.01	22.03	20.96	22.78	49.18
	constantcompare	0.36	1.76	3.00	3.64	0.30	0.20	0.03	0.01	22.04	20.92	20.16	49.17
	nodefaultswitch	0.36	1.76	3.01	3.62	0.29	0.19	0.02	0.01	22.15	20.96	20.16	49.18
	stringcompare	0.37	1.77	3.01	3.63	0.24	0.16	0.05	0.00	22.14	20.93	20.68	49.18
	unusedparameter	0.36	1.76	3.00	3.64	0.33	0.21	0.03	0.00	22.44	21.45	20.16	49.17
sdedit	catch	1.68	3.91	6.70	10.94	1.46	0.62	0.02	0.00	46.06	81.55	88.95	330.14
	concatenate	1.68	3.93	6.69	10.83	1.46	0.62	0.37	0.01	46.06	81.55	88.28	331.29
	constantcompare	1.68	3.92	6.68	10.84	1.54	0.62	0.08	0.01	46.06	81.55	89.62	331.16
	nodefaultswitch	1.68	3.91	6.70	10.81	1.47	0.62	0.02	0.01	46.06	81.55	88.47	331.29
	stringcompare	1.69	3.92	6.71	10.85	1.59	0.56	0.08	0.01	46.06	81.56	91.15	331.00
	unusedparameter	1.67	3.92	6.72	10.80	1.60	0.73	0.12	0.01	46.06	81.55	90.57	331.30
stendhal- maps- 0.75.1	catch	1.87	4.72	7.37	12.01	0.75	0.61	0.01	0.00	34.48	106.20	122.12	410.93
	concatenate	1.87	4.71	7.41	12.09	0.77	0.61	0.36	0.00	34.66	106.14	146.34	410.93
	constantcompare	1.86	4.72	7.38	12.02	0.82	0.62	0.03	0.00	34.61	106.20	123.11	410.90
	nodefaultswitch	1.88	4.71	7.38	12.02	0.74	0.61	0.01	0.00	34.61	106.20	122.13	410.93
	stringcompare	1.87	4.70	7.39	12.00	0.82	0.55	0.10	0.00	28.26	106.14	127.08	410.93
	unusedparameter	1.87	4.75	7.40	12.05	0.85	0.68	0.10	0.00	35.67	93.48	125.13	410.93
stendhal- server- 0.75.1	catch	2.11	5.12	7.87	13.17	1.86	0.72	0.02	0.01	48.70	114.59	122.47	491.27
	concatenate	2.10	5.14	7.94	13.21	1.84	0.72	0.38	0.01	48.70	114.59	148.96	491.57
	constantcompare	2.11	5.13	7.85	13.16	1.91	0.74	0.08	0.01	48.70	114.59	123.44	491.61
	nodefaultswitch	2.10	5.13	7.94	13.16	1.87	0.72	0.02	0.01	48.70	114.59	122.47	491.46
	stringcompare	2.11	5.11	7.92	13.16	2.16	0.64	0.15	0.01	48.70	114.67	128.38	491.53
	unusedparameter	2.11	5.14	7.87	13.15	1.93	0.83	0.14	0.01	48.70	116.38	125.39	491.51
svnkit	catch	3.94	9.91	15.16	29.80	1.91	1.02	0.02	0.00	83.09	238.49	290.88	1205.60
	concatenate	3.92	10.14	15.13	29.94	2.25	1.00	0.63	0.00	78.71	238.49	350.04	1205.60
	constantcompare	3.93	10.05	15.16	29.70	1.94	1.01	0.07	0.00	79.75	238.49	277.57	1205.59
	nodefaultswitch	3.92	10.12	15.14	29.80	1.88	0.99	0.01	0.00	78.87	238.49	277.55	1205.59
	stringcompare	3.95	9.95	15.15	29.78	1.77	0.90	0.15	0.00	83.97	238.49	283.35	1205.59
	unusedparameter	3.93	10.07	15.19	29.75	2.23	1.10	0.21	0.00	81.73	241.61	285.04	1205.60
svnkit-cli	catch	1.29	3.27	5.54	8.30	0.93	0.52	0.01	0.00	38.30	64.54	61.11	249.87
	concatenate	1.29	3.28	5.52	8.29	0.91	0.51	0.25	0.00	38.30	64.54	74.94	249.81
	constantcompare	1.29	3.28	5.55	8.32	0.92	0.51	0.03	0.00	38.30	64.54	61.71	249.81
	nodefaultswitch	1.29	3.27	5.58	8.27	0.91	0.51	0.01	0.00	38.30	64.54	61.11	249.81
	stringcompare	1.29	3.27	5.55	8.30	0.95	0.51	0.06	0.00	38.66	64.55	63.31	249.96
	unusedparameter	1.29	3.29	5.53	8.28	1.00	0.54	0.05	0.00	38.30	64.54	61.70	249.80
svnkit-javahl	catch	0.87	2.95	4.99	7.17	0.85	0.43	0.01	0.00	36.10	50.72	59.20	130.31
	concatenate	0.87	2.95	4.98	7.15	0.85	0.43	0.20	0.00	36.19	50.72	58.66	130.31
	constantcompare	0.87	2.94	4.96	7.21	0.78	0.43	0.03	0.00	35.76	50.72	59.20	130.31
	nodefaultswitch	0.87	2.95	4.98	7.17	0.87	0.43	0.01	0.00	35.70	50.72	59.20	130.31
	stringcompare	0.88	2.94	4.98	7.16	0.86	0.40	0.02	0.00	35.93	50.72	60.27	130.31
	unusedparameter	0.87	2.95	4.95	7.16	0.88	0.48	0.06	0.00	37.20	50.72	60.27	130.31
webwork -2.2.7	catch	2.28	5.23	7.95	15.36	0.86	0.68	0.02	0.00	54.50	99.75	121.22	403.71
	concatenate	2.28	5.23	7.93	15.41	0.84	0.66	0.34	0.00	54.50	99.66	145.84	403.71
	constantcompare	2.28	5.24	7.96	15.37	0.92	0.67	0.09	0.00	54.50	99.66	121.92	403.71
	nodefaultswitch	2.29	5.26	7.93	15.41	0.84	0.66	0.02	0.00	54.50	99.46	121.22	403.71
	stringcompare	2.29	5.26	7.99	15.46	0.76	0.58	0.14	0.01	54.51	99.69	127.04	403.71
	unusedparameter	2.28	5.28	8.00	15.37	0.98	0.78	0.13	0.00	54.50	102.34	124.19	403.71
weka	catch	9.22	23.13	34.56	71.02	2.37	1.83	0.03	0.00	234.03	740.76	701.42	3306.91
	concatenate	9.33	23.19	34.38	71.23	2.30	1.78	1.43	0.00	234.03	740.75	908.25	3306.62
	constantcompare	9.18	23.03	34.37	70.95	2.50	1.78	0.12	0.00	234.03	740.75	705.90	3306.74
	nodefaultswitch	9.05	23.17	34.44	71.43	2.45	1.76	0.01	0.00	234.03	740.75	701.43	3316.87
	stringcompare	9.27	22.91	34.39	71.47	2.20	1.61	0.26	0.00	234.03	740.75	737.18	3317.30
	unusedparameter	9.08	22.89	34.32	71.29	2.80	1.98	0.32	0.00	234.67	745.70	715.73	3316.49

The measurements showed that model load time is largely independent from the query selection. It can be seen that the load time doubles when using an EMF-based implementation over the manual Java ASG, and further increases can be seen when initializing the pattern matchers for local search and incremental queries.

The required two-phase load algorithm for the EMF model (EMF case), and the time to set up the indexes (local search) and partial matches (Rete) can account for these increases. A similar increase can be seen on the memory usage: the EMF representation uses more than twice as much memory, while the incremental engine uses 10 times more memory to store its partial result caches compared to the ASG. The smaller memory footprint increase of Java ASG representation are probably caused by model-specific optimizations not applicable in generic EMF models. The additional increase for local search and Rete-based pattern matchers mainly represent the index and partial match set sizes, respectively.

For each model and each program query the average search time is listed at first. The visitor implementations perform similarly, as they traverse the entire model to find the results. The time differences between the ASG and EMF visitor implementations are mainly the result of memory optimizations, as traversing the model requires more indirection then in the EMF case.

The local search and Rete based solutions provide a two or three orders of magnitude faster query execution, achieved by replacing model traversal by calls to a pre-populated (and incrementally updated) index.

Additionally, the runtime of visitor implementation increases linearly with the code size. This is in line with our expectation, as visitors have to traverse the entire model during search. On the other hand, the search time for incremental queries are roughly the same for all queries, as the search means only returning the results. Surprisingly, in most of our patterns, local search time behaves similarly to incremental queries. However, the concatenation pattern behaves more like the visitors in this regard. Based on our earlier experience with different pattern matching strategies the execution performance for local search based approaches depends on the query complexity and the model structure.

Usage Profiles

In order to compare the approaches, we calculated the total time required to execute program queries for three different usage profiles: one-time, commit-time and save-time analysis.

One-time analysis consists of loading the model and executing each program query in a batch mode. In case the analysis needs to be repeated, the model is reloaded. In our measurements, this mode is represented by a load operation followed by a single query evaluation.
Commit-time analysis can be used in a program analysis server that keeps the model in-memory, and on each commit, it updates this model as opposed to reloading it entirely, and then it re-executes all queries of the analysis. In our measurements, this mode is represented by a load operation followed by 10 query evaluation.
Save-time analysis is executed whenever the programmer saves a file in the IDE, and then the IDE either executes the analysis itself, or notifies the analysis server. This mode is similar to commit-time analysis, just it is executed more often (with all queries executed as well). In our measurements, this mode is represented by a load operation followed by 100 query evaluation.

Diagrams

One-time analysis

Commit-time analysis results

On-save analysis results

Box-plot showing the distribution of measurement results

Raw, tabular data

Model	Case	One-time Analysis				Commit-time Analysis				Save-time Analysis
Model	Case	ASG	EMF	LS	IQ	ASG	EMF	LS	IQ	ASG	EMF	LS	IQP
argouml	catch	7.42	15.75	19.69	35.17	55.44	145.48	196.77	351.67	535.60	1442.76	1967.52	3516.65
	concatenate	7.37	15.88	20.39	35.20	55.30	147.06	197.88	352.01	534.69	1458.89	1972.77	3520.03
	constantcompare	7.45	15.95	19.78	35.20	55.38	147.57	196.79	352.00	534.66	1463.81	1966.88	3519.94
	nodefaultswitch	7.35	15.83	19.68	35.27	55.28	146.65	196.67	352.63	534.57	1454.82	1966.61	3526.28
	stringcompare	7.37	15.55	19.83	35.23	55.42	144.89	197.07	352.25	535.99	1438.27	1969.44	3522.44
	unusedparameter	7.60	16.06	19.90	35.27	55.48	147.29	197.20	352.69	534.23	1459.55	1970.15	3526.84
argouml- diagrams- sequence	catch	1.55	2.73	4.78	5.48	7.84	23.89	47.68	54.77	70.71	235.47	476.66	547.69
	concatenate	1.55	2.73	4.99	5.50	7.87	23.88	48.32	54.93	71.06	235.38	481.58	549.28
	constantcompare	1.57	2.72	4.81	5.51	7.90	23.82	47.78	55.06	71.20	234.81	477.53	550.55
	nodefaultswitch	1.56	2.73	4.75	5.49	7.87	23.87	47.36	54.84	71.03	235.32	473.49	548.39
	stringcompare	1.43	2.72	4.82	5.48	7.80	23.94	47.99	54.83	71.58	236.11	479.78	548.24
	unusedparameter	1.75	2.79	4.81	5.47	8.06	23.87	47.72	54.72	71.17	234.65	476.75	547.20
argouml-mdr	catch	2.05	4.04	5.79	8.77	13.72	35.05	57.73	87.62	130.43	345.21	577.17	876.22
	concatenate	2.03	3.95	6.05	8.75	13.68	34.37	58.04	87.46	130.18	338.61	577.92	874.53
	constantcompare	2.03	3.92	5.80	8.73	13.68	34.07	57.69	87.32	130.17	335.49	576.63	873.19
	nodefaultswitch	2.02	4.01	5.81	8.73	13.68	35.02	58.00	87.27	130.29	345.14	579.96	872.71
	stringcompare	2.00	4.00	5.83	8.75	13.72	34.93	57.87	87.50	130.92	344.19	578.25	874.98
	unusedparameter	2.11	3.95	5.86	8.77	13.77	34.01	57.90	87.71	130.33	334.62	578.35	877.05
argouml-model	catch	0.69	2.31	3.67	4.15	4.26	21.48	36.60	41.47	39.95	213.24	365.91	414.69
	concatenate	0.69	2.31	3.72	4.15	4.27	21.56	36.44	41.47	40.07	213.99	363.67	414.69
	constantcompare	0.69	2.33	3.70	4.16	4.26	21.75	36.82	41.53	39.97	215.91	368.01	415.32
	nodefaultswitch	0.69	2.30	3.66	4.16	4.27	21.44	36.52	41.56	40.04	212.80	365.15	415.59
	stringcompare	0.65	2.29	3.67	4.16	4.28	21.55	36.58	41.53	40.55	214.22	365.67	415.31
	unusedparameter	0.75	2.34	3.66	4.16	4.31	21.61	36.33	41.59	39.99	214.30	363.12	415.90
cachecleaner	catch	0.15	0.69	1.04	1.47	1.22	6.53	10.30	14.62	11.91	64.94	102.90	146.14
	concatenate	0.15	0.69	1.07	1.47	1.22	6.55	10.32	14.64	11.94	65.15	102.78	146.39
	constantcompare	0.15	0.69	1.05	1.47	1.21	6.55	10.30	14.64	11.90	65.14	102.82	146.39
	nodefaultswitch	0.15	0.69	1.05	1.47	1.22	6.57	10.35	14.64	11.91	65.39	103.40	146.38
	stringcompare	0.15	0.69	1.05	1.46	1.26	6.57	10.30	14.59	12.34	65.41	102.80	145.85
	unusedparameter	0.15	0.69	1.05	1.47	1.22	6.56	10.31	14.65	11.91	65.20	102.89	146.49
djvuframe	catch	1.25	2.34	3.82	4.95	6.27	20.86	38.13	49.50	56.48	206.08	381.24	495.00
	concatenate	1.25	2.32	3.91	4.94	6.27	20.74	38.15	49.37	56.50	204.91	380.57	493.67
	constantcompare	1.25	2.33	3.85	4.94	6.25	20.79	38.12	49.36	56.28	205.33	380.88	493.59
	nodefaultswitch	1.25	2.33	3.82	4.94	6.28	20.77	38.09	49.39	56.58	205.17	380.77	493.92
	stringcompare	1.16	2.32	3.83	4.96	6.22	20.85	38.05	49.59	56.86	206.16	380.30	495.83
	unusedparameter	1.37	2.36	3.83	4.92	6.38	20.81	37.89	49.15	56.46	205.32	378.50	491.49
extract	catch	3.47	5.49	7.69	12.50	22.37	48.54	76.73	124.91	211.39	479.01	767.13	1249.01
	concatenate	3.38	5.43	8.00	12.50	22.29	48.00	76.66	124.92	211.40	473.63	763.21	1249.13
	constantcompare	3.52	5.45	7.78	12.61	22.32	48.04	76.69	126.01	210.35	473.96	765.69	1260.01
	nodefaultswitch	3.43	5.43	7.65	12.53	22.40	48.00	76.38	125.20	212.05	473.70	763.67	1251.98
	stringcompare	3.62	5.37	7.76	12.50	22.60	47.98	76.47	124.91	212.33	474.13	763.55	1249.05
	unusedparameter	3.49	5.53	7.80	12.51	22.25	48.26	76.65	125.04	209.84	475.63	765.16	1250.34
guess	catch	0.40	1.67	2.59	3.10	2.64	15.31	25.66	30.97	24.97	151.65	256.35	309.62
	concatenate	0.40	1.68	2.66	3.10	2.63	15.37	25.61	30.96	24.97	152.31	255.09	309.51
	constantcompare	0.40	1.68	2.59	3.10	2.63	15.42	25.60	30.93	24.88	152.80	255.66	309.22
	nodefaultswitch	0.40	1.69	2.57	3.10	2.64	15.43	25.48	30.90	24.98	152.86	254.60	308.98
	stringcompare	0.39	1.67	2.60	3.09	2.66	15.39	25.54	30.84	25.41	152.56	255.01	308.38
	unusedparameter	0.43	1.69	2.59	3.10	2.67	15.42	25.45	30.90	25.00	152.64	254.13	308.90
javadjvu	catch	2.80	4.18	5.86	9.78	15.65	36.93	58.36	97.78	144.10	364.43	583.39	977.79
	concatenate	2.79	4.13	6.14	9.75	15.65	36.50	58.71	97.46	144.19	360.14	584.41	974.52
	constantcompare	2.80	4.12	5.93	9.81	15.56	36.27	58.76	98.08	143.14	357.82	586.98	980.74
	nodefaultswitch	2.79	4.12	5.86	9.79	15.63	36.21	58.36	97.79	144.05	357.13	583.33	977.88
	stringcompare	2.79	4.14	5.91	9.71	15.62	36.72	58.50	97.05	143.91	362.50	584.46	970.47
	unusedparameter	2.88	4.21	5.95	9.79	15.76	36.74	58.64	97.88	144.54	362.07	585.50	978.74
jml-1.0b3	catch	1.99	3.17	4.74	6.94	9.84	27.68	47.34	69.34	88.33	272.83	473.31	693.34
	concatenate	1.97	3.15	4.94	6.95	9.80	27.52	47.48	69.43	88.11	271.16	472.84	694.27
	constantcompare	2.00	3.16	4.75	6.95	9.86	27.48	47.30	69.51	88.47	270.73	472.70	695.09
	nodefaultswitch	1.98	3.15	4.72	6.94	9.80	27.49	47.08	69.36	88.04	270.94	470.72	693.53
	stringcompare	1.95	3.12	4.76	6.94	9.88	27.53	47.24	69.39	89.20	271.72	472.08	693.89
	unusedparameter	2.10	3.20	4.76	6.93	9.94	27.67	47.19	69.23	88.33	272.40	471.44	692.28
jsr80	catch	0.44	1.81	2.58	3.29	2.88	16.15	25.63	32.86	27.23	159.59	256.04	328.52
	concatenate	0.44	1.80	2.68	3.29	2.87	16.12	25.69	32.80	27.17	159.23	255.84	327.94
	constantcompare	0.44	1.81	2.60	3.29	2.86	16.17	25.66	32.82	27.09	159.70	256.27	328.17
	nodefaultswitch	0.44	1.80	2.59	3.29	2.87	16.09	25.65	32.81	27.16	158.93	256.24	328.03
	stringcompare	0.43	1.78	2.60	3.28	2.89	16.16	25.64	32.76	27.54	159.96	255.98	327.52
	unusedparameter	0.47	1.83	2.62	3.28	2.90	16.14	25.68	32.77	27.20	159.31	256.32	327.67
jtransforms-1.7	catch	3.21	5.57	7.98	16.64	22.47	50.63	79.50	166.33	215.03	501.27	794.69	1663.27
	concatenate	3.22	5.58	8.47	16.52	22.47	50.68	80.04	165.12	215.03	501.66	795.80	1651.18
	constantcompare	3.24	5.56	8.01	16.60	22.54	50.63	79.85	166.01	215.56	501.26	798.23	1660.08
	nodefaultswitch	3.21	5.58	7.99	16.52	22.48	50.72	79.68	165.20	215.28	502.17	796.55	1651.95
	stringcompare	3.06	5.52	8.03	16.51	22.39	50.76	79.75	165.08	215.74	503.12	796.89	1650.74
	unusedparameter	3.33	5.61	8.04	16.53	22.58	50.82	79.62	165.25	215.13	502.90	795.46	1652.43
log4j-1.2.15	catch	2.94	4.45	6.55	10.53	17.48	38.97	65.20	105.24	162.93	384.15	651.72	1052.36
	concatenate	2.90	4.40	6.89	10.51	17.41	38.81	65.95	105.01	162.49	382.87	656.51	1050.03
	constantcompare	2.92	4.39	6.64	10.51	17.45	38.62	65.48	105.05	162.78	380.96	653.83	1050.40
	nodefaultswitch	2.86	4.40	6.58	10.52	17.37	38.76	65.58	105.14	162.47	382.41	655.61	1051.28
	stringcompare	3.14	4.34	6.66	10.53	17.69	38.67	65.49	105.19	163.16	381.93	653.73	1051.83
	unusedparameter	3.10	4.50	6.71	10.52	17.68	38.89	65.61	105.13	163.50	382.75	654.62	1051.25
logdoc	catch	0.19	1.05	1.54	2.13	1.41	10.09	15.32	21.25	13.66	100.42	153.10	212.47
	concatenate	0.18	1.05	1.57	2.13	1.40	10.07	15.31	21.27	13.51	100.31	152.79	212.62
	constantcompare	0.18	1.05	1.55	2.13	1.39	10.08	15.32	21.31	13.48	100.40	153.06	213.02
	nodefaultswitch	0.18	1.05	1.54	2.13	1.40	10.05	15.27	21.23	13.54	100.10	152.62	212.25
	stringcompare	0.19	1.05	1.55	2.13	1.44	10.10	15.35	21.22	13.95	100.59	153.28	212.18
	unusedparameter	0.20	1.05	1.55	2.12	1.41	10.05	15.35	21.18	13.50	100.02	153.36	211.71
openproj- contrib	catch	0.95	2.29	3.31	4.33	5.29	20.71	33.01	43.24	48.66	204.82	329.99	432.31
	concatenate	0.95	2.28	3.43	4.33	5.27	20.65	33.19	43.27	48.46	204.35	330.79	432.63
	constantcompare	0.96	2.28	3.38	4.32	5.28	20.65	33.43	43.15	48.51	204.38	333.89	431.48
	nodefaultswitch	0.95	2.30	3.35	4.32	5.27	20.69	33.32	43.13	48.42	204.57	333.09	431.28
	stringcompare	0.87	2.29	3.37	4.33	5.25	20.78	33.42	43.28	49.05	205.76	333.84	432.79
	unusedparameter	1.04	2.31	3.37	4.32	5.37	20.72	33.32	43.17	48.61	204.79	332.80	431.67
openproj- reports	catch	0.84	2.15	3.22	4.16	4.87	19.56	32.10	41.60	45.10	193.63	320.93	415.97
	concatenate	0.84	2.14	3.29	4.15	4.84	19.48	32.10	41.46	44.85	192.84	320.13	414.54
	constantcompare	0.84	2.15	3.23	4.16	4.86	19.50	31.99	41.59	45.00	193.08	319.64	415.83
	nodefaultswitch	0.84	2.15	3.22	4.17	4.84	19.57	32.06	41.62	44.80	193.74	320.49	416.20
	stringcompare	0.79	2.14	3.22	4.14	4.87	19.63	32.02	41.42	45.70	194.51	320.02	414.17
	unusedparameter	0.90	2.17	3.23	4.15	4.92	19.52	32.04	41.43	45.12	193.00	320.08	414.27
org.tmatesoft.svnkit- 1.3.0- bundle	catch	5.66	11.08	15.22	30.23	45.71	101.72	151.99	302.27	446.21	1008.14	1519.67	3022.67
	concatenate	5.66	11.38	15.89	30.14	46.00	104.79	153.05	301.35	449.41	1038.85	1524.59	3013.44
	constantcompare	5.71	11.21	15.35	30.08	45.94	102.99	152.81	300.78	448.29	1020.75	1527.41	3007.74
	nodefaultswitch	5.61	11.25	15.20	30.18	45.70	103.45	151.95	301.74	446.58	1025.51	1519.46	3017.34
	stringcompare	5.56	11.06	15.35	30.25	45.79	102.45	152.23	302.46	448.05	1016.26	1521.01	3024.54
	unusedparameter	5.79	11.34	15.60	30.34	46.09	103.43	154.07	303.34	449.12	1024.36	1538.79	3033.42
physhun	catch	1.05	2.51	3.94	5.07	6.06	22.41	39.24	50.63	56.23	221.42	392.30	506.29
	concatenate	1.05	2.49	4.09	5.07	6.09	22.31	39.46	50.72	56.49	220.43	393.08	507.18
	constantcompare	1.06	2.51	3.98	5.07	6.07	22.42	39.53	50.72	56.21	221.53	395.04	507.19
	nodefaultswitch	1.04	2.49	3.96	5.07	6.06	22.29	39.47	50.66	56.16	220.24	394.61	506.63
	stringcompare	1.00	2.48	3.98	5.10	6.07	22.43	39.46	50.99	56.76	221.84	394.32	509.85
	unusedparameter	1.11	2.53	3.98	5.09	6.11	22.45	39.44	50.88	56.12	221.70	394.07	508.73
ProteinShader	catch	2.76	5.03	7.17	11.47	19.41	44.72	71.43	114.63	185.92	441.60	714.10	1146.24
	concatenate	2.76	5.05	7.48	11.48	19.28	44.86	71.76	114.79	184.48	442.90	714.62	1147.81
	constantcompare	2.77	5.06	7.25	11.49	19.33	44.89	71.75	114.79	184.92	443.19	716.74	1147.82
	nodefaultswitch	2.75	5.05	7.19	11.50	19.29	44.88	71.63	114.89	184.64	443.15	716.08	1148.87
	stringcompare	2.77	4.99	7.26	11.51	19.45	44.53	71.78	115.06	186.31	440.02	716.98	1150.51
	unusedparameter	2.87	5.16	7.27	11.47	19.49	45.22	71.64	114.64	185.63	445.82	715.31	1146.33
robocode	catch	3.00	5.54	7.57	12.49	21.38	49.03	75.52	124.81	205.20	483.97	755.01	1248.02
	concatenate	2.97	5.51	7.92	12.42	21.35	49.06	76.33	124.14	205.22	484.50	760.50	1241.39
	constantcompare	3.03	5.53	7.65	12.39	21.39	49.13	75.61	123.86	204.93	485.17	755.19	1238.54
	nodefaultswitch	2.97	5.52	7.58	12.44	21.33	49.08	75.61	124.35	204.95	484.68	755.98	1243.46
	stringcompare	2.91	5.44	7.63	12.39	21.39	48.91	75.39	123.86	206.16	483.55	752.93	1238.54
	unusedparameter	3.10	5.59	7.65	12.45	21.51	49.16	75.42	124.44	205.57	484.86	753.09	1244.33
robocode- setup- 1.5.4	catch	3.34	5.64	7.93	14.42	23.29	50.04	79.12	144.14	222.79	494.02	791.02	1441.32
	concatenate	3.32	5.72	8.22	14.44	23.21	50.98	79.20	144.31	222.13	503.51	789.08	1443.02
	constantcompare	3.37	5.71	8.05	14.44	23.15	50.69	79.72	144.33	220.97	500.52	796.38	1443.22
	nodefaultswitch	3.31	5.64	7.95	14.49	23.17	50.09	79.36	144.86	221.80	494.58	793.43	1448.57
	stringcompare	3.26	5.58	7.98	14.45	23.13	50.10	78.74	144.47	221.92	495.33	786.34	1444.61
	unusedparameter	3.44	5.74	8.02	14.47	23.30	50.32	79.12	144.62	221.88	496.15	790.14	1446.18
roborumble	catch	0.65	1.97	3.02	3.64	3.91	17.96	29.97	36.39	36.44	177.89	299.45	363.81
	concatenate	0.65	1.96	3.14	3.64	3.92	17.83	30.20	36.37	36.63	176.56	300.78	363.62
	constantcompare	0.66	1.95	3.03	3.65	3.93	17.75	30.05	36.42	36.62	175.77	300.28	364.13
	nodefaultswitch	0.65	1.95	3.03	3.63	3.92	17.77	30.11	36.25	36.64	175.96	300.94	362.49
	stringcompare	0.61	1.93	3.06	3.64	3.91	17.82	30.12	36.35	36.98	176.69	300.75	363.42
	unusedparameter	0.70	1.97	3.03	3.64	3.98	17.80	30.04	36.37	36.76	176.14	300.07	363.62
sdedit	catch	3.14	4.53	6.72	10.94	18.26	39.72	67.00	109.36	169.53	391.62	669.82	1093.61
	concatenate	3.14	4.56	7.06	10.83	18.26	39.95	67.26	108.27	169.47	393.87	669.27	1082.62
	constantcompare	3.22	4.54	6.76	10.85	18.32	39.81	66.84	108.43	169.31	392.51	667.66	1084.23
	nodefaultswitch	3.14	4.53	6.72	10.81	18.24	39.72	67.02	108.09	169.24	391.57	670.05	1080.87
	stringcompare	3.28	4.48	6.79	10.86	18.45	39.72	67.18	108.51	170.15	392.16	671.08	1085.09
	unusedparameter	3.27	4.65	6.84	10.81	18.33	39.97	67.28	108.05	168.90	393.11	671.71	1080.46
stendhal- maps- 0.75.1	catch	2.62	5.32	7.38	12.01	19.46	47.76	73.71	120.12	187.85	472.12	737.00	1201.18
	concatenate	2.63	5.32	7.77	12.09	19.43	47.74	74.48	120.89	187.39	471.96	741.62	1208.84
	constantcompare	2.68	5.35	7.42	12.02	19.45	47.83	73.87	120.19	187.13	472.72	738.37	1201.90
	nodefaultswitch	2.63	5.31	7.38	12.02	19.58	47.66	73.76	120.18	189.13	471.14	737.56	1201.79
	stringcompare	2.69	5.25	7.49	12.01	19.54	47.54	74.00	120.04	188.08	470.45	739.04	1200.39
	unusedparameter	2.72	5.43	7.50	12.05	19.58	48.21	74.12	120.47	188.16	475.96	740.31	1204.63
stendhal- server- 0.75.1	catch	3.97	5.84	7.89	13.17	22.96	51.91	78.70	131.70	212.79	512.58	786.77	1316.93
	concatenate	3.95	5.85	8.31	13.22	22.87	52.09	79.73	132.14	212.13	514.47	793.94	1321.31
	constantcompare	4.02	5.86	7.93	13.17	23.01	51.99	78.56	131.58	212.95	513.30	784.87	1315.75
	nodefaultswitch	3.98	5.85	7.96	13.17	22.91	52.03	79.39	131.64	212.20	513.79	793.72	1316.32
	stringcompare	4.27	5.75	8.07	13.17	23.30	51.78	79.35	131.58	213.63	512.08	792.21	1315.75
	unusedparameter	4.04	5.97	8.01	13.16	23.02	52.22	78.87	131.54	212.84	514.78	787.42	1315.32
svnkit	catch	5.84	10.93	15.18	29.81	41.27	100.09	151.60	298.04	395.54	991.72	1515.77	2980.38
	concatenate	6.17	11.14	15.77	29.95	41.43	102.40	151.95	299.42	394.09	1015.01	1513.77	2994.19
	constantcompare	5.87	11.06	15.23	29.71	41.27	101.49	151.67	297.05	395.27	1005.83	1516.01	2970.50
	nodefaultswitch	5.80	11.11	15.14	29.81	41.04	102.19	151.37	298.04	393.51	1013.04	1513.65	2980.35
	stringcompare	5.72	10.85	15.30	29.79	41.26	100.41	151.67	297.83	396.67	995.92	1515.39	2978.27
	unusedparameter	6.16	11.17	15.40	29.76	41.52	101.78	152.15	297.53	395.19	1007.85	1519.65	2975.27
svnkit-cli	catch	2.22	3.79	5.55	8.30	13.80	33.22	55.37	82.97	129.64	327.47	553.64	829.65
	concatenate	2.20	3.79	5.77	8.29	13.79	33.36	55.48	82.90	129.70	328.98	552.62	828.94
	constantcompare	2.21	3.78	5.58	8.32	13.80	33.26	55.49	83.18	129.67	328.07	554.68	831.73
	nodefaultswitch	2.20	3.78	5.59	8.28	13.77	33.25	55.77	82.74	129.43	327.96	557.66	827.39
	stringcompare	2.25	3.78	5.61	8.30	13.86	33.17	55.59	82.97	130.04	327.10	555.43	829.67
	unusedparameter	2.29	3.84	5.58	8.28	13.86	33.49	55.35	82.78	129.62	330.02	553.11	827.73
svnkit-javahl	catch	1.72	3.38	5.00	7.17	9.55	29.94	49.94	71.68	87.93	295.60	499.31	716.75
	concatenate	1.72	3.37	5.18	7.15	9.58	29.92	49.99	71.50	88.17	295.40	498.03	714.98
	constantcompare	1.65	3.37	4.99	7.22	9.49	29.84	49.66	72.14	87.90	294.56	496.28	721.42
	nodefaultswitch	1.73	3.38	4.99	7.17	9.56	29.95	49.81	71.66	87.83	295.65	498.00	716.59
	stringcompare	1.74	3.35	5.01	7.17	9.63	29.83	49.85	71.63	88.49	294.72	498.26	716.31
	unusedparameter	1.75	3.43	5.01	7.16	9.60	29.96	49.55	71.56	88.07	295.25	494.97	715.59
webwork -2.2.7	catch	3.13	5.91	7.97	15.36	23.65	53.02	79.49	153.59	228.76	524.08	794.73	1535.82
	concatenate	3.12	5.89	8.28	15.41	23.68	52.95	79.68	154.08	229.27	523.58	793.76	1540.80
	constantcompare	3.20	5.92	8.05	15.37	23.74	53.11	79.70	153.66	229.14	525.01	796.21	1536.60
	nodefaultswitch	3.13	5.92	7.95	15.41	23.71	53.30	79.30	154.06	229.51	527.07	792.88	1540.59
	stringcompare	3.05	5.84	8.13	15.47	23.67	53.15	80.05	154.65	229.81	526.27	799.31	1546.48
	unusedparameter	3.26	6.06	8.13	15.37	23.81	53.57	80.12	153.69	229.28	528.60	800.02	1536.85
weka	catch	11.59	24.96	34.58	71.02	94.56	233.13	345.61	710.20	924.28	2314.87	3455.85	7101.94
	concatenate	11.62	24.98	35.82	71.24	95.55	233.72	345.28	712.35	934.86	2321.18	3439.87	7123.43
	constantcompare	11.68	24.80	34.48	70.95	94.25	232.03	343.77	709.49	920.00	2304.32	3436.62	7094.89
	nodefaultswitch	11.50	24.94	34.45	71.44	92.98	233.47	344.43	714.35	907.75	2318.84	3444.22	7143.45
	stringcompare	11.47	24.52	34.65	71.48	94.89	230.73	344.13	714.73	929.09	2292.76	3438.95	7147.24
	unusedparameter	11.88	24.88	34.64	71.30	93.64	230.93	343.57	712.92	911.18	2291.40	3432.79	7129.19

Lessons learned

Generic model implementations, such as EMF, can substitute manually optimized, specific model implementations, as their increased memory consumption is less then a factor of 2.
Generic solutions require additional memory to execute up to a factor of 5 and an additional increased initialization time. However, this additional resource consumption seems acceptable for a large set of problems based on the evaluation.
The runtime of the analysis itself using generic model query implementations is orders of magnitude faster than manually optimized visitor queries.

Attachment:

Raw measurement results

Primary links