stats: Add SQLite database as an output format

Information
Submitter:	Andreas Hansson
Repository:	gem5
Branch:	default
Bugs:
Depends On:
Reviewers
Groups:	Default
People:

Description

Changeset 9709:8a3d4a250bc9
---------------------------
stats: Add SQLite database as an output format

This patch adds a second output format (in addition to the stats.txt) which
stores the stats in an SQLite database using SQLAlchemy. The information about
each stat is written to the database once to avoid repeated data. Each dump is
given an accociated dump ID which is based on the the number of dumps stored
in the database.

To enable the SQL output, a parameter has been added to gem5. For example,
./build/ARM/gem5.fast --stats-db-file=stats.db ....
will stored the stats in a file called "stats.db". By default SQL stats are
disabled. In order to disable the text-based stats output use --stats-file="".
For backwards compatibility the default behaviour creates stats.txt.

In case SQLAlchemy is unavailable, only stats.txt is generated. If text-based
stats are explicitly disabled, gem5 will fatal.

Testing Done

Generated a number of databases

Issue Summary

Description	From	Last Updated	Status
In the case where sql fails and you have a stats_file, will this initialize the stats file twice?	Jason Lowe-Power	Jan. 28, 2013, 6:31 p.m.	Resolved
Why do you want this check here?	Nilay Vaish	April 23, 2013, 10:51 p.m.	Resolved
Can this reopen an existing database? If so, the create_tables is going to be a problem. If not, does this ...	Nathan Binkert	April 25, 2013, 5:41 a.m.	Open
Is it possible that subsequent calls to this have information that the initial ones do not? Do you support reopening ...	Nathan Binkert	April 25, 2013, 5:41 a.m.	Open
the format of the identifier as "init_SQL" follows no existing convention in the code base that I know of :) ...	Nathan Binkert	April 25, 2013, 5:41 a.m.	Open
this looks like an accidental duplicate	Nathan Binkert	April 25, 2013, 5:41 a.m.	Open
This try catch seems superfluous.	Nathan Binkert	April 25, 2013, 5:41 a.m.	Open
You're duplicating stuff here. Shouldn't create_tables simply be declared after all of the tables and simply be create_tables = Base.metadata.create_all	Nathan Binkert	April 25, 2013, 5:41 a.m.	Open

src/python/m5/main.py (Diff revision 1)

In the case where sql fails and you have a stats_file, will this initialize the stats file twice?

Show all issues

Description:

~		Changeset 9491:a129a7aecb97
	~	Changeset 9505:bfc1e913d287

		stats: Add SQLite database as an output format

		This patch adds a second output format (in addition to the stats.txt) which
		stores the stats in an SQLite database using SQLAlchemy. The information about
		each stat is written to the database once to avoid repeated data. Each dump is
		given an accociated dump ID which is based on the the number of dumps stored
		in the database.

		To enable the SQL output, a parameter has been added to gem5. For example,
		./build/ARM/gem5.fast --stats-db-file=stats.db ....
		will stored the stats in a file called "stats.db". By default SQL stats are
		disabled. In order to disable the text-based stats output use --stats-file="".
		For backwards compatibility the default behaviour creates stats.txt.

		In case SQLAlchemy is unavailable, only stats.txt is generated. If text-based
		stats are explicitly disabled, gem5 will fatal.

Diff:

Revision 2 (+389 -4)

Show changes

	src/python/SConscript
	src/python/m5/main.py
	src/python/m5/stats/__init__.py
	src/python/m5/stats/sql.py

src/python/m5/main.py (Diff revision 2)

Why do you want this check here?

Show all issues

Sascha Bischoff Jan. 28, 2013, 7:09 p.m. (Jan. 28, 2013, 7:09 p.m.)

If both the text-based stats and the sql stats are disabled, then gem5 will not produce any stats. In my mind, this is wrong. Do you think we should just warn, rather than calling fatal?

Nilay Vaish Jan. 29, 2013, 2:43 a.m. (Jan. 29, 2013, 2:43 a.m.)

What I am trying to suggest is that the check used here it self is not sufficient to 
ensure that whether or not the stats will get collected properly. Consider a 
situtation in which there are more checks that need to be carried before ensuring 
that the stats text file can actually be used. You would not want to
replicate those checks in both places (here and below). Instead use a boolean variable
that is initialized to false and then it is changed to true if the stats can be 
recorded in at least one format. Later, on the basis of the value of the variable, you
can choose to call panic / fatal / warn, whatever you like and print the required message.

Description:

~		Changeset 9505:bfc1e913d287
	~	Changeset 9515:1331b07a1479

		stats: Add SQLite database as an output format

		This patch adds a second output format (in addition to the stats.txt) which
		stores the stats in an SQLite database using SQLAlchemy. The information about
		each stat is written to the database once to avoid repeated data. Each dump is
		given an accociated dump ID which is based on the the number of dumps stored
		in the database.

		To enable the SQL output, a parameter has been added to gem5. For example,
		./build/ARM/gem5.fast --stats-db-file=stats.db ....
		will stored the stats in a file called "stats.db". By default SQL stats are
		disabled. In order to disable the text-based stats output use --stats-file="".
		For backwards compatibility the default behaviour creates stats.txt.

		In case SQLAlchemy is unavailable, only stats.txt is generated. If text-based
		stats are explicitly disabled, gem5 will fatal.

Diff:

Revision 3 (+394 -4)

Show changes

	src/python/SConscript
	src/python/m5/main.py
	src/python/m5/stats/__init__.py
	src/python/m5/stats/sql.py

Description:

~		Changeset 9515:1331b07a1479
	~	Changeset 9709:8a3d4a250bc9

		stats: Add SQLite database as an output format

		This patch adds a second output format (in addition to the stats.txt) which
		stores the stats in an SQLite database using SQLAlchemy. The information about
		each stat is written to the database once to avoid repeated data. Each dump is
		given an accociated dump ID which is based on the the number of dumps stored
		in the database.

		To enable the SQL output, a parameter has been added to gem5. For example,
		./build/ARM/gem5.fast --stats-db-file=stats.db ....
		will stored the stats in a file called "stats.db". By default SQL stats are
		disabled. In order to disable the text-based stats output use --stats-file="".
		For backwards compatibility the default behaviour creates stats.txt.

		In case SQLAlchemy is unavailable, only stats.txt is generated. If text-based
		stats are explicitly disabled, gem5 will fatal.

Diff:

Revision 4 (+410 -5)

Show changes

	src/python/SConscript
	src/python/m5/main.py
	src/python/m5/stats/__init__.py
	src/python/m5/stats/sql.py

I'm glad to see you using a real database!

src/python/m5/stats/__init__.py (Diff revision 4)

Can this reopen an existing database?  If so, the create_tables is going to be a problem.  If not, does this error out or delete the original?

Show all issues

src/python/m5/stats/__init__.py (Diff revision 4)

Is it possible that subsequent calls to this have information that the initial ones do not?  Do you support reopening an existing file to add data to it?  If not, you should probably add some error code.

Show all issues

src/python/m5/stats/__init__.py (Diff revision 4)

the format of the identifier as "init_SQL" follows no existing convention in the code base that I know of :)

I'd prefer init_sql. Of course, we are inconsistent with how we treat

Show all issues

src/python/m5/stats/__init__.py (Diff revision 4)

this looks like an accidental duplicate

Show all issues

src/python/m5/stats/sql.py (Diff revision 4)

This try catch seems superfluous.

Show all issues

src/python/m5/stats/sql.py (Diff revision 4)

You're duplicating stuff here.  Shouldn't create_tables simply be declared after all of the tables and simply be

create_tables = Base.metadata.create_all

Show all issues

You have a pending review.

Review Board 2.0.15

Screenshots

Files

Issue Summary

Description:

Changeset 9491:a129a7aecb97

Changeset 9505:bfc1e913d287

Diff:

Description:

Changeset 9505:bfc1e913d287

Changeset 9515:1331b07a1479

Diff:

Description:

Changeset 9515:1331b07a1479

Changeset 9709:8a3d4a250bc9

Diff: