config: Add the ability to read a config file using C++ and Python

Information
Submitter:	Andreas Hansson
Repository:	gem5
Branch:	default
Bugs:
Depends On:
Reviewers
Groups:	Default
People:

Description

Changeset 10440:faea9dd0b3b1
---------------------------
config: Add the ability to read a config file using C++ and Python

This patch adds the ability to load in config.ini files generated from
gem5 into another instance of gem5 built without Python configuration
support. The intended use case is for configuring gem5 when it is a
library embedded in another simulation system.

A parallel config file reader is also provided purely in Python to
demonstrate the approach taken and to provided similar functionality
for as-yet-unknown use models. The Python configuration file reader
can read both .ini and .json files.

C++ configuration file reading:

A command line option has been added for scons to enable C++ configuration
file reading: --with-cxx-config

There is an example in util/cxx_config that shows C++ configuration in action.
util/cxx_config/README explains how to build the example.

Configuration is achieved by the object CxxConfigManager. It handles
reading object descriptions from a CxxConfigFileBase object which
wraps a config file reader. The wrapper class CxxIniFile is provided
which wraps an IniFile for reading .ini files. Reading .json files
from C++ would be possible with a similar wrapper and a JSON parser.

After reading object descriptions, CxxConfigManager creates
SimObjectParam-derived objects from the classes in the (generated with this
patch) directory build/ARCH/cxx_config

CxxConfigManager can then build SimObjects from those SimObjectParams (in an
order dictated by the SimObject-value parameters on other objects) and bind
ports of the produced SimObjects.

A minimal set of instantiate-replacing member functions are provided by
CxxConfigManager and few of the member functions of SimObject (such as drain)
are extended onto CxxConfigManager.

Python configuration file reading (configs/example/read_config.py):

A Python version of the reader is also supplied with a similar interface to
CxxConfigFileBase (In Python: ConfigFile) to config file readers.

The Python config file reading will handle both .ini and .json files.

The object construction strategy is slightly different in Python from the C++
reader as you need to avoid objects prematurely becoming the children of other
objects when setting parameters.

Port binding also needs to be strictly in the same port-index order as the
original instantiation.

Testing Done

Description:

~		Changeset 10426:c5ec71d6ca6f
	~	Changeset 10440:faea9dd0b3b1

		config: Add the ability to read a config file using C++ and Python

		This patch adds the ability to load in config.ini files generated from
		gem5 into another instance of gem5 built without Python configuration
		support. The intended use case is for configuring gem5 when it is a
		library embedded in another simulation system.

		A parallel config file reader is also provided purely in Python to
		demonstrate the approach taken and to provided similar functionality
		for as-yet-unknown use models. The Python configuration file reader
		can read both .ini and .json files.

		C++ configuration file reading:

		A command line option has been added for scons to enable C++ configuration
		file reading: --with-cxx-config

		There is an example in util/cxx_config that shows C++ configuration in action.
		util/cxx_config/README explains how to build the example.

		Configuration is achieved by the object CxxConfigManager. It handles
		reading object descriptions from a CxxConfigFileBase object which
		wraps a config file reader. The wrapper class CxxIniFile is provided
		which wraps an IniFile for reading .ini files. Reading .json files
		from C++ would be possible with a similar wrapper and a JSON parser.

		After reading object descriptions, CxxConfigManager creates
		SimObjectParam-derived objects from the classes in the (generated with this
		patch) directory build/ARCH/cxx_config

		CxxConfigManager can then build SimObjects from those SimObjectParams (in an
		order dictated by the SimObject-value parameters on other objects) and bind
		ports of the produced SimObjects.

		A minimal set of instantiate-replacing member functions are provided by
		CxxConfigManager and few of the member functions of SimObject (such as drain)
		are extended onto CxxConfigManager.

		Python configuration file reading (configs/example/read_config.py):

		A Python version of the reader is also supplied with a similar interface to
		CxxConfigFileBase (In Python: ConfigFile) to config file readers.

		The Python config file reading will handle both .ini and .json files.

		The object construction strategy is slightly different in Python from the C++
		reader as you need to avoid objects prematurely becoming the children of other
		objects when setting parameters.

		Port binding also needs to be strictly in the same port-index order as the
		original instantiation.

Diff:

Revision 2 (+3168)

Show changes

	SConstruct
	configs/example/read_config.py
	src/SConscript
	src/python/m5/SimObject.py
	src/python/m5/params.py
	src/sim/SConscript
	src/sim/cxx_config.hh
	src/sim/cxx_config.cc
	9 more

Status: Closed (submitted)

This patch seems to be broken for X86 when restoring from checkpoints. A sample test crashes with segmentation fault. Here are the steps:
0. Sample test program does int array manipulation and creates a checkpoint before computation.
1. Run in se mode with --cpu-type=atomic --mem-type=simple_mem to keep things simple. This will create the checkpoint.
2. Run in se mode with --cpu-type=atomic --mem-type=simple_mem -r 1 and it will run fine.
3. Now run

gem5.opt.cxx m5out/config.ini -r m5out/cpt.XYZ

and it will seg fault right away at:

Simulate:: Entering event queue @ 7088000. Starting simulation...
EventQueue (MainEventQueue-0)::schedule event (Event_21) for 18446744073709551615
EventQueue (MainEventQueue-0)::serviceOne event named Event_8 at 7088500
Program received signal SIGSEGV, Segmentation fault.
getPage (addr=4195355, this=0x0) at build/X86/cpu/decode_cache.hh:86
86              if (recent[0] != pageMap.end()) {
(gdb)

Trying it with other cpu-type and other mem-type results in similar seg faults. It looks like this patch is missing some state initialization or some other checkpoint restore functionality for X86.

Ali Saidi Nov. 6, 2014, 12:57 p.m. (Nov. 6, 2014, 12:57 p.m.)

I assume it's working for you without restoring from checkpoints? 

The code looks like state is being restored, but we've only ever tested it with ARM although I'm not sure why there would be a difference.

Cagdas Dirik Nov. 6, 2014, 1:35 p.m. (Nov. 6, 2014, 1:35 p.m.)
```
Correct!
```

Cagdas Dirik Nov. 6, 2014, 3:01 p.m. (Nov. 6, 2014, 3:01 p.m.)

In addition to X86, ARM also seg faulted switching to arm_detailed cpu after restoring from checkpoint. Here are the steps:

0. Sample test program does int array manipulation and creates a checkpoint before computation.
1. Run in se mode with --cpu-type=atomic --mem-type=simple_mem to keep things simple. This will create the checkpoint
2. Run in se mode with --caches --l2cache --cpu-type=arm_detailed --mem-type=simple_mem -r 1
3. Now run

gem5.opt.cxx m5out/config.ini -r m5out/cpt.XYZ -c system.cpu system.switch_cpus 10000

and it will seg fault soon (at about tick delta 48500) at:

EventQueue (MainEventQueue-0)::serviceOne event named Event_82 at 3837000
EventQueue (MainEventQueue-0)::schedule event (Event_82) for 3837500
EventQueue (MainEventQueue-0)::serviceOne event named Event_82 at 3837500
EventQueue (MainEventQueue-0)::schedule event (Event_82) for 3838000
EventQueue (MainEventQueue-0)::serviceOne event named Event_82 at 3838000
Program received signal SIGSEGV, Segmentation fault.
0x00002aaaab3908e8 in ROB<O3CPUImpl>::updateHead (this=this@entry=0xa90438)
    at build/ARM/cpu/o3/rob_impl.hh:413
413         list<ThreadID>::iterator threads = activeThreads->begin();
(gdb)

Cagdas Dirik Nov. 6, 2014, 4:39 p.m. (Nov. 6, 2014, 4:39 p.m.)

As Andrew Bardsley pointed out adding

config_manager->startup()

at line 268 in util/cxx_config/main.cc (after the loadState(checkpoint) line).

resolves the issues - both on X86 and ARM.

Details from email thread:
"...The startup call corresponds to startup in simulate(...) in simulate.py which is guarded by need_startup...The proximal error when switching to O3, BTW, was a lack of a call to ROB<>::setActiveThreads because of the lack of a call to Commit::startupStage() because of the lack of a call to O3::startup()."

Ali Saidi Nov. 7, 2014, 10:59 a.m. (Nov. 7, 2014, 10:59 a.m.)
```
We'll be posting a patch to correct this issue shortly.
```

You have a pending review.

Review Board 2.0.15

This change has been marked as submitted.

Screenshots

Files

Description:

Changeset 10426:c5ec71d6ca6f

Changeset 10440:faea9dd0b3b1

Diff:

Status: Closed (submitted)