MEM: Enable multiple distributed generalized memories

Information
Submitter:	Andreas Hansson
Repository:	gem5
Branch:	default
Bugs:
Depends On:
Reviewers
Groups:	Default
People:

Description

MEM: Enable multiple distributed generalized memories

This patch removes the assumption on having on single instance of
PhysicalMemory, and enables a distributed memory where the individual
memories in the system are each responsible for a single contiguous
address range.

All memories inherit from an AbstractMemory that encompasses the basic
behaviuor of a random access memory, and provides untimed access
methods. What was previously called PhysicalMemory is now
SimpleMemory, and a subclass of AbstractMemory. All future types of
memory controllers should inherit from AbstractMemory.

To enable e.g. the atomic CPU and RubyPort to access the now
distributed memory, the system has a wrapper class, called
PhysicalMemory that is aware of all the memories in the system and
their associated address ranges. This class thus acts as an
infinitely-fast bus and performs address decoding for these "shortcut"
accesses. Each memory can specify that it should not be part of the
global address map (used e.g. by the functional memories by some
testers). Moreover, each memory can be configured to be reported to
the OS configuration table, useful for populating ATAG structures, and
any potential ACPI tables.

Checkpointing support currently assumes that all memories have the
same size and organisation when creating and resuming from the
checkpoint. A future patch will enable a more flexible
re-organisation.

Testing Done

util/regress all passing (disregarding t1000 and eio)

Issue Summary

Description	From	Last Updated	Status
i don't think this should be conf table reported	Ali Saidi	April 1, 2012, 4:25 a.m.	Resolved
this looks like it's wrong	Ali Saidi	March 21, 2012, 7:26 p.m.	Resolved
when would you have a memory that wasn't?	Ali Saidi	March 21, 2012, 8:06 p.m.	Dropped
wrong style and below	Ali Saidi	March 21, 2012, 7:37 p.m.	Resolved
I wonder about caching this information vs the indirection. Since it's in the fetch paths it's going to get called ...	Ali Saidi	April 2, 2012, 7:34 a.m.	Resolved
might want to consider caching the latest start/end here to avoid the constant lookup.	Ali Saidi	April 2, 2012, 7:34 a.m.	Resolved
Does memRanges only support continous segments? Would it be possible to support physical pages striped across all NUMA memories?	Brad Beckmann	April 5, 2012, 4:05 a.m.	Dropped

Seems like a reasonable start. i have same question below and would like a better description of the options.

Andreas Hansson March 21, 2012, 8:06 p.m. (March 21, 2012, 8:06 p.m.)

I assume the description you are looking for is for the parameters of the AbstractMemory? I can either spill over the 80 char or add the descriptions as code comments in the Python code. Preferences?

src/dev/arm/RealView.py (Diff revision 1)

i don't think this should be conf table reported

Show all issues

Andreas Hansson March 21, 2012, 8:06 p.m. (March 21, 2012, 8:06 p.m.)

Have I got this wrong then? This is used by the arm/linux/system.cc to populate the AtagMem struct, and if this is not "conf_table_reported" (or what ever we choose to call this property of a memory) then the realview regressions fail.

Are you suggesting a name change for the parameter?

src/dev/arm/RealView.py (Diff revision 1)

this looks like it's wrong

Show all issues

Andreas Hansson March 21, 2012, 8:06 p.m. (March 21, 2012, 8:06 p.m.)
```
Indeed, thanks.
```

src/mem/AbstractMemory.py (Diff revision 1)

when would you have a memory that wasn't?

Show all issues

Andreas Hansson March 21, 2012, 8:06 p.m. (March 21, 2012, 8:06 p.m.)

The memtesters use a "reference" memory that shadows the normal memory system. These memories are overlapping in address ranges and should not appear in the global address map.

src/mem/abstract_mem.hh (Diff revision 1)

wrong style and below

Show all issues

Andreas Hansson March 21, 2012, 8:06 p.m. (March 21, 2012, 8:06 p.m.)
```
Thanks. Fixed.
```

Diff:

Revision 2 (+1437 -959)

Show changes

	configs/common/FSConfig.py
	configs/example/memtest.py
	configs/example/ruby_direct_test.py
	configs/example/ruby_mem_test.py
	configs/example/ruby_network_test.py
	configs/example/ruby_random_test.py
	configs/example/se.py
	configs/ruby/MESI_CMP_directory.py
	51 more

src/cpu/simple/atomic.cc (Diff revision 2)

I wonder about caching this information vs the indirection. Since it's in the fetch paths it's going to get called a lot.

Show all issues

src/mem/physical.cc (Diff revision 2)

might want to consider caching the latest start/end here to avoid the constant lookup.

Show all issues

Andreas Hansson April 2, 2012, 9:55 p.m. (April 2, 2012, 9:55 p.m.)

Basic caching is added and a new diff is uploaded. For the issue pointed out in the atomic cpu, I assume that it is still the best option to rely on the caching in the physical memory class. Agreed?

Diff:

Revision 3 (+1452 -954)

Show changes

	configs/common/FSConfig.py
	configs/example/memtest.py
	configs/example/ruby_direct_test.py
	configs/example/ruby_mem_test.py
	configs/example/ruby_network_test.py
	configs/example/ruby_random_test.py
	configs/example/se.py
	configs/ruby/MESI_CMP_directory.py
	51 more

Ship It!

Hi Andreas,

Sorry it took me a couple weeks to look this patch over.  It is an impressive amount of work.  My biggest concern is it appears that this patch will break all existing checkpoints, correct?  Your comment states that a later patch will enable checkpoint migration between different NUMA layouts, but will it also support checkpoints created prior to this change?  Can you delay checking this patch in until that later patch with checkpoint support is ready to be checked in?

Thanks

Andreas Hansson April 4, 2012, 5:36 p.m. (April 4, 2012, 5:36 p.m.)

Hi Brad,

Thanks :)

The checkpointing will continue to work just as usual with all your existing systems, as they only use one physmem, and that is not going to cause you any problems. At the moment, the serialize/unserialize is still part of each individual memory, so as long as they are not modified before/after the checkpoint it is "business as usual" even with multiple memories. Therefore I'd prefer to get this patch out as it is.

Later on, if we want to think about potential features to add, one would be to change the number and type of memories for different distributed memory layouts. That would require moving some intelligence into the "PhysicalMemory" wrapper now as it would have to "distribute" the state to the appropriate memories. This is an added function that should be a separate patch and I will try and add support for this on a "need to have" basis as there is currently a lot in the pipeline. I don't see it as critical to the current patch as it maintains current functionality.

Similarly, for the striping, I'm currently looking at how to improve the support for this in gem5. This will also be a separate patch and added later.

I hope that all makes sense and that we can commit this as it is.

Brad Beckmann April 5, 2012, 5 a.m. (April 5, 2012, 5 a.m.)

I see, so despite replacing the PhysicalMemory instantiations with the SimpleMemory instantiations, everything will still work because the config name of SimpleMemory is still physmem.  Ok, if you've already checked that existing checkpoints work with this change (and I now suspect they do) then go ahead and check this in.

Btw, in case you're not aware, some Ruby protocols support numa stripping right now via the numa_high_bit option.

Andreas Hansson April 5, 2012, 7:06 a.m. (April 5, 2012, 7:06 a.m.)

The checkpointng remains unaffected for now, and can be extended later, just as the address mapping.

If you're fine with this then I assume I can ship it?

src/sim/system.cc (Diff revision 3)

Does memRanges only support continous segments?  Would it be possible to support physical pages striped across all NUMA memories?

Show all issues

Ship It!

You have a pending review.

Review Board 2.0.15

This change has been marked as submitted.

Screenshots

Files

Issue Summary

Diff:

Diff:

Status: Closed (submitted)