Information
Submitter:	Abdul Mutaal Ahmad
Repository:	gem5
Branch:
Bugs:
Depends On:
Reviewers
Groups:	Default
People:

Description

In this new hmc configuration we have used the existing components in gem5 mainly [SerialLink] [NoncoherentXbar]& [DRAMCtrl] to define 3 different architecture for HMC.

Highlights

1- It explores 3 different HMC architectures
2- It creates 4-HMC crossbars and attaches 16 vault controllers with it. This will connect vaults to serial links
3- From the previous version, HMCController with round robin funtionality is being removed and all the serial links are being accessible directly from user ports
4- Latency incorporated by HMCController (in previous version) is being added to SerialLink

Testing Done

Current version only works for traffic generator setup {can be extended for full-system and system-emulation}

How to run it

from you gem5 home directory run the following commands

build/ARM/gem5.opt configs/example/hmctest.py --mem-type=HMC_2500_x32 --mode=RANDOM --arch=same
build/ARM/gem5.opt configs/example/hmctest.py --mem-type=HMC_2500_x32 --mode=RANDOM --arch=distributed
build/ARM/gem5.opt configs/example/hmctest.py --mem-type=HMC_2500_x32 --mode=RANDOM --arch=mixed

Dependency

This patch required another tester patch to be applied before running the test

http://reviews.gem5.org/r/3374/

Issue Summary

Description	From	Last Updated	Status
present	Stephan Diestelhorst	May 19, 2016, 5:34 a.m.	Resolved
covers	Stephan Diestelhorst	May 19, 2016, 5:34 a.m.	Resolved
access complete	Stephan Diestelhorst	May 19, 2016, 5:34 a.m.	Resolved
What is the significance of "modified" here?	Stephan Diestelhorst	May 19, 2016, 5:35 a.m.	Resolved
spurious space	Stephan Diestelhorst	May 19, 2016, 5:35 a.m.	Resolved
What is the unit here?	Stephan Diestelhorst	May 19, 2016, 5:35 a.m.	Resolved
spaces around operators.	Stephan Diestelhorst	May 19, 2016, 5:35 a.m.	Resolved
are these three cases not the same?	Stephan Diestelhorst	June 29, 2016, 4:51 a.m.	Resolved
I am not sure I understand the special case, here.	Stephan Diestelhorst	May 19, 2016, 5:35 a.m.	Resolved
merge these two	Stephan Diestelhorst	May 19, 2016, 5:35 a.m.	Resolved
loop and lists of buffers?	Stephan Diestelhorst	June 29, 2016, 4:51 a.m.	Resolved
who wires up things if arch == 1? Please also add the arch descriptions inthe comments.	Stephan Diestelhorst	June 29, 2016, 4:51 a.m.	Resolved
unit?	Stephan Diestelhorst	May 19, 2016, 5:35 a.m.	Resolved
unit?	Stephan Diestelhorst	May 19, 2016, 5:35 a.m.	Resolved
where do you actually increase the packet size?	Stephan Diestelhorst	May 19, 2016, 5:35 a.m.	Resolved
Is this param used?	Radhika Jagtap	June 29, 2016, 4:27 a.m.	Resolved
space around operator	Radhika Jagtap	June 29, 2016, 4:27 a.m.	Resolved
Here and elsewhere in the changed lines: space around operators (, = == etc.) Also, 0 is the default start ...	Radhika Jagtap	June 29, 2016, 4:27 a.m.	Resolved
Pass meaning 'do nothing' instead of None meaning 'no object'. You could do if i != j instead of placing ...	Radhika Jagtap	June 29, 2016, 4:28 a.m.	Resolved
Can these two params be set during instantiation of the Bridge objects, and only set the ranges inside this loop? ...	Radhika Jagtap	June 29, 2016, 4:28 a.m.	Resolved
Change the arch option names.	Radhika Jagtap	June 29, 2016, 4:28 a.m.	Resolved

Dear Abdul,
I disagree with breaking the crossbar into 4 smaller crossbar, because I am quite sure that this is not how it works in the HMC.
Maybe if you could modify the crossbar into a hierarchical-crossbar with non-uniform access latencies to the local and non-local quadrants that would make much more sense.
However having 4 independent crossbars not correct, and will limit the functionality of the serial links: Chaining of multiple HMC devices won't be possible because the address ranges of the serial links would be different. Also link power down cannot be implemented, because again links deliver different address ranges.

Finally, about the "round robin" HMC controller. Again I disagree with removing it. Because something should be there to connect 1 host to more than one serial link. It would be great if you could provide a better implementation for an HMC controller. But I believe that this component is necessary otherwise the model won't be an HMC device anymore.

Thank you,

Andreas Hansson April 1, 2016, 9:21 a.m. (April 1, 2016, 9:21 a.m.)

Two things:

1) The round robin is just spreading traffic across multiple links. The hashing and interleaving built in to the crossbar does exactly the same. Agreed? There is no difference in functionality, and the 1 host with multiple links is still working just fine.

2) The four crossbars is just making it multi hop. The overall address range distribution should be exactly the same. It is effectively just a way of creating non-uniform latencies within and between quadrants. Chaining should work just as before.

Link power down is an interesting use-case that indeed gets more tricky since effectively the address mapping has to change. I would say this is the only usecase that really warrants a separate host-side "HMC controller". You could argue that it's easier to modulate each link (lane scaling) as opposed to link scaling, but I see your point here. Do we really care about that use-case though?

Erfan Azarkhish April 2, 2016, 2:16 p.m. (April 2, 2016, 2:16 p.m.)

Dear Andreas,

I agree that round-robin itself does not make any sense, and hashing is a better solution.
I also like that you have added a link-aggregator, but when I set the --linkaggr=1, nothing happens and when I looked into the patch, I did not see any implementation for a link aggregator.
Also, it is great that Abdul supports two architectures (I prefer the second one). May be if he could just add some comments, references, and explanation in the code it would be great.

I won't argue if you remove the HMC controller, but please don't forget that, the host protocol is different from the HMC protocol, and some component is required between the host XBar and the serial links. This component for sure has some static latency + some queueing latency. And it should have a large queue size, because HMC is a component with large access latency.
Maybe a Link Aggregator component can be enough for now.

Finally, as you say, the link power management can be ignored for now, but some explanation can be added to the code.

Abdul Mutaal Ahmad April 18, 2016, 3:45 a.m. (April 18, 2016, 3:45 a.m.)

Hi All,

Regarding linkaggr, I added it in tester. I will probably remove it from the tester in next update. I will add more comments regarding each architecture in the next release.I agree to Andreas, that link-power is a 'use-case' which can be ignored right now. Probably it can be added in next releases with retry functionality etc. Any further improvements? or comments?

Change Summary:

In this version, we have added comments and references to previous work. Also each vault in HMC is configured to have different memory range.

Diff:

Revision 2 (+320 -99)

Show changes

	configs/common/HMC.py
	configs/common/MemConfig.py

Change Summary:

Serial Link model has been updates with link speed. The latency over a serial link is calculated using link_speed and num_lanes. This is done because we want to keep the Serial Link as one component and which models both controller latency + serial (serDes) latency. The clock in this case is the controller frequency.

Diff:

Revision 3 (+340 -106)

Show changes

	configs/common/HMC.py
	configs/common/MemConfig.py
	src/mem/SerialLink.py
	src/mem/serial_link.hh
	src/mem/serial_link.cc

Thanks for getting these out! However, there are a few low-level issues, and some funky wiring that goes on here.

configs/common/HMC.py (Diff revision 3)

present

Show all issues

configs/common/HMC.py (Diff revision 3)

covers

Show all issues

configs/common/HMC.py (Diff revision 3)

access complete

Show all issues

configs/common/HMC.py (Diff revision 3)

What is the significance of "modified" here?

Show all issues

configs/common/HMC.py (Diff revision 3)

spurious space

Show all issues

configs/common/HMC.py (Diff revision 3)

What is the unit here?

Show all issues

configs/common/HMC.py (Diff revision 3)

spaces around operators.

Show all issues

configs/common/HMC.py (Diff revision 3)

are these three cases not the same?

Show all issues

Abdul Mutaal Ahmad May 19, 2016, 5:35 a.m. (May 19, 2016, 5:35 a.m.)

I can do that but the tester has be changed according. so it configures memory range for each arch accordingly.

configs/common/HMC.py (Diff revision 3)

I am not sure I understand the special case, here.

Show all issues

configs/common/HMC.py (Diff revision 3)

merge these two

Show all issues

configs/common/HMC.py (Diff revision 3)

loop and lists of buffers?

Show all issues

configs/common/HMC.py (Diff revision 3)

who wires up things if arch == 1? Please also add the arch descriptions inthe comments.

Show all issues

Abdul Mutaal Ahmad May 19, 2016, 5:35 a.m. (May 19, 2016, 5:35 a.m.)

for arch1, we don't any buffers. There are comments on each arcitecture at the very top of this file.

src/mem/SerialLink.py (Diff revision 3)

unit?

Show all issues

src/mem/serial_link.hh (Diff revision 3)

unit?

Show all issues

src/mem/serial_link.cc (Diff revision 3)

where do you actually increase the packet size?

Show all issues

Change Summary:

some modifications to improve code readability

Diff:

Revision 4 (+319 -104)

Show changes

	configs/common/HMC.py
	configs/common/MemConfig.py
	src/mem/SerialLink.py
	src/mem/serial_link.hh
	src/mem/serial_link.cc

Change Summary:

changed the logic of linking crossbars within loops, added more comments

Diff:

Revision 5 (+301 -104)

Show changes

	configs/common/HMC.py
	configs/common/MemConfig.py
	src/mem/SerialLink.py
	src/mem/serial_link.hh
	src/mem/serial_link.cc

Hi Abdul,

The code is now more compact after your recent updates and additional comments help. A few style and readability issues still remain, but getting there.

Radhika

configs/common/HMC.py (Diff revision 5)

Is this param used?

Show all issues

configs/common/HMC.py (Diff revision 5)

Purely for readability, how about xbar_buffer_size_* instead of cross_buffer_size_*?

configs/common/HMC.py (Diff revision 5)

space around operator

Show all issues

Abdul Mutaal Ahmad June 20, 2016, 8:33 a.m. (June 20, 2016, 8:33 a.m.)
```
on every operator ? * or + etc ?
```

Radhika Jagtap June 20, 2016, 8:37 a.m. (June 20, 2016, 8:37 a.m.)

As per coding guidlines: http://www.m5sim.org/Coding_Style
one space around binary operators (+, -, <, >, etc.) including assignment operators (=, +=, etc.)

configs/common/HMC.py (Diff revision 5)

Here and elsewhere in the changed lines: space around operators (, = == etc.)

Also, 0 is the default start for the range function.

Show all issues

configs/common/HMC.py (Diff revision 5)

Pass meaning 'do nothing' instead of None meaning 'no object'. You could do if i != j instead of placing the work in the else clause?

Show all issues

configs/common/HMC.py (Diff revision 5)

Can these two params be set during instantiation of the Bridge objects, and only set the ranges inside this loop? But do retain the comment currently there for the None statement.

Show all issues

Change Summary:

made the code more readable

Diff:

Revision 6 (+299 -105)

Show changes

	configs/common/HMC.py
	configs/common/MemConfig.py
	src/mem/SerialLink.py
	src/mem/serial_link.hh
	src/mem/serial_link.cc

Change Summary:

changed the run script

Testing Done:

		Current version only works for traffic generator setup {can be extended for full-system and system-emulation}

		How to run it

		from you gem5 home directory run the following commands

~		build/ARM/gem5.opt configs/example/hmctest.py --mem-type=HMC_2500_x32 --mode=RANDOM --arch=0
~		build/ARM/gem5.opt configs/example/hmctest.py --mem-type=HMC_2500_x32 --mode=RANDOM --arch=1
~		build/ARM/gem5.opt configs/example/hmctest.py --mem-type=HMC_2500_x32 --mode=RANDOM --arch=2
	~	build/ARM/gem5.opt configs/example/hmctest.py --mem-type=HMC_2500_x32 --mode=RANDOM --arch=same
	~	build/ARM/gem5.opt configs/example/hmctest.py --mem-type=HMC_2500_x32 --mode=RANDOM --arch=distributed
	~	build/ARM/gem5.opt configs/example/hmctest.py --mem-type=HMC_2500_x32 --mode=RANDOM --arch=mixed

		Dependency

		This patch required another tester patch to be applied before running the test

		http://reviews.gem5.org/r/3374/

Good to go from my side after addressing the minor comment. But I think a few more people should review it too.

configs/common/HMC.py (Diff revision 6)

Change the arch option names.

Show all issues

Change Summary:

Updated comments

Diff:

Revision 7 (+300 -105)

Show changes

	configs/common/HMC.py
	configs/common/MemConfig.py
	src/mem/SerialLink.py
	src/mem/serial_link.hh
	src/mem/serial_link.cc

You have a pending review.

Review Board 2.0.15

This change has been marked as submitted.

mem: different HMC configuration

In this new hmc configuration we have used the existing components in gem5 mainly [SerialLink] [NoncoherentXbar]& [DRAMCtrl] to define 3 different architecture for HMC.

Highlights

Current version only works for traffic generator setup {can be extended for full-system and system-emulation}

How to run it

Dependency

Screenshots

Files

Issue Summary

Change Summary:

Diff:

Change Summary:

Diff:

Change Summary:

Diff:

Change Summary:

Diff:

Change Summary:

Diff:

Change Summary:

Testing Done:

Current version only works for traffic generator setup {can be extended for full-system and system-emulation}

How to run it

Dependency

Change Summary:

Diff:

Status: Closed (submitted)