ruby: Fix checkpointing and restore

Information
Submitter:	Timothy Jones
Repository:	gem5
Branch:	default
Bugs:
Depends On:
Reviewers
Groups:	Default, Ruby
People:

Description

There are 2 problems with the existing checkpoint and restore code in ruby.
The first is that when the event queue is altered by ruby during serialization,
some events that are currently scheduled cannot be found (e.g. the event to
stop simulation that always lives on the queue), causing a panic. The second
is that ruby is sometimes serialized after the memory system, meaning that the
dirty data in its cache is flushed back to memory too late and so isn't
included in the checkpoint.

These are fixed by implementing memory writeback in ruby, using the same
technique of hijacking the event queue, but first descheduling all events that
are currently on it. They are saved, along with their scheduled time, so that
the event queue can be faithfully reconstructed after writeback has finished.
Writeback is still implemented using flushing, so the cache recorder object,
that is created to generate the trace and manage flushing, is kept around and
used during serialization to write the trace to disk.

Testing Done

Issue Summary

Description	From	Last Updated	Status
I am not a fan. This should be solved by the draining logic. Once the object claims to be drained, ...	Andreas Hansson	July 1, 2015, 12:45 p.m.	Resolved

Status: Discarded

Status: Re-opened

Summary:

+	ruby: Fix checkpointing and restore

Description:

		ruby: Fix checkpointing and restore

		There are 2 problems with the existing checkpoint and restore code in ruby. The first is that the event queue is altered by ruby during serialization, meaning that the event to stop simulation that always lives on the queue can't be found, causing a panic. This is fixed by explicitly descheduling it before swapping events off the main queue.

~		The happens occasionally when ruby is serialized after the memory system. In this case the dirty data in ruby's caches is flushed back to memory too late and so isn't included in the checkpoint. This is fixed by adding serialization priorities, thus ensuring that ruby can be serialized first, perform its flush, and make sure memory has the most up-to-date data before it is checkpointed too.
	~	The other happens occasionally when ruby is serialized after the memory system. In this case the dirty data in ruby's caches is flushed back to memory too late and so isn't included in the checkpoint. This is fixed by adding serialization priorities, thus ensuring that ruby can be serialized first, perform its flush, and make sure memory has the most up-to-date data before it is checkpointed too.

Diff:

Revision 1 (+69 -5)

	src/mem/ruby/system/CacheRecorder.cc
	src/mem/ruby/system/System.cc
	src/sim/sim_object.hh
	src/sim/sim_object.cc

src/mem/ruby/system/System.cc (Diff revision 1)

I am not a fan. This should be solved by the draining logic. Once the object claims to be drained, the order of serialisation should not matter.

Show all issues

I do not think this is the way to go. There is already an established methodology to solve the issue.

Timothy Jones June 24, 2015, 1:06 a.m. (June 24, 2015, 1:06 a.m.)

I don't think draining is the way forward, but there could well be other, better solutions than the one I've got here. My aim was to get this working again, which is does, but I'm happy now to rework it into something more acceptable.

From my understanding, draining is there to remove transient state from objects. In the classic memory model caches keep dirty data around after draining, so I don't see why ruby shouldn't either.

However, in the classic model, a call to memWriteback() is made to flush data back to memory before checkpointing. Perhaps it would be better if I implemented ruby's flushing in a similar function? The main problem with that is that ruby's flushing is tied up with creating a trace for checkpointing, so if someone wanted to call memWriteback() without creating then creating a checkpoint, then they'd end up with an unnecessary trace file.

Andreas Sandberg June 24, 2015, 2:17 a.m. (June 24, 2015, 2:17 a.m.)

I agree with Andreas here. Having checkpoint priorities seems like a really bad idea.

You're right that the expectation is that drain does not flush or write back caches. The simulator is drained when switching CPU models (there are cases where we write back and invalidate caches here, but I don't think those are supported by Ruby) and you really don't want to lose your cache in that case.

The flow when taking a checkpoint is slightly different since we assume that caches in the classic memory system can't be checkpointed. This means that the simulator goes through the following flow:
* drain() (called multiple times until the whole simulator is drained)
* memWriteback()
* serialize()
* drainResume()

I think the right solution here would be to write back dirty data to the memory system in memWriteback() and create the trace in serialize(). This would ensure that the backing store is in sync with Ruby when you take the checkpoint. Note that you aren't required to change the state of dirty cache lines in memWriteback(), this would allow you to retain the old state in the trace.

Timothy Jones June 24, 2015, 2:36 a.m. (June 24, 2015, 2:36 a.m.)

I don't think draining priorities are necessarily a bad idea, it's just that we want to incorporate these fixes into the existing mechanisms if at all possible.

I think we probably need a ruby person to comment on this. The issue is that the flushing is based on the trace that is generated. So we can either:

1) Create the trace twice (once in memWriteback() for flushing, then again in serialize() for writing out)
2) Create the trace in memWriteback() but keep it hanging around for the later serialize() call (but what if this is never called...)

Neither strikes me as an ideal solution, so perhaps I'm missing something that someone with better knowledge of ruby can spot.

Change Summary:

Moved code to flush ruby data back to memory into new memWriteback() method,
which removes the need for serialization priorities.

Description:

		ruby: Fix checkpointing and restore

~		There are 2 problems with the existing checkpoint and restore code in ruby. The first is that the event queue is altered by ruby during serialization, meaning that the event to stop simulation that always lives on the queue can't be found, causing a panic. This is fixed by explicitly descheduling it before swapping events off the main queue.
	~	There are 2 problems with the existing checkpoint and restore code in ruby.
	+	The first is that when the event queue is altered by ruby during serialization,
	+	some events that are currently scheduled cannot be found (e.g. the event to
	+	stop simulation that always lives on the queue), causing a panic. The second
	+	is that ruby is sometimes serialized after the memory system, meaning that the
	+	dirty data in its cache is flushed back to memory too late and so isn't
	+	included in the checkpoint.

~		The other happens occasionally when ruby is serialized after the memory system. In this case the dirty data in ruby's caches is flushed back to memory too late and so isn't included in the checkpoint. This is fixed by adding serialization priorities, thus ensuring that ruby can be serialized first, perform its flush, and make sure memory has the most up-to-date data before it is checkpointed too.
	~	These are fixed by implementing memory writeback in ruby, using the same
	+	technique of hijacking the event queue, but first descheduling all events that
	+	are currently on it. They are saved, along with their scheduled time, so that
	+	the event queue can be faithfully reconstructed after writeback has finished.
	+	Writeback is still implemented using flushing, so the cache recorder object,
	+	that is created to generate the trace and manage flushing, is kept around and
	+	used during serialization to write the trace to disk.

Diff:

Revision 2 (+114 -65)

Show changes

	src/mem/ruby/system/CacheRecorder.cc
	src/mem/ruby/system/System.hh
	src/mem/ruby/system/System.cc
	src/sim/eventq.hh

Thanks for getting this in shape Tim! Much nicer.

Timothy Jones July 2, 2015, 1:13 a.m. (July 2, 2015, 1:13 a.m.)

Thanks Andreas. I'd just like to get comments from a ruby person too before I
push. Nilay, Brad, Joel, any thoughts?

Looks good to me. Thanks for tackling this!

What did you use to test this? It would be wonderful if we had a regression script for Ruby and checkpointing! No need to hold up this bugfix patch for it, but if you could cobble a test together quickly it would be nice to have. If not, let me know how you are testing this patch and I might put something together.

Timothy Jones July 3, 2015, 1:36 a.m. (July 3, 2015, 1:36 a.m.)

Hi Jason,

Unfortunately it was a bit long-winded to test. I had to boot Linux with 2 cores and start running x264 from Parsec, taking the checkpoint at the ROI. I couldn't seem to get the second error to happen earlier (e.g. at the start of OS booting) but I didn't try extensively. Theoretically, it should have manifested itself as soon as there was dirty data in the cache and a checkpoint was taken. I'm happy to share my kernel and disk image, if necessary, but perhaps this is overkill?

Cheers
Tim

Jason Lowe-Power July 3, 2015, 7:41 a.m. (July 3, 2015, 7:41 a.m.)

OK. It's unfortunate that things like this are so hard to track down. Maybe a solution is to test checkpointing while running the Ruby tester? I'll put it on the list of "things it would be nice to do if I never slept" ;).

Ship It!

Hi Tim,

Sorry to come back to this patch, but I just applied it and tried to test it and ran into a problem. When restoring the original event queue in line 187 of System.cc, I get an error that the event is already on the event queue. Below is how I ran into the problem:

scons build/X86_MOESI_hammer/gem5.opt -j5 --default=X86 PROTOCOL=MOESI_hammer
build/X86_MOESI_hammer/gem5.opt configs/example/fs.py --ruby --cpu-type=detailed -m 4118117000 --checkpoint-at-end

Am I missing some other patch that is also needed in conjuction with this one?

Timothy Jones July 4, 2015, 6:24 a.m. (July 4, 2015, 6:24 a.m.)

Hm, this sounds quite like one of the issues I was having and I thought I had fixed. Could you tell me which disk image and kernel you were using and I'll see what's going on?

Jason Lowe-Power July 6, 2015, 7:15 a.m. (July 6, 2015, 7:15 a.m.)

I'd be surprised if my kernel and disk were the difference here, since it is a problem in checkpointing. I chose a totally random tick to checkpoint at (about 30 seconds into simulation). I think the problem is checkpointing after using --cpu-type=detailed. Running the same test with --cpu-type=timing seems to work.

Timothy Jones July 7, 2015, 2:44 a.m. (July 7, 2015, 2:44 a.m.)

I agree - I didn't mean to imply that it's your kernel or disk, just that I wanted to get the setup as similar as possible. For me, you see, this is all fine. I tried with several different random tick values for checkpointing too. I used your command lines above and here is my restore commmand line:

build/X86_MOESI_hammer/gem5.opt configs/example/fs.py --ruby --cpu-type=detailed -m 10000000000 --restore-with detailed -r 0

Timothy Jones July 8, 2015, 1:34 p.m. (July 8, 2015, 1:34 p.m.)

Jason and I worked on this offline. I managed to reproduce his error and have posted a fix in review request #2951

You have a pending review.

Review Board 2.0.15

This change has been marked as submitted.

Screenshots

Files

Issue Summary

Status: Discarded

Status: Re-opened

Summary:

Description:

Diff:

Change Summary:

Description:

Diff:

Status: Closed (submitted)