Review Board 2.0.15


util/regress: make default action a more thorough regression

Review Request #649 - Created April 20, 2011 and submitted

Information
Steve Reinhardt
gem5
Reviewers
Default
ali, gblack, nate, stever 
util/regress: make default action a more thorough regression

Changed the --variants option to --test-variants and added a new
--compile-variants option for variants that are only compiled
(not tested).  The former still defaults to 'opt' and the latter
defaults to 'debug,fast'.

Also changed the behavior when no tests are specified from just
compiling to running the 'quick' tests.

As a result, a plain 'util/regress' invocation will now compile
(but not test) the debug and fast builds, and compile and run the
quick regressions on the opt build.  This should be the default
set of tests that are run before committing.  Since the nightly
regressions use this same script, this will also be the new
nightly regression behavior.

Test-only regressions can still be done by setting --compile=''.
Compile-only regressions can be done by setting --test=''.

   

  


 


 

  

  









  
 


  

 


















 

  


 


  



   

   
Gabe Black

   
Posted  (April 21, 2011, 6:18 a.m.)


  


  


  

   
I like this change from a design perspective, and I didn't see anything obviously wrong with the code. I'll assume you've identified and fixed all possible bugs. :-)



  

   

   
    
    
   


  




   







    

     
     

      

       
        

         
          util/regress
          

           (Diff revision 1)

          
         
        

       
       
        
 


        
 


       
      

     

    

    


     
This hack may no longer be necessary. This is outside the scope of your change, but I thought I'd point it out while I'm thinking of it.




  

   

   
    
    
   


  


    


   



  

 


  

 

















 



 

  


 


  



   

   
Nathan Binkert

   
Posted  (April 21, 2011, 6:37 a.m.)


  


  


  

   
I'm going to also send you my code for building.  I'd like to incorporate some of the ideas. So I can import regress (which would mean creating build.py or something like that) and use it directly in my environment.  (We can do that as a separate diff though).



  

   

   
    
    
   


  




   







    

     
     

      

       
        

         
          util/regress
          

           (Diff revision 1)

          
         
        

       
       
        
 


        
 


       
      

     

    

    


     
While you're at it you could add the following to the help:

(0 to use total number of cores)




  

   

   
    
    
   


  


    


    

     
     

      

       
        

         
          util/regress
          

           (Diff revision 1)

          
         
        

       
       
        
 


        
 


       
      

     

    

    


     
Another common option is --update-ref, can we add that here and pass it through?




  

   

   
    
    
   


  


    


    

     
     

      

       
        

         
          util/regress
          

           (Diff revision 1)

          
         
        

       
       
        
 


        
 


       
      

     

    

    


     
I have nicer code for this sort of thing.  You can see it in src/python/m5/options.py, examples of usage in main.py

Look for action='append' (and notice the necessity of split)




  

   

   
    
    

   
  1. 
    
    
     
    
      
     
    
     
    So I played around with this, and the first problem is that your current code can't handle default values.  I fixed that in options.py, but it still doesn't print the defaults nicely, e.g., instead of:

  --test-variants=TEST_VARIANTS
                        comma-separated build variants to test (default:
                        'opt')
  --compile-variants=VARIANTS
                        comma-separated build variants to compile only (not
                        test) (default: 'debug,fast')

we get:

--test-variants=TEST_VARIANTS
                        comma-separated build variants to test (default:
                        '['opt']')
--compile-variants=VARIANTS
                        comma-separated build variants to compile only (not
                        test) (default: '['debug', 'fast']')

Not tragic, but I like the former better, so I may keep the original solution.

    
    
    
   
    2. 


   
  2. 
    
    
     
    
      
     
    
     
    On further consideration, 'append' is not even what I want, if the user gives a list I want it to replace (not append to) any existing list.
    
    
    
   
    3. 

   


  


    


    

     
     

      

       
        

         
          util/regress
          

           (Diff revision 1)

          
         
        

       
       
        
 


        
 


       
      

     

    

    


     
Seems like we should fix this on the SCons side.  Is that difficult?  Also, does NOISA have any tests?  Seems like we could have a memory tester for NOISA.