Review Board 2.0.15


python: Add mechanism to override code compiled into the exectuable

Review Request #29 - Created July 5, 2010 and submitted

Information
Nathan Binkert
gem5
Reviewers
Default 
python: Add mechanism to override code compiled into the exectuable
If the user sets the environment variable M5_OVERRIDE to True, then
imports that would normally find python code compiled into the executable
will instead first check in the absolute location where the code was
found during the build of the executable.  This only works for files
in the src (or extras) directories, not automatically generated files.

This is a developer feature!

   

  


 


 

  

  









  
 


  

 
















 



 

  


 


  



   

   
Steve Reinhardt

   
Posted  (July 5, 2010, 2:01 p.m.)


  


  


  

   



  

   

   
    
    
   


  




   







    

     
     

      

       
        

         
          src/SConscript
          

           (Diff revision 1)

          
         
        

       
       
        
 


        
 


       
      

     

    

    


     
What is srcpath used for?  I see it getting set here and then passed around a lot but I don't see it getting used.




  

   

   
    
    

   
  1. 
    
    
     
    
      
     
    
     
    Actually, I was going to support an override where a path was specified and the relative path would then be appended, but I thought that it was overkill to do that.  I could remove this code.  This brings up another question.  Right now, when you get an error message, that message uses arcname as the filename.  arcname is basically the path that the file would be if it were in a zip archive, and it has no real relationship to where the file came from this seems to have confused people in the past.  We could instead use something like srcname (with the target name if there is no source file).  What do you think?
    
    
    
   
    2. 


   
  2. 
    
    
     
    
      
     
    
     
    If using srcname in error messages makes more sense, that's fine by me.  To the extent that srcpath isn't being used though, I'd get rid of it.

Not to get too far off topic, but python error handling in general could be cleaned up... there are cases where you run into a straightforward problem, sometimes even with a decent error message, but an exception gets raised and we get a stack backtrace that totally overwhelms the error message.  Seems like we should try to catch some common exceptions at the top level to avoid that.  If we did, then maybe we could move toward using exceptions to flag all kinds of errors rather than trying to duplicate fatal() and panic() in python.

    
    
    
   
    3. 


   
  3. 
    
    
     
    
      
     
    
     
    Well, I'm not sure if it makes more sense or not to be honest.  I guess I'll just leave things as they are.

I'd like to improve exception handling as well, but I'm not sure how exactly how.  We could just catch all exceptions and just print the error message unless some --python-backtrace were specified on the commandline.  Not sure how much better we could do.
    
    
    
   
    4. 

   


  


    


    

     
     

      

       
        

         
          src/python/importer.py
          

           (Diff revision 1)

          
         
        

       
       
        
 


        
 


       
      

     

    

    


     
I think a more descriptive/specific env var is called for... how about M5_USE_PY_SOURCE?




  

   

   
    
    

   
  1. 
    
    
     
    
      
     
    
     
    M5_OVERRIDE_PY_SOURCE?  I'm not sure yours really explains what's going on either.  I should clearly add a comment somewhere.  I could also hack this in as a command line option, though that'd mean doing some command line parsing in C++, which may not be so bad since I could just search argv for the specific argument.  (The argument could be parsed in C++, but it would also be a no-op argument in python and would get a help string as a result.)
    
    
    
   
    2. 


   
  2. 
    
    
     
    
      
     
    
     
    It explains it to me since I think of the "source" as the original source file/tree, and not some compressed copy squirreled away somewhere, but I can see if you take a broader view of "source" as anything that's not compiled then maybe it's still not too specific.  I'm not convinced your idea is a lot better, since it doesn't explain what's overriding what.  How about M5_OVERRIDE_EMBEDDED_PYTHON_ARCHIVE_WITH_FILES_FROM_SOURCE_TREE?  :-)

I'm fine with the env var, it's just a developer hack as you say, no need to get too crazy with the code.

    
    
    
   
    3. 


   
  3. 
    
    
     
    
      
     
    
     
    I guess I think of source as a synonym for code (i.e. any code, generated or not).  I'll figure something out.
    
    
    
   
    4.