9# Weekly Report [2011/07/18 – 2011/07/24]

Good news is that a simple patch aiming to add automatic script generation functionality to packaging can work now. Current patch can create executable script with no extension on POSIX and .exe file on Windows platform, which is a good starting point for future deeper work. So, I told others this news on the bug tracker, hoping guys help me test on other platforms, esp. non Windows platform.

To create an executable script file on POSIX, my patch mainly does the following three things:

1) add a correct shebang line
2) add a correct import line, which is constructed from the input dotted path
3) add the specific ‘main’ function which is the executing entry of the script; create the script file and change its mode to make it executable

To create an .exe file on Windows, the patch does the following things:

1) create the wrapper script file as above
2) reuse cli.exe to get the resource string
3) create the .exe file

Future work will focus on supporting kind of ‘gui_scripts’ features for packaging, by reusing gui.exe, what’s more, writting robust tests to make this patch ok for other platforms.

In the mean time, I would have to blog two wonderful things which I have learned in this week after mentor’s good suggestion:

1) how to test the output of an executable script

There is a good utility function in packaging’s test module – captured_stdout, which can redirect our standard output(std.out), thus we can capture the output and test if it’s correct as expected. Actually, I somtimes also wanted to write such a function in my test, but I gave it up, because it seems making tests more complex. As a result, I used another workaround which may make others a little confused. I got to know that there always better solutions already supported in current Python, which I was not familiar with and didn’t know how to look for them.

I also spent a little time on readint the source of captured_stdout; it just defines a variable which has the ‘io.StringIO’ type to redirect the standard output. It’s so concise and useful, and I think there are still a lot of things to learn in future.

2) how to test the .exe files on Windows

The only way I knew to test .exe files is using ‘os.system()’, while I didn’t know how to pass the modified sys.path to it, because os.system will run Python in another process. I also know there is a subprocess module, but I’m also not sure how to pass it the sys.path. Thanks to mentor’s suggestion, I spent a little time on reading the source code of test_sys and found that many test functions are using such module to test the output of an executable file. subprocess.Popen function can take an enhanced os.environ as its parameter and can also easily get the output or error message from it. How exciting! It solved the problems which troubled me days.

Thank you, Eric, I have learned a lot in this summer, especially after your suggestions.

Best regards,
higery

Advertisements
Posted in Distutils2, GSOC2011, Python | 1 Comment

8# Weekly Report [2011/07/11 – 011/07/17]

Because I was on moving out home last week , I didn’t spend much time on my summer work and may fall a little behind the schedule. This week I have gotten away from all the busy things , then can spend a good time on my working stuff to catch up with the timeline.

Things done this week:

1 retrieve the recent emails again to get to know what I may miss , forget to reply , or  should do

Thanks to Eric’s valuable comments on my weekly report and detailed reviews against my  patches, I can always get to know whether I make some mistakes in current solution or fall
too deep in reading the source code of setuptools. But I’m still sorry that I was a little
busy , did not spend much time on my work, and didn’t reply the emails in a responsive
time. So, at the beginning of this week, I spend some time on reading the emails, among
which some are direct emails from my mentor, some are discussing emails on the bug  tracker, some are messages from the reviews against my patches. For a good starting up in future days, I think it’s worthwhile.

2 improve corresponding code on which mentor has made reviews

I noticed the reviews about the bug “Add **kwargs to get_reinitialized_command”, but I have spent almost the most of coding time on the ‘develop’ command, so I just can spare alittle time this week to improve my code corresponding with this issue. Thanks to  Thomas’s improvements work, I can continue with the ‘develop’ work now.

Eric has made many comments in the review about my ‘develop’ command, so it takes me
several day hours to fix problems and make improvements. Anyway, if there are not  things missed, I think I have finished the majority of improvements work and replied to every review line.

3 write two posts when reading the source code about auto script generation of setuptools

To make things more clear, I just wrote two posts to talk about what setuptools has done  when execute auto script generation. One is titled “[Automatic Script Creation]  ‘console_scripts’ on POSIX”, and the other is titled “[Automatic Script Creation] the
entry_points on Windows”. Although it may seems a little deep for me in reading the source code, but it still help a lot when I coding to implement auto-script-generation
functionality for packaging.

4 begin to write code about auto-script-generation and already write a simple test

After improving ‘develop’ command in a deep way, I can have time writing code to  implement auto-script-generation, even though in a very simple way. As mentor said, it’s a good starting point to firstly implement for a simple entry like “sphinx-build =  sphinx.build.main”. In addition, the tests take me a little time, but there are also some
problems in my tests and I will try to improve them asap.

Future work:
1 keep enhancing ‘develop’: functions, docs, tests
2 keep developing ‘scripts’

Best regards,
higery

Posted in Distutils2, GSOC2011 | Leave a comment

[Automatic Script Creation] the entry_points on Windows

This post just focuses on the entry_points on Windows platform, which is learned from setuptools. Different from POSIX, there are two entry_points – ‘console_scripts’ and ‘gui_scripts’ for windows.

Besides the things done in ‘console_scripts’ on POSIX, three extra things should be done on windows:

1.    match the correct extension and laucher for different entry_point
     1) string example:
    for gui_scripts: ext, launcher = ‘-script.pyw’, ‘gui.exe’
    for console_scripts: ext, launcher = ‘-script.py’, ‘cli.exe’
     2) reference in setuptools:
     lines in get_script_args() function
     3) need entry_points or pkg_resource support ?
     no
     4) consideration:
     On windows, setuptools always use a laucher (gui.exe or cli.exe) to lauch a file, it may need some improvements as people say on the bug tracker.

2.    get the resource string when creating .exe file
     1) string example:
     yield (name+’.exe’, resource_string(‘setuptools’, launcher),’b’)
     2) reference in setuptools:
     lines in get_script_args() function;
    the resource_string() function of setuptools
     3) need entry_points or pkg_resource support ?
     Yes
     4) consideration:
     how to get the resource string without pkg_resource there? Is it necessary?

3.    generate the manifest string when creating .exe file
     1) string example:
_launcher_manifest = “””
<?xml version=”1.0″ encoding=”UTF-8″ standalone=”yes”?>
<assembly xmlns=”urn:schemas-microsoft-com:asm.v1″ manifestVersion=”1.0″>
 <assemblyIdentity version=”1.0.0.0″
 processorArchitecture=”X86″
 name=”%s.exe”
 type=”win32″/>

 <!– Identify the application security requirements. –>
 <trustInfo xmlns=”urn:schemas-microsoft-com:asm.v3″>
 <security>
 <requestedPrivileges>
 <requestedExecutionLevel level=”asInvoker” uiAccess=”false”/>
 </requestedPrivileges>
 </security>
 </trustInfo>
</assembly>”””
     2) reference in setuptools:
     lines in get_script_args() function;
     the global ‘_launcher_manifest‘ variable in Easy_install.py
     3) need entry_points or pkg_resource support ?
     No.
     4) consideration:
     I’m not sure what’s the usage of this manifest string and if it will need some changes in future packaging.

Posted in Uncategorized | Leave a comment

[Automatic Script Creation] ‘console_scripts’ on POSIX

To make things more clear, I plan to write different posts to talk about what should be done for a specific entry_point on a specific platform. This post just focuses on the ‘console_scripts’ entry_point on POSIX, which is learned from setuptools. Future implementation in packaging may be much different, but we can enhance it against this post.

Five things should be done (already done in setuptools) to implement ‘console_scripts’ on POSIX:

1.    generate the script header
  1) string example:
     #!python.exe
     # EASY-INSTALL-ENTRY-SCRIPT:’    setuptools==0.6c11′,’console_scripts’,’easy_install-2.7′
  2) reference in setuptools:
     get_script_header() function
  3) need entry_points or pkg_resource support ?
     no
  4) consideration:
     It does not need entry_points or pkg_resource support, so this kind of function can be implemented for packaging at first.

2.    get the requirement specification
  1) string example:
     __requires__ = ‘setuptools==0.6c11’
  2) reference in setuptools:
     the line in get_script_args: spec = str(dist.as_requirement())
  3) need entry_points or pkg_resource support ?
     Yes
  4) consideration:
     how to get the requirement specification string without pkg_resource there?

3.    construct the script text to be written into the script file
  1) string example:
     __requires__ = ‘setuptools==0.6c11’
     import sys
     from pkg_resources import load_entry_point

     sys.exit(
     load_entry_point(‘setuptools==0.6c11’, ‘console_scripts’, ‘easy_install-2.7’)()
     )
  2) reference in setuptools:
     the ‘script_text’ variable in the get_script_args() function
  3) need entry_points or pkg_resource support ?
     no
  4) consideration:
     There is no pkg_resource in future packaging module, so the kind of ‘load_entry_point’ string will be much different. After we implement such kind of entry point loading function for packaging, this script text constructed can be meaningful.

4.    write the (script header + script text) into the script file
     Just write the file and change it to an executive mode, so no more to say.

5.    make this script run (platform support)
  1) string example:
     ‘load_entry_point’ in the script file
  2) reference in setuptools:
     check the ‘load_entry_point’ function in pkg_resource
  3) need entry_points or pkg_resource support ?
     definitely yes
  4) consideration:
     we should implement such kind of ‘load_entry_point’ function in packaging, thus all problems can be solved

Posted in Distutils2, GSOC2011, Python | 1 Comment

7# Weekly Report [2011/07/04 – 2011/07/10]

I’m back now, please forgive my disappearing – I’m moving out off home, and I cann’t reach the internet, so I didn’t make quick response to Eric on the bug tracker.

 

I’m reading the source code about script automatic creation in setuptools these days, and I will prepare a simple counterpart for packaging as soon as possible.

 

Anyway, finally, still sorry for not working hard these days. 😦

Posted in GSOC2011 | Leave a comment

6# Weekly Report [2011/06/27 – 2011/07/03]

I’m sorry that I’m a little busy this week, so 40 working hours maybe not assured. 😦 I will try to catch up with the original timeline and work harder.

Two simple things done in this week:

1) do some enhancement work basing on Eric’s review against the issue 8668.

2) reading the source code about script generation.

 

Best regards,

higery

Posted in GSOC2011 | Leave a comment

5# Weekly Report [2011/6/20 – 2011/6/26]

The ‘develop’ command still needs some improvement work, but there are different opinions which has brought a serious arguing in our fellowship list, so I have a little break and listen to what other people say.

Because different people may have different purposes and use the ‘develop’ command in a very different way, the continuous arguing will not help reach an extensive consensus, what’s more, may trap people in a chaos situation. So I just created a wiki page to collect these different use cases talked about in our list. Interested people can take a look at: http://wiki.python.org/moin/UsecasesOfDevelop . This page will be taken as a good idea base in future improvement work, and in my initial thought, some good features of setuptools will be kept in packaging, and some new features may be added if there are some good use cases that setuptools can not cover.

With regard to the ‘automatic script creation’ work, I have noticed there already are some good discussion on bug tracker, initiated by Eric. The issue870479 and issue976869 have already been set as duplicate with issue12394, so the final product of my second task is mainly to fix the issue12394, while there are many good opinions on these three issue pages, which will take me some time to read and collect the useful information.

Regards,

higery

Posted in GSOC2011 | 2 Comments