Where foo meets the bar

GSoC 2010 is history, but more and more is going on.

Let me rephrase a bit what was my project over this summer, what does it provide and what will the future bring.

gpypi2 has become quite more than it's primary goal, installation of Python packages has never been easier:

• Create/echo Gentoo Linux ebuilds from Python Package Index or from Python source (with help of distutils)
• Install that ebuild with invoking emerge
• Create overlay from all working packages on Python Package Index

Install

Very easy.

$ layman -f -o http://gpo.zugaina.org/lst/gpo-repositories.xml

### unmask gpypi2, easiest with autounmask
$ autounmask -a dev-python/gpypi2-9999

$ emerge -av gpypi2

gpypi2 -h for list of commands. Please report bugs to BitBucket issue tracker

Not over yet

My work at Gentoo. I will be back.

Special thanks to my mentor Jesus "neurogeek" Rivero, without him it would be impossible.

• Repository, issue tracker
• Hudson CI instance
• Sphinx documentation generated by hudson on each commit

GPyPi knows about metadata.xml, ChangeLog and Manifest!

Previous week (2nd - 9th August)

Task: metadata, echangelog and manifest generation

In the last week of implementing features I will add support for Gentoo developers workflow.

Done. Implementation is in gpypi2/workflow.py. One issue though: echangelog has no idea how to add files to SCM, will have to find another abstraction tool for that.

Task: Atomic actions

Currently if anything fails in the process, data will lay around the filesytem.
I'll implement most basic cleanup (rollback) on failures.

I haven't touched this one, mostly because I don't have a good idea how to handle tracking of files.

Task: Exceptions

Code currently uses a bit of exceptions and a bit of log.error.
I'll make that a bit structured and also document what exceptions are raised in functions

Done. Exceptions are documented and properly catched in cli.py code.

Task: integration tests and documentation

Update documentation and write some integration tests.

Partly done. Tests still need some love, but that will not be part of the GSOC (I still have to properly implement SrcUriNamer.)

GSoC is over, I have met my deliverables. It was a lot of fun, can't wait for Gentoo community to start using my tool. Project is far far from finished, I will continue to work on it and polish it. At the end, Gentoo is my primary choice for platform (even on laptop).

I can't wait for next year to apply for the project and continue to work on OSS. Cheers guys!

• Repository, issue tracker
• Hudson CI instance
• Sphinx documentation generated by hudson on each commit

GPyPi now supports building ebuilds directly from Python packages. It's tied into distutils:

        python setup.py sdist_ebuild

Configuration is not as verbose as through pypi, but that is a subject to change.

Previous 2 weeks (19th July - 4th August)

Task: fix TODOs in the code

Code currently has about 60 TODOs that need to be address for a better codebase.

Most important ones are cleaned. Some of highlights:

• echo command does not use overlay anymore for storing temporary ebuild but /tmp/
• conditional use flags are used properly now
• remember configuration values given by interactive questionnaire

Task: Implement "python setup.py bdist_ebuild" command

I'll peek into distutils heart and implement most basic bdist command to output an ebuild.
I will basically the same stuff we currently do,
except it will parse data from setup.py itself and not through pypi.

Done. When gpypi2 is called the first time it installs support for distutils command.

Upcoming week (4th - 11th August)

Task: metadata, echangelog and manifest generation

In the last week of implementing features I will add support for Gentoo developers workflow.

Task: Atomic actions

Currently if anything fails in the process, data will lay around the filesytem.
I'll implement most basic cleanup (rollback) on failures.

Task: Exceptions

Code currently uses a bit of exceptions and a bit of log.error.
I'll make that a bit structured and also document what exceptions are raised in functions

Task: integration tests and documentation

Update documentation and write some integration tests.

• Repository, issue tracker
• Hudson CI instance
• Sphinx documentation generated by hudson on each commit

Previous 2 weeks (5th - 19th July)

Task: Write configuration skeleton

Currently, gpypi2 code only takes configuration from command line.
I'd like to extend that to config file (.ini), get data directly from setup.py
and interactive questionaire. In order for that to function properly,
configuration needs to be done in right sequence.

Following week I will write down the design and implement initial working prototype.

Configuration dispatcher fully implemented! Also we have already working questionnaire functionality. Details can be found at http://docs.fubar.si/gpypi2/userguide.html#configuration

Task: sync command - populate overlay with successful ebuilds from pypi

When I will get bored or stuck with task #1, I'll start working on command that will
generate ebuilds from all packages in pypi to an overlay.

Initial "sync" command is implemented. Still lacking proper error handling, metadata generation and manifest update support. Will be a lot easier to implement with current configuration management.

Additional, following mini-tasks were performed:

• fixed echo command to work properly
• correct handling of whitespace in rendered ebuilds
• replace 4 spaces with a tab in rendered ebuilds

Upcoming week (20th - 26th July)

Task: fix TODOs in the code

Code currently has about 60 TODOs that need to be address for a better codebase.

Task: Implement "python setup.py bdist_ebuild" command

I'll peek into distutils heart and implement most basic bdist command to output an ebuild.
I will basically the same stuff we currently do,
except it will parse data from setup.py itself and not through pypi.

wget http://googleappengine.googlecode.com/files/google_appengine_1.3.5.zip
unzip google_appengine_1.3.5.zip
cd google_appengine
git clone git://github.com/mrsteveman1/aeservmon.git {app-name}
vim {app-name}/app.yaml # set application
python appcfg.py update {app-name}

• Repository, issue tracker
• Hudson CI instance
• Sphinx documentation generated by hudson on each commit
• wiki/user stories for scrum development

Last two weeks were crazy at university. Now I shall have time to work on gpypi2 full-time until the end of program.

$ sudo pip install http://bitbucket.org/iElectric/g-pypi2/get/tip.zip

or

$ sudo easy_install http://bitbucket.org/iElectric/g-pypi2/get/tip.zip

$ sudo gpypi2 create flask

 * Generating ebuild: Flask 0.4
 * Your ebuild is here: /usr/local/portage/dev-python/flask/flask-0.4.ebuild
 * Dependency needed: Werkzeug
 * Dependency needed: Jinja2
 * Generating ebuild: Werkzeug 0.6.2
 * Your ebuild is here: /usr/local/portage/dev-python/werkzeug/werkzeug-0.6.2.ebuild
 * Generating ebuild: Jinja2 2.5
 * Your ebuild is here: /usr/local/portage/dev-python/jinja2/jinja2-2.5.ebuild
 * Dependency needed: Babel
 * Generating ebuild: Babel 0.9.5
 * Your ebuild is here: /usr/local/portage/dev-python/babel/babel-0.9.5.ebuild

Previous week (21st June - 5th July)

Task: Use SrcUriNamer

gpypi2.enamer.SrcUriNamer is awaiting for quite 2 weeks now
to be finally used.

It's basically factory that generates SRC_URI and HOMEPAGE
according to mirror provider.
A lot of additional refactoring will be needed to Enamer to
use this utility instead of current guessing/parsing.

Task: Fix bugs reported by users

Lot of bugs are going to pop up; fix bugs, write tests and
add documentation.

Task: Increase test coverage

I'm still not satisfied with test coverage.
I'll try to increase it a bit more.
It's 10 days week of exams on my university, soon I will be able to put
in more effort in implementing features.

Here is a rundown of additional changes to code:

• slight improvements to documentation
• option to choose PyPi base URL
• changed behaviour to use XML-RPC and SETUPTOOLS/SIMPLEPAGE method for retrieving SRC_URI
• updated licences dictionary mapping
• fixed echo command to work properly (without cleanup atm)

Upcoming week (22nd June - 28th June)

Task: Write configuration skeleton

Currently, gpypi2 code only takes configuration from command line.
I'd like to extend that to config file (.ini), get data directly from setup.py
and interactive questionaire. In order for that to function properly,
configuration needs to be done in right sequence.

Following week I will write down the design and implement initial working prototype.

Task: sync command - populate overlay with successful ebuilds from pypi

When I will get bored or stuck with task #1, I'll start working on command that will
generate ebuilds from all packages in pypi to an overlay.

• Repository, issue tracker
• Hudson CI instance
• Sphinx documentation generated by hudson on each commit
• wiki/user stories for scrum development

I'm more than happy to announce first pre-alpha prototype. Intended goal is to make people start using the tool and reporting back their critics.

$ sudo pip install http://bitbucket.org/iElectric/g-pypi2/get/tip.zip

or

$ sudo easy_install http://bitbucket.org/iElectric/g-pypi2/get/tip.zip

$ sudo gpypi2 create flask

 * Generating ebuild: Flask 0.4
 * Your ebuild is here: /usr/local/portage/dev-python/flask/flask-0.4.ebuild
 * Dependency needed: Werkzeug
 * Dependency needed: Jinja2
 * Generating ebuild: Werkzeug 0.6.2
 * Your ebuild is here: /usr/local/portage/dev-python/werkzeug/werkzeug-0.6.2.ebuild
 * Generating ebuild: Jinja2 2.5
 * Your ebuild is here: /usr/local/portage/dev-python/jinja2/jinja2-2.5.ebuild
 * Dependency needed: Babel
 * Generating ebuild: Babel 0.9.5
 * Your ebuild is here: /usr/local/portage/dev-python/babel/babel-0.9.5.ebuild

Previous week (15-21 June)

Main goal of previous week was to get closer to first prototype. Done.

Task: First snippet of command line interface

Put enough together to assemble ebuilds from command line. I think
many issues may pop up, and lots of bugs shall be fixed. I'm currently
mostly doing stuff from my head, with not much practise whatsoever. 

Mostly, we should get a result of better test coverage (currrently about
50%).

The only component left untouched is the configuration. I would like to
touch this topic after all my exams are over, leaving me time and
productivity.

Command line interface was added, test coverage is up to 60%. Lots of small fixes were done. As a bonus, I added portage output syntax colors.

Task: get_dependencies - another refactor to enamer

Title says it all. Also handling of multiple version operators should be
added.

Example: foobar>=0.5&& foobar<0.6

should resolve in <0.6 blocker and >=0.5 requirement.

Multiple operators on version dependencies are handled as blockers, implemented in tip. However get_dependencies was not refactored, since I still might be adding functionality. Not really necessary for now.

Upcoming week (22-28 June)

Task: Use SrcUriNamer

gpypi2.enamer.SrcUriNamer is awaiting for quite 2 weeks now
to be finally used.

It's basically factory that generates SRC_URI and HOMEPAGE
according to mirror provider.
A lot of additional refactoring will be needed to Enamer to
use this utility instead of current guessing/parsing.

Task: Fix bugs reported by users

Lot of bugs are going to pop up; fix bugs, write tests and
add documentation.

Task: Increase test coverage

I'm still not satisfied with test coverage.
I'll try to increase it a bit more.
It's 10 days week of exams on my university, soon I will be able to put
in more effort in implementing features.

from pyparsing import *

nginx_conf_expr = OneOrMore(Suppress(SkipTo('server' + White())) + originalTextFor(Word('server ') + nestedExpr('{', '}'))).parseWithTabs()
server_name_expr = (Suppress(SkipTo('server_name' + White()) + Word('server_name') + White()) +  CharsNotIn(' ;')).parseWithTabs()

nginx_conf = open('nginx.conf').read()
new_nginx_conf = str(nginx_conf)

for server in nginx_conf_expr.parseString(nginx_conf):
    try:
        name = server_name_expr.parseString(server)[0]
        f = open('nginxsite_%s.conf' % name, 'w+')
        f.write(str(server))
        f.close()

	# update nginx config
	new_nginx_conf = new_nginx_conf.replace(str(server), '')
    except:
        print 'Entry failed:\n', server

open('new_nginx.conf', 'w').write(new_nginx_conf)

This script basically extracts all server {} entries and writes them to separate files named "nginxsite_sitename.conf" (all in current working directory). It also produces new_nginx.conf that does not include extracted entries. All you need to do is to add include directive to new_nginx.conf like this:

  # path is relative to this global nginx.conf
  include sites/*.conf;

temp $ ls
migrate_nginx_config.py
nginx.conf

temp $ python migrate_nginx_config.py
migrate_nginx_config.py
nginx.conf
new_nginx.conf
nginxsite_www.fubar.si.conf
nginxsite www.tvnext.si.conf
...