Show “python” — Where foo meets the bar

Entries tagged “python”

Public release of BurnerOnFire

written by Domen Kožar, on Oct 3, 2009 9:59:00 AM.

I'm proud to release first public version of BurnerOnFire.

BurnerOnFire is written in Python, providing simple GUI(GTK+) and CLI interface to burning .iso images simultaneously to multiple CD/DVD burners.

Documentation and install instructions are located at kiberpipa.org.

Features:

write iso image to multiple CD/DVD burners
count number of successfully burned discs
option to limit number of discs to be burned
detects write speeds and intersect results
eject/close tray burner
TODO: write multiple iso images from a folder

Writing post-installation-script to create shortcut on Windows desktop

written by Domen Kožar, on Sep 27, 2009 3:53:00 PM.

I'm building an GUI program that will be used on Windows platform. I already accepted the fact that I will need three installers (Python, GTK stuff, and one for my package).

Now, I want my installer to place shiny little shortcut on my desktop. Here is the command to generate Window installer:

python setup.py egg_info -RDb "" bdist_wininst --install-script postinstall.py

egg_info -RDb will clear and developemnt tags from release name, so the output will be package-0.1 instead of package-0.1dev
bdist_wininst will invoke Windows Installer builder
--install-script postinstall.py commands that postinstall.py script mentioned in setup.py file will be used for post installation.

Here is the setup.py slice. Note that this script must lie in root of our package:

...
setup(name='package',
      ...
      scripts=['postinstall.py'],
      ...
      )

And finally, the postinstall.py:

#! python
# -*- coding: utf-8 -*-

import os
import sys
import shutil
import my_package

DESKTOP_FOLDER = get_special_folder_path("CSIDL_DESKTOPDIRECTORY")
NAME = 'program.lnk'

if sys.argv[1] == '-install':
    create_shortcut(
        os.path.join(sys.prefix, 'pythonw.exe'), # program
        'Description of the shortcut', # description
        NAME, # filename
        mypackage.__file__, # parameters
        '', # workdir
        os.path.join(os.path.dirname(my_package.__file__), 'favicon.ico'), # iconpath
    )
    # move shortcut from current directory to DESKTOP_FOLDER
    shutil.move(os.path.join(os.getcwd(), NAME),
                os.path.join(DESKTOP_FOLDER, NAME))
    # tell windows installer that we created another 
    # file which should be deleted on uninstallation
    file_created(os.path.join(DESKTOP_FOLDER, NAME))

if sys.argv[1] == '-remove':
    pass
    # This will be run on uninstallation. Nothing to do.

[1] Note that I'm using some custom builtin functions, you can read more about here.
[2] I'm invoking pythonw.exe instead of python.exe because I don't want console to be visible (we are using GUI, remember).
[3] In __init__.py of the my_package, use the casual __name__ == '__main__' trick.

Read popen.stdout object asynchronously (or why low level knowledge holes are killing me)

written by Domen Kožar, on Sep 13, 2009 12:07:00 PM.

I've been programming for about two years, all this time only in Python. I'll say that those 500 lines I wrote in Flash AS2 before going to Python does not count. I'm fluent with standard library that Python provides, but I've noticed quite a few times now; I'm getting stuck on issues that correlate to low level knowledge. I'm thinking it may trigger someday motivation to learn C.

Going on topic, everyone that has used Popen at least once knows that file-alike objects (stdout, stderr, etc.) are blocking. Let me give you an example:

from subprocess import Popen, PIPE

p = Popen(['command doing lots of I/O and takes long to complete'],
    shell=True, stdout=PIPE)

while p.poll() == None:
    print p.stdout.read()

This snippet will block on "p.stdout.read()" line until spawned process will finish. This behaviour happens because Python I/O implemenation reads everything up until EOF (at least I thought so).

Now, let's add few lines of code:

import os
import fcntl
import gobject
from subprocess import Popen, PIPE

p = Popen(['command doing lots of I/O and takes long to complete'],
    shell=True, stdout=PIPE)
fd = p.stdout.fileno()

#file_flags = fcntl.fcntl(fd, fcntl.F_GETFL)
#fcntl.fcntl(fd, fcntl.F_SETFL, file_flags | os.O_NDELAY)

def test_io_watch(f, cond):
    out = f.read()
    if out == '':
        return False
    print out
    return True

gobject.io_add_watch(p.stdout,
                     gobject.IO_IN | gobject.IO_HUP,
                     test_io_watch)
gobject.MainLoop().run()

I'll comment what happens here (leaving the two lines commented). Popen object is created and subprocess spawned. Later we tell gobject to watch p.stdout for data and if there is some, call test_io_watch. test_io_watch function will read file and print it to stdout. This will happen async, leaving mainloop runing. This will also block output and desired effect is the same as in first snippet.

Now, let's uncomment those two lines and explain why it gives us expected results. fcntl will aquire file-descriptor flags and set additional located in os.O_NDELAY. If we Google for O_NDELAY first hit returns:

The O_NDELAY flag causes read() or write() to return zero instead of blocking.

This changes my guessing of how I/O blocking is handled in Python; it behaves by C implementation based on flags used by filedescriptors. Thus, Python does not (literally) wait for EOF (as it is stated in documentation) but it is just behavior of flags passed when opening a file.

This blog post triggers questions in me:
How deep may one's knowledge be to write code of seemingly simple tasks?
How much time would one spend to find that exact flag without Google?
Do I even know... ?

[#]Reference: http://www.mail-archive.com/pygtk@daa.com.au/msg18159.html

Ideas for a week project

written by Domen Kožar, on Aug 2, 2009 11:33:00 PM.

On 9th of August I return from vacations and I decided to research on some new area in the week after. Currently, there are two ideas, both easy-to-implement:

1. Ping my host every 5 minutes and send me email if down
I know there are many websites which touch this topic but; Most of them are limited and need subscription. I want something basic, to do just what I want.

WHY:
- learn basic Google App Engine development
- GAE provides reliable hosting with strong uptimes

2. List of websites mapped to tagged keywords
I have not found a simple, yet populated website that would provide list of websites based on some tag/keyword. Example: search for a "subtitle" tag and it would yield all websites where you can download subtitles for movies.

WHY:
- first time get known with MongoDB and it's usage
- learn how to estimate project timeline management with Pylons

What's your opinion? Out in the Web World, is there something exactly what I'm trying to do?

Advanced FormEncode usage

written by Domen Kožar, on Jul 22, 2009 2:11:00 PM.

I've been working on advanced HTML form that uses some Mootools (JavaScript framework) love. In most of my projects I use FormEncode in combination with FormAlchemy. Although both libraries have very different philosophies and areas, they work great together with some patches. I want to share schema because I spent quite some time putting it together - might save you some trobule.

This was the first time I used variabledecode module, and found best reference in this blog post.

Rendered form looks something like this, and if you choose another option in dropdown menu, it rearanges to something like this.

As you can see, AJAX removes some fields, others are rearranged. First form accepts many Name/Surname entries, the second one only gets one pair. I wanted to write a FA schema that can validate both cases and prepares my resulting dictionary to be directly inserted to SQLAlchemy model.

class ApplyForm(formencode.Schema):
    allow_extra_fields = True
    filter_extra_fields = True

    billIsNaturalPerson = validators.StringBool(not_empty=True)

    billEmail = validators.Email(not_empty=True)
    billAddress = validators.UnicodeString(not_empty=True)
    billPost = validators.UnicodeString(not_empty=True)
    billPhone = InternationalPhoneNumber(default_cc=386, not_empty=True)

    billFirm = validators.UnicodeString(if_missing=None)
    billTaxNumber = SlovenianTaxNumber(if_missing=None)
    billIDNumber = validators.Int(if_missing=None)
    billContactPerson = validators.UnicodeString(if_missing=None)

    billIsDDV = validators.Bool(not_empty=True, if_missing=None)
    billIsProformaInvoice = validators.Bool(not_empty=True)
    billIsSIR = validators.Bool(not_empty=True)
    billIsDinner = validators.Bool(not_empty=True)

    relParticipants = formencode.ForEach(ParticipantForm)
    relDaysAttending = validators.DictConverter({
        '1': [model.meta.Session()\
            .query(model.PortorozDay)\
            .filter(model.PortorozDay.plPDayDate == date(2009, 11, 18)).one()], # wed
        '2': [model.meta.Session()\
            .query(model.PortorozDay)\
            .filter(model.PortorozDay.plPDayDate == date(2009, 11, 19)).one()], # thu
        '3': model.meta.Session()\
            .query(model.PortorozDay).all() , # both
    }, not_empty=True, hideDict=True)

    pre_validators = [NestedVariables()]
    chained_validators = [
        validators.RequireIfPresent('billFirm', missing='billIsNaturalPerson'),
        validators.RequireIfPresent('billTaxNumber', missing='billIsNaturalPerson'),
        validators.RequireIfPresent('billIDNumber', missing='billIsNaturalPerson'),
        validators.RequireIfPresent('billContactPerson', missing='billIsNaturalPerson'),
    ]

class ParticipantForm(formencode.Schema):
    allow_extra_fields = True
    filter_extra_fields = False

    parName = validators.UnicodeString(if_missing=None)
    parSurname = validators.UnicodeString(if_missing=None)

    chained_validators = [
        validators.RequireIfPresent('parName', present='parSurname'),
        validators.RequireIfPresent('parSurname', present='parName'),
        MapToModel(model.PortorozParticipant),
    ]

First of all, it's all about the state of billIsNaturalPerson. This is the key boolean that separates both forms. As you can see in chained_validators, if it's value is False, it will require some additional fields. Fields with if_missing=None are required only in first form. Javascript removes input elements from DOM, so SQLAlchemy model is populated with None values.

relDaysAttending is a SQLAlchemy relation and in form terminology, a radioset. Values are mapped with help of DictConverter validator to model instances, which results in correctly populated ForeignKeys.

relParticipants is also a SQLAlchemy relation that lists name/surname pairs. As you can see, it's validated with help of variabledecode algorithm and validator that corresponds to that is NestedVariables. Because NestedVariables validators lies in pre_validators attribute, when validator occurs, values are already mapped to list/dictionary. Input names look something like this:

<div>
    <input type="text" name="relParticipants-1.parName" value="John" />
    <input type="text" name="relParticipants-1.parSurname" value="Smih" />
</div>
<div>
    <input type="text" name="relParticipants-2.parName" value="..." />
...

Which maps to list of dicts:

{'relParticipants': [
   {'parName': 'John', 'parSurname': 'Smith'},
   ...
]}

ForEach validator later maps relParticipant list to ParticipantForm schema. This schema is pretty basic, chained validators are used to require second field (if one is supplied) and map resulting dict to model instance. MapToModel implementation is very simple:

class MapToModel(validators.FormValidator):
    __unpackargs__ = ('model',)
    model = None

    def _to_python(self, value_dict, state):
        return self.model(**value_dict)

Example of parsed POST data, mapped to be ready for SQLAlchemy model:

18:24:07,349 DEBUG [project.controllers.portoroz] Form input:
{'billAddress': u'Foo street 12',
 'billContactPerson': u'John Smith',
 'billEmail': u'john.smith@gmail.com',  
 'billFirm': u'Firm',
 'billIDNumber': 1337,
 'billIsDDV': None,
 'billIsDinner': False,
 'billIsNaturalPerson': False,
 'billIsProformaInvoice': True,
 'billIsSIR': False,
 'billPhone': '+386-11-444489',
 'billPost': u'1000 Ljubljana',
 'billTaxNumber': 40952649,
 'relDaysAttending': [<PortorozDay 2009-11-18>, <PortorozDay 2009-11-19>],
 'relParticipants': [<PortorozParticipant(None) parName=u'foo', parSurname=u'bar'>,
                     <PortorozParticipant(None) parName=u'hello', parSurname=u'kitty'>,
                     ]
}

Problems started when I tried htmlfill and it's error rendering. First issue was that I needed to reverse variabledecode algorithm, this can be achieved with following e.unpack_errors call:

        try:
            c.form_result = ApplyForm().to_python(request.POST)
        except formencode.validators.Invalid, e:
            return formencode.htmlfill.render(
                self.apply_action(),
                defaults=request.POST,
                errors=e.unpack_errors(formencode.variabledecode.variable_encode))

Second issue was if user added lots of Participants, they were not correctly populated in form inputs because by default only one was being rendered and additional ones are inserted by javascript. I decided to take most ugly but simple approach: by default render 30 fields and onDOMReady, remove the all the empty fields except one.

There are details I left out of this post, since main concern was FormEncode usage. I hope you learned something, cheers!

Persuading Paste HTTP server to use correct REMOTE_ADDR

written by Domen Kožar, on Jun 27, 2009 12:58:00 AM.

Configuring reverse proxy headers was on bottom of my TODO list because I never felt it was important for my applications. Today I tried to fuddle with nginx to pass those headers but it gave me a headache.

For sake of reference and because I'm sure many others tripped on this one, here is what I figured out from mailing lists and source code (please inform me of a better, straight forward way if there is one):

Add to your deployment.ini file:

[app:main]
# ...
filter-with = proxy-headers

[filter:proxy-headers]
use = egg:PasteDeploy#prefix

Prefix middleware actually extracts proxy headers information and passes results forward. Now, nginx config:

    location / {
        proxy_pass http://localhost:5000;

        proxy_set_header X_FORWARDED_SERVER $server_name;
        proxy_set_header X_FORWARDED_FOR $proxy_add_x_forwarded_for;
        proxy_set_header X_FORWARDED_HOST $proxy_host;

    }

Joining strings in Python

written by Domen Kožar, on Jun 11, 2009 2:58:00 PM.

There is already a lot of good material to read about performance. Therefore, it is best to use the following method:

    >>> print ' '.join(['The', 'fox', 'jumped', 'over', 'the', 'dog.'])
    The fox jumped over the dog.

Perfect. We have very fast string contatenation that even allows us to choose separator. Now, here is the problem:

    >>> print ', '.join(['apples', 'oranges', '', 'cocos'])
    apples, oranges, , cocos

So, if we dynamically generate string and we have a separator for a reason, we do not want this to happen. This is basically the idea:

    >>> print ', '.join(filter(None, ['apples', 'oranges', '', 'cocos']))
    apples, oranges, cocos

As you can see, filter(None, iterable) returns list of elements that evalue to True. This way we have somewhat a safe string contatenation. Real world example:

class StringBuffer:

	def __init__(self, sep=''):
		self.sep = sep
		self.output = list()

	def write(self, content):
		self.output.extend(content)

	def getvalue(self):
		return self.sep.join(filter(None, self.output))


>>> sb = StringBuffer(', ')
>>> sb.write(['apples'])
>>> sb.write(['', None])
>>> sb.write(['oranges', 'cocos'])
>>> print sb.getvalue()
apples, oranges, cocos

Mocking logging module for unittests

written by Domen Kožar, on Mar 4, 2009 6:07:00 PM.

Today I was writing tests for upcoming application I'm writing (more information will come later) and I needed some way to redirect logging to some kind of storage that could be easily asserted to values in tests. First I tried with monkey patching it, but that wasn't the solution since all logging setup is done before pylons tests come to play, so somebody in IRC pointed me toward writing a handler. I thought to do so in the first place, but were too stubborn;) Here is the handler code:

class ListHandler(logging.Handler):

    debug = []
    warning = []
    info = []
    error = []

    def emit(self, record):
        getattr(self.__class__, record.levelname.lower()).append(record.getMessage())

    @classmethod
    def reset(cls):
        for attr in dir(cls):
            if isinstance(getattr(cls, attr), list):
                setattr(cls, attr, [])

So basically, all records are stored in handlers class(!), which is restored to zero on every tearDown() call. Later in the code, I can do stuff like:

    def test_foobar(self):
        some_function_that_outputs_logging()
        self.failUnless('Invalid status' in  ListHandler.warning)

« Previous 1 · 2 Next »

Entries tagged “python”

Archive