Where foo meets the bar

During last two days I managed to install my HTPC (soon I will add printers and samba to it). There is a lot of useful information around the internet which helped me on the way. I really like to create Gentoo from scratch logs, so I would like to share it with you together with used patches (in my iElectric gentoo repository).

DISCLAIMER: This is far from perfect (even the ebuilds), but it works;)

DISCLAIMER2: I'm aware this is not the noobs guide, it's more of a reference manual

# some stolen from http://en.gentoo-wiki.com/wiki/Acer_Aspire_Revo_R3600_/_R3610#VDPAU

* ping google.com
* fdisk /dev/sda
# boot partition
n
p
1
<enter>
+30M
# swap partition
n
p
2
<enter>
+1G
# root partition
n
p
3
<enter>
<enter>
w

# create filesystems
mkfs.ext2 /dev/sda1
mkswap /dev/sda2
mkfs.ext4 /dev/sda3

# activate swap
swapon /dev/sda2

# prepare/mount chroot
mount /dev/sda3 /mnt/gentoo
mkdir /mnt/gentoo/boot
mount /dev/sda1 /mnt/gentoo/boot
cd /mnt/gentoo

# stage3/portage download (use d shortcut for download)
links http://www.gentoo.org/main/en/mirrors.xml
server->releases->x86->current stage3 i686 + DIGEST + CONTENTS
server->snapshosts->portage-latests + digest

# validate size
md5sum -c stage*.DIGEST
md5sum -c portage*.md5sum

# install portage/stage3
tar xvjpf stage3-*.tar.bz2
tar xvjf portage-latests.tar.bz2 -C /mnt/gentoo/usr

# select mirrors (in terminal 2)
mirrorselect -i -o >> /mnt/gentoo/etc/make.conf
mirrorselect -i -r -o >> /mnt/gentoo/etc/make.conf
# add slovenian mirror

# copy dns settings
cp -L /etc/resolv.conf /mnt/gentoo/etc/

# create local portage overlay
mkdir /mnt/gentoo/usr/local/portage

# edit make.conf (back in terminal 1)
vim /mnt/gentoo/etc/make.conf

# chroot
mount -t proc none /mnt/gentoo/proc
mount -o bind /dev /mnt/gentoo/dev
chroot /mnt/gentoo /bin/bash
env-update
. /etc/profile

# refresh tree
emerge --sync

# select profile
eselect profile list
eselect profile set #n

# emerge must-have stuff under install
emerge -v vim eix pciutils gentoo-sources

# setup slovene locale (terminal 2)
vim /etc/locale.gen
vim /etc/env.d/02locale
locale-gen

# setup slovene keymap
vim /etc/conf.d/keymaps

# setup zone
cp /usr/share/zoneinfo/Europe/Ljubljana /etc/localtime

# setup clock
vim /etc/conf.d/clock

# setup hostname
vim /etc/conf.d/hostname (server.domain.si)

# add basic startup scripts

rc-update add net.eth0 default
rc-update add sshd default

# install distcc
# http://www.gentoo.org/doc/en/distcc.xml
# http://www.gentoo.org/doc/en/cross-compiling-distcc.xml
emerge -av distcc
vim /etc/distcc/hosts
# do the magic in /usr/lib/distcc/bin/

# compile kernel
lspci (terminal 2)
cd /usr/src/linux
make menuconfig

- kernel settings
Procesesor type and features -->
  (Intel Atom) Processor family
Power management and ACPI options -->
  CPU frequency scaling -->
    everything
Device drivers -->
  Serial Ata and Parallel ATA drivers -->
    NVIDIA SATA support
  Networking drivers ->
    Ethernet (10 or 1000mbit) -->
      [M] nForce Ethernet support
    Wireless LAN -->
      Atheros Wireless Cards -->
        [M] everything
  Sound card support -->
    ALSA -->
      PCI Sound devices -->
        Intel HD Audio -->
          [*] everything
  [ ] Multiple devices driver support -->
    
File systems ->
  [*] Ext2
  [*] Ext4
  [*] FUSE

# compile kernel and place it
make && make modules_install
cp arch/i386/bzImage /boot/kernel-

# add modules to startup
vim /etc/modules.autoload.d/kernel2.6

# set root password
emerge -av pwgen dhcpcd grub
pwgen
passwd

# create user
# gost==guest
useradd -m -G users,wheel,video,cdrom -U -s /bin/bash ielectric
useradd -m -G users,video,cdrom -U -s /bin/bash gost

# setup grub conf
vim /boot/grub/grub.conf

# setup fstab
vim /etc/fstab

# setup MBR
grub --no-floppy
root (hd0,0)
setup (hd0)
quit

# cleanup and reboot
rm /portage*
rm /stage*

reboot

### after basic OS setup

# on local machine setup hosts
vim /etc/hosts
# back to server

# setup rc.conf
vim /etc/rc.conf

# local: copy ssh key over
cat ~/.ssh/id_rsa.pub | ssh server.domain.si "cat - >> ~/.ssh/authorized_keys"
# back to remote
chmod 744 /home/ielectric/.ssh/autorhized_keys

# setup sshd
vim /etc/ssh/sshd_config

# system update
eix-sync
emerge -DuNav world

# install autounmask flagedit
emerge -av autounmask gentoolkit flagedit

# eix and screen install
emerge -av eix screen
echo "*" > /etc/eix-sync.conf

# layman
flagedit app-portage/layman +subversion +git +mercurial
emerge -av layman
layman -a iElectric
layman -a sunrise

# ntp
emerge -av ntp
/etc/init.d/ntpd start
/etc/init.d/ntp-client start
rc-update add ntpd default
rc-update add ntp-client default

# update startup script
rc-update add acpid default
rc-update add nscd default
rc-update add dbus default
rc-update add hald default
rc-update add consolekit default

# bacula client
flagedit app-backup/bacula +bacula-nodir +bacula-nosd +bacula-clientonly
emerge -av bacula
# update directory name in bacula-df.conf
# update bacula-dir.conf on central server
/etc/init.d/bacula-fd start
rc-update add bacula-fd default

# logrotate, syslog-ng, vixie-cron, slocate
emerge -av slocate logrotate syslog-ng vixie-cron

rc-update add syslog-ng default
rc-update add vixie-cron default

# munin
echo "net-analyzer/munin postgres ssl" >> /etc/portage/package.use
echo "x11-libs/cairo svg" >> /etc/portage/package.use
emerge -av munin
sudo -u munin munin-node-configure --shell | bash
# update allowed host in /etc/munin/munin-node.conf
# update munin.conf on central server
/etc/init.d/munin-node start
rc-update add munin-node default

# update cronjob
crontab -e
slocate, eix-sync

# random stuff
emerge -av links mailx ntfs3g

# xbmc
# ebuild should be taken from iElectric overlay
autounmask -a media-tv/xbmc-9.11-r5
emerge -av xdm gdm gnome-session xbmc
rc-update add xdm default

# set default session to xbmc
cd /etc/X11/Sessions/
ln -s /usr/bin/xbmc-standalone
vim /etc/rc.conf

# switch to nvidia opengl implementation
eselect opengl set 1

# give guest user permissions for reboot/shutdown..
polkit-auth --user gost --grant org.freedesktop.hal.power-management.reboot
polkit-auth --user gost --grant org.freedesktop.hal.power-management.shutdown
polkit-auth --user gost --grant org.freedesktop.hal.power-management.suspend

# install transperancy skin
cd /home/gost/.xbmc/skin/
svn checkout http://transparency-xbmc.googlecode.com/svn/trunk/ Transparency

# install svn repo installer
http://marshalleq.wordpress.com/2009/08/22/how-to-install-svn-installer-on-xbmc-live-to-enable-automatic-skin-and-plugin-upgradesinstalls/

# install OpenSubtitles OSD
http://code.google.com/p/opensubtitles-osd/wiki/Instalation
# go to scripts and setup the plugin
# go to skin settings and enable subtitle OSD plugin -> default.py

# install remote support
# ebuild should be taken from iElectric overlay
# BIG BIG THANKS to
# http://forum.sabayon.org/viewtopic.php?f=56&t=20113&start=10#p114286
autounmask -a app-misc/lirc-0.8.6-r4
echo 'LIRC_DEVICES="wpc8769l"' >> /etc/make.conf
emerge -av lirc
rc-update add lirc default
echo "lirc_wb677" >> /etc/modules.autoload.d/kernel-2.6

# check if everything boots correctly
reboot

# TODO: list of all conf files
# TODO esata?

UPDATE: fixed typo and added dummy module for lirc

GSoC 2010 is history, but more and more is going on.

Let me rephrase a bit what was my project over this summer, what does it provide and what will the future bring.

gpypi2 has become quite more than it's primary goal, installation of Python packages has never been easier:

• Create/echo Gentoo Linux ebuilds from Python Package Index or from Python source (with help of distutils)
• Install that ebuild with invoking emerge
• Create overlay from all working packages on Python Package Index

Install

Very easy.

$ layman -f -o http://gpo.zugaina.org/lst/gpo-repositories.xml

### unmask gpypi2, easiest with autounmask
$ autounmask -a dev-python/gpypi2-9999

$ emerge -av gpypi2

gpypi2 -h for list of commands. Please report bugs to BitBucket issue tracker

Not over yet

My work at Gentoo. I will be back.

Special thanks to my mentor Jesus "neurogeek" Rivero, without him it would be impossible.

• Repository, issue tracker
• Hudson CI instance
• Sphinx documentation generated by hudson on each commit

GPyPi knows about metadata.xml, ChangeLog and Manifest!

Previous week (2nd - 9th August)

Task: metadata, echangelog and manifest generation

In the last week of implementing features I will add support for Gentoo developers workflow.

Done. Implementation is in gpypi2/workflow.py. One issue though: echangelog has no idea how to add files to SCM, will have to find another abstraction tool for that.

Task: Atomic actions

Currently if anything fails in the process, data will lay around the filesytem.
I'll implement most basic cleanup (rollback) on failures.

I haven't touched this one, mostly because I don't have a good idea how to handle tracking of files.

Task: Exceptions

Code currently uses a bit of exceptions and a bit of log.error.
I'll make that a bit structured and also document what exceptions are raised in functions

Done. Exceptions are documented and properly catched in cli.py code.

Task: integration tests and documentation

Update documentation and write some integration tests.

Partly done. Tests still need some love, but that will not be part of the GSOC (I still have to properly implement SrcUriNamer.)

GSoC is over, I have met my deliverables. It was a lot of fun, can't wait for Gentoo community to start using my tool. Project is far far from finished, I will continue to work on it and polish it. At the end, Gentoo is my primary choice for platform (even on laptop).

I can't wait for next year to apply for the project and continue to work on OSS. Cheers guys!

• Repository, issue tracker
• Hudson CI instance
• Sphinx documentation generated by hudson on each commit

GPyPi now supports building ebuilds directly from Python packages. It's tied into distutils:

        python setup.py sdist_ebuild

Configuration is not as verbose as through pypi, but that is a subject to change.

Previous 2 weeks (19th July - 4th August)

Task: fix TODOs in the code

Code currently has about 60 TODOs that need to be address for a better codebase.

Most important ones are cleaned. Some of highlights:

• echo command does not use overlay anymore for storing temporary ebuild but /tmp/
• conditional use flags are used properly now
• remember configuration values given by interactive questionnaire

Task: Implement "python setup.py bdist_ebuild" command

I'll peek into distutils heart and implement most basic bdist command to output an ebuild.
I will basically the same stuff we currently do,
except it will parse data from setup.py itself and not through pypi.

Done. When gpypi2 is called the first time it installs support for distutils command.

Upcoming week (4th - 11th August)

Task: metadata, echangelog and manifest generation

In the last week of implementing features I will add support for Gentoo developers workflow.

Task: Atomic actions

Currently if anything fails in the process, data will lay around the filesytem.
I'll implement most basic cleanup (rollback) on failures.

Task: Exceptions

Code currently uses a bit of exceptions and a bit of log.error.
I'll make that a bit structured and also document what exceptions are raised in functions

Task: integration tests and documentation

Update documentation and write some integration tests.

• Repository, issue tracker
• Hudson CI instance
• Sphinx documentation generated by hudson on each commit

Previous 2 weeks (5th - 19th July)

Task: Write configuration skeleton

Currently, gpypi2 code only takes configuration from command line.
I'd like to extend that to config file (.ini), get data directly from setup.py
and interactive questionaire. In order for that to function properly,
configuration needs to be done in right sequence.

Following week I will write down the design and implement initial working prototype.

Configuration dispatcher fully implemented! Also we have already working questionnaire functionality. Details can be found at http://docs.fubar.si/gpypi2/userguide.html#configuration

Task: sync command - populate overlay with successful ebuilds from pypi

When I will get bored or stuck with task #1, I'll start working on command that will
generate ebuilds from all packages in pypi to an overlay.

Initial "sync" command is implemented. Still lacking proper error handling, metadata generation and manifest update support. Will be a lot easier to implement with current configuration management.

Additional, following mini-tasks were performed:

• fixed echo command to work properly
• correct handling of whitespace in rendered ebuilds
• replace 4 spaces with a tab in rendered ebuilds

Upcoming week (20th - 26th July)

Task: fix TODOs in the code

Code currently has about 60 TODOs that need to be address for a better codebase.

Task: Implement "python setup.py bdist_ebuild" command

I'll peek into distutils heart and implement most basic bdist command to output an ebuild.
I will basically the same stuff we currently do,
except it will parse data from setup.py itself and not through pypi.

wget http://googleappengine.googlecode.com/files/google_appengine_1.3.5.zip
unzip google_appengine_1.3.5.zip
cd google_appengine
git clone git://github.com/mrsteveman1/aeservmon.git {app-name}
vim {app-name}/app.yaml # set application
python appcfg.py update {app-name}

Packaging love-hate world in Python is with every Python developer. Each time you need something from setuptools/distribute/distutils you actually find something new, useful. Can not deny that. There are quite some people working hard on distutils2 to remove the "miserable" word from packaging. I'm really hoping that lack of communication with Python developers and Linux distribution will be fixed when people will not only care, but also give constructive thoughts how distutils could be improved (and thus make life easier even to Linux distributions).

Enough of non-sense philosophy. Recently I found about "Feature" class in setuptools/distribute. Let's see what setuptools has to say about "features" keyword passed to the famous setup function:

'features' -- a dictionary mapping option names to 'setuptools.Feature'
        objects.  Features are a portion of the distribution that can be
        included or excluded based on user options, inter-feature dependencies,  
        and availability on the current system.  Excluded features are omitted
        from all setup commands, including source and binary distributions, so
        you can create multiple distributions from the same source tree.
        Feature names should be valid Python identifiers, except that they may
        contain the '-' (minus) sign.  Features can be included or excluded
        via the command line options '--with-X' and '--without-X', where 'X' is
        the name of the feature.  Whether a feature is included by default, and 
        whether you are allowed to control this from the command line, is  
        determined by the Feature object.  See the 'Feature' class for more                                                                                  
        information.

A subset of the distribution that can be excluded if unneeded/wanted

Features are created using these keyword arguments:

  'description' -- a short, human readable description of the feature, to
     be used in error messages, and option help messages.

  'standard' -- if true, the feature is included by default if it is
     available on the current system.  Otherwise, the feature is only
     included if requested via a command line '--with-X' option, or if
     another included feature requires it.  The default setting is 'False'.

  'available' -- if true, the feature is available for installation on the
     current system.  The default setting is 'True'.

  'optional' -- if true, the feature's inclusion can be controlled from the
     command line, using the '--with-X' or '--without-X' options.  If
     false, the feature's inclusion status is determined automatically,
     based on 'availabile', 'standard', and whether any other feature
     requires it.  The default setting is 'True'.

  'require_features' -- a string or sequence of strings naming features
     that should also be included if this feature is included.  Defaults to
     empty list.  May also contain 'Require' objects that should be
     added/removed from the distribution.

  'remove' -- a string or list of strings naming packages to be removed
     from the distribution if this feature is *not* included.  If the
     feature *is* included, this argument is ignored.  This argument exists
     to support removing features that "crosscut" a distribution, such as
     defining a 'tests' feature that removes all the 'tests' subpackages
     provided by other features.  The default for this argument is an empty
     list.  (Note: the named package(s) or modules must exist in the base
     distribution when the 'setup()' function is initially called.)

  other keywords -- any other keyword arguments are saved, and passed to
     the distribution's 'include()' and 'exclude()' methods when the
     feature is included or excluded, respectively.  So, for example, you
     could pass 'packages=["a","b"]' to cause packages 'a' and 'b' to be
     added or removed from the distribution as appropriate.

A feature must include at least one 'requires', 'remove', or other
keyword argument.  Otherwise, it can't affect the distribution in any way.
Note also that you can subclass 'Feature' to create your own specialized
feature types that modify the distribution in other ways when included or
excluded.  See the docstrings for the various methods here for more detail.
Aside from the methods, the only feature attributes that distributions look
at are 'description' and 'optional'.

• Repository, issue tracker
• Hudson CI instance
• Sphinx documentation generated by hudson on each commit
• wiki/user stories for scrum development

Last two weeks were crazy at university. Now I shall have time to work on gpypi2 full-time until the end of program.

$ sudo pip install http://bitbucket.org/iElectric/g-pypi2/get/tip.zip

or

$ sudo easy_install http://bitbucket.org/iElectric/g-pypi2/get/tip.zip

$ sudo gpypi2 create flask

 * Generating ebuild: Flask 0.4
 * Your ebuild is here: /usr/local/portage/dev-python/flask/flask-0.4.ebuild
 * Dependency needed: Werkzeug
 * Dependency needed: Jinja2
 * Generating ebuild: Werkzeug 0.6.2
 * Your ebuild is here: /usr/local/portage/dev-python/werkzeug/werkzeug-0.6.2.ebuild
 * Generating ebuild: Jinja2 2.5
 * Your ebuild is here: /usr/local/portage/dev-python/jinja2/jinja2-2.5.ebuild
 * Dependency needed: Babel
 * Generating ebuild: Babel 0.9.5
 * Your ebuild is here: /usr/local/portage/dev-python/babel/babel-0.9.5.ebuild

Previous week (21st June - 5th July)

Task: Use SrcUriNamer

gpypi2.enamer.SrcUriNamer is awaiting for quite 2 weeks now
to be finally used.

It's basically factory that generates SRC_URI and HOMEPAGE
according to mirror provider.
A lot of additional refactoring will be needed to Enamer to
use this utility instead of current guessing/parsing.

Task: Fix bugs reported by users

Lot of bugs are going to pop up; fix bugs, write tests and
add documentation.

Task: Increase test coverage

I'm still not satisfied with test coverage.
I'll try to increase it a bit more.
It's 10 days week of exams on my university, soon I will be able to put
in more effort in implementing features.

Here is a rundown of additional changes to code:

• slight improvements to documentation
• option to choose PyPi base URL
• changed behaviour to use XML-RPC and SETUPTOOLS/SIMPLEPAGE method for retrieving SRC_URI
• updated licences dictionary mapping
• fixed echo command to work properly (without cleanup atm)

Upcoming week (22nd June - 28th June)

Task: Write configuration skeleton

Currently, gpypi2 code only takes configuration from command line.
I'd like to extend that to config file (.ini), get data directly from setup.py
and interactive questionaire. In order for that to function properly,
configuration needs to be done in right sequence.

Following week I will write down the design and implement initial working prototype.

Task: sync command - populate overlay with successful ebuilds from pypi

When I will get bored or stuck with task #1, I'll start working on command that will
generate ebuilds from all packages in pypi to an overlay.

• Repository, issue tracker
• Hudson CI instance
• Sphinx documentation generated by hudson on each commit
• wiki/user stories for scrum development

I'm more than happy to announce first pre-alpha prototype. Intended goal is to make people start using the tool and reporting back their critics.

$ sudo pip install http://bitbucket.org/iElectric/g-pypi2/get/tip.zip

or

$ sudo easy_install http://bitbucket.org/iElectric/g-pypi2/get/tip.zip

$ sudo gpypi2 create flask

 * Generating ebuild: Flask 0.4
 * Your ebuild is here: /usr/local/portage/dev-python/flask/flask-0.4.ebuild
 * Dependency needed: Werkzeug
 * Dependency needed: Jinja2
 * Generating ebuild: Werkzeug 0.6.2
 * Your ebuild is here: /usr/local/portage/dev-python/werkzeug/werkzeug-0.6.2.ebuild
 * Generating ebuild: Jinja2 2.5
 * Your ebuild is here: /usr/local/portage/dev-python/jinja2/jinja2-2.5.ebuild
 * Dependency needed: Babel
 * Generating ebuild: Babel 0.9.5
 * Your ebuild is here: /usr/local/portage/dev-python/babel/babel-0.9.5.ebuild

Previous week (15-21 June)

Main goal of previous week was to get closer to first prototype. Done.

Task: First snippet of command line interface

Put enough together to assemble ebuilds from command line. I think
many issues may pop up, and lots of bugs shall be fixed. I'm currently
mostly doing stuff from my head, with not much practise whatsoever. 

Mostly, we should get a result of better test coverage (currrently about
50%).

The only component left untouched is the configuration. I would like to
touch this topic after all my exams are over, leaving me time and
productivity.

Command line interface was added, test coverage is up to 60%. Lots of small fixes were done. As a bonus, I added portage output syntax colors.

Task: get_dependencies - another refactor to enamer

Title says it all. Also handling of multiple version operators should be
added.

Example: foobar>=0.5&& foobar<0.6

should resolve in <0.6 blocker and >=0.5 requirement.

Multiple operators on version dependencies are handled as blockers, implemented in tip. However get_dependencies was not refactored, since I still might be adding functionality. Not really necessary for now.

Upcoming week (22-28 June)

Task: Use SrcUriNamer

gpypi2.enamer.SrcUriNamer is awaiting for quite 2 weeks now
to be finally used.

It's basically factory that generates SRC_URI and HOMEPAGE
according to mirror provider.
A lot of additional refactoring will be needed to Enamer to
use this utility instead of current guessing/parsing.

Task: Fix bugs reported by users

Lot of bugs are going to pop up; fix bugs, write tests and
add documentation.

Task: Increase test coverage

I'm still not satisfied with test coverage.
I'll try to increase it a bit more.
It's 10 days week of exams on my university, soon I will be able to put
in more effort in implementing features.

• 30 elderberry blossoms
• 5L of water
• 3kg of sugar
• big spoon of Citronka
• about six 1L glass bottles with non-plastic closure

Getting blossoms

Blossoms should be chipped with scissors right after all small blossoms group together. As recipe says, 30 medium sized blossoms are recommended, but extra 4 or 5 will do no harm. Blossoms should not be put in plastic bag while collected, since they are very sensitive. Blossoms should be carefully picked, since we are not washing them before making syrup, just make sure there are no bugs around.

Procedure

We take a big container dish, put about 5L of water inside (important, container should be able to hold atleast 6L if not 7L). Inside, place 30 blossoms and make them sink. Close the container with a lid and let it stand for one day.

Next day, use sieve to remove blossoms from container, put 3kg of sugar inside and mix it. You will notice sugar staying at the bottom of the container, that's normal. Let the container stay closed with a lid for one more day and in that time, mix it as many times as you remember. The goal here is to have no sugar the bottom of the container.

Last day, we only have to add Citronka. I belived it is selled only in Slovenia and BiH, try to get something similar (it's basically Citric acid ready to make drinks). Its really hard to say how much should be put inside the container, I used the whole bottle for resulting 30L of syrup.

Now, the syrup should be ready to be tasted. Notice that dosage for 2dl glass should be a little bit more than for syrups you can buy in stores. Add more sugar according to taste and mix. We are ready to put the syrup in bottles. There is no special magic, funnel and gauze ( to filter small parts that normally float in the syrup) are your friends.

At the end out of 5L of water we get about 6-7L of syrup.

Release

Finally we have 30 bottles of our beloved syrup: