How to search super-quickly in all the remote Gentoo overlays with eix-remote

You probably know eix the portage indexer. If not, you should use it.
I just discovered that and it is awesome, too bad the documentation doesn't explain/sell it well enough.
First add a remote index, gpo is building a cache for you :

eix-remote fetch "http://gpo.zugaina.org/eix_cache/eix-cache.tbz2"

Then, tell eix to not drop this remote index after eix-update by setting KEEP_VIRTUALS=true in your /etc/eixrc:

# cat /etc/eixrc 
KEEP_VIRTUALS=true

Update your remote index !

eix-remote update

Now if you want to search in the remote indexes, just add -R (for remote) in your search

eix -R scratch

Eix will differentiate clearly the remote indexed ones from the local / portage ones

Enjoy !

skypipe : a cloud based named pipe

skypipe is a really nice idea about an ubiquitous named pipe on the net.

Basically it allows you to pipe data on a shell from one machine to another without knowing/doing anything, no IP to know, no tunnel to setup, nothing.

See at the end of the post how to set it up.

I show you a small demo here with the default anonymous pipe:

.
MachineA # skypipe < /etc/issue.logo

-------------

MachineB # skypipe

     .vir.                                d$b
  .d$$$$$$b.    .cd$$b.     .d$$b.   d$$$$$$$$$$$b  .d$$b.      .d$$b.
  $$$$( )$$$b d$$$()$$$.   d$$$$$$$b Q$$$$$$$P$$$P.$$$$$$$b.  .$$$$$$$b.
  Q$$$$$$$$$$B$$$$$$$$P"  d$$$PQ$$$$b.   $$$$.   .$$$P' `$$$ .$$$P' `$$$
    "$$$$$$$P Q$$$$$$$b  d$$$P   Q$$$$b  $$$$b   $$$$b..d$$$ $$$$b..d$$$
   d$$$$$$P"   "$$$$$$$$ Q$$$     Q$$$$  $$$$$   `Q$$$$$$$P  `Q$$$$$$$P
  $$$$$$$P       `"""""   ""        ""   Q$$$P     "Q$$$P"     "Q$$$P"
  `Q$$P"                                  """


This is \n.\O (\s \m \r) \t

Magical isn't it ?

skypipe takes a parameter to give you a named pipe, so alternatively you can do the more precise :

.
MachineA # skypipe gentoologo < /etc/issue.logo

-------------

MachineB # skypipe gentoologo

     .vir.                                d$b
  .d$$$$$$b.    .cd$$b.     .d$$b.   d$$$$$$$$$$$b  .d$$b.      .d$$b.
  $$$$( )$$$b d$$$()$$$.   d$$$$$$$b Q$$$$$$$P$$$P.$$$$$$$b.  .$$$$$$$b.
  Q$$$$$$$$$$B$$$$$$$$P"  d$$$PQ$$$$b.   $$$$.   .$$$P' `$$$ .$$$P' `$$$
    "$$$$$$$P Q$$$$$$$b  d$$$P   Q$$$$b  $$$$b   $$$$b..d$$$ $$$$b..d$$$
   d$$$$$$P"   "$$$$$$$$ Q$$$     Q$$$$  $$$$$   `Q$$$$$$$P  `Q$$$$$$$P
  $$$$$$$P       `"""""   ""        ""   Q$$$P     "Q$$$P"     "Q$$$P"
  `Q$$P"                                  """


This is \n.\O (\s \m \r) \t

Skypipe even bufferize your output so it doesn't require any precise timing to do it.

Setup

A prerequisite is to go to https://www.dotcloud.com/ and request a free account (it is relatively hassle free BTW).

Once you get that, you can install skypipe from my overlay if you are under gentoo :

# layman -L
# layman -a gbin
# echo "=net-analyzer/skypipe-9999 **" >> /etc/portage/packages.keywords

# emerge skypipe

Under other systems you will need for now to go with a manual setup :

# git clone git://github.com/progrium/skypipe.git
# cd skypipe
# python setup.py install

Then simply starts once the skypipe command on both machines so it first deploys itself to your newly dotcloud account then asks you and remember your credential for access on any new machine.

# skypipe                      
==> First time use, please setup dotCloud account
dotCloud username: gbin
Password: 
==> Launching skypipe satellite. This only has to happen once for your account.
==> This may take about a minute...
==> DEBUG: Found satellite

Thanks to Jeff Lindsay  for this cool tool.
Edit: removed the useless cat usage thx to sigil

Backup a gmail / pop3 account straight from command line to an mbox

I found this tip on the net and I love it.

emerge getmail
or pip install getmail

then follow the guide there  : http://howto.wired.com/wiki/Make_a_Local_Backup_Of_Your_Gmail_Account#Backup_Gmail_Using_Getmail

Why do I love to break my system ?

Beeing on the linux gentoo beta (~amd64) for my desktop machines is often seen by people as a pure "loss of productivity" as a software developer.

Yes it brakes sometimes... But for me those breakages (most of the time) worth it, it keeps me aware of the inner workings of my system.

When would I look for how the bluetooth system is related to udev if my BT mouse did not work this morning after the udev upgrade ?

This kind of system knowledge gives you as a software engineer some depth, it makes you see a larger picture than just the logic of your code. It makes you a better developer.

Once I helped a medior developer to solve a performance issue in his Java code he could not solve :
The classic trap of the database round tripping. Retreiving a list of ids then going to the database for each one of them. His answer when I finger pointed it was : "but Java is optimizing that !!".
The guy had no clue about system, networking, database drivers, database optimization engine, disk accesses.. For him he is just working on a "magical" system that applies his logic the most efficient way.

As a general attitude, I really recommend to my fellow engineers : break your systems ! hack your kernel, tinker with compilation options, try out new stuff like wayland or any other experimental systems... Push back the limits, learn what your code is calling, then what this code is calling then again and again until you find yourself reading about how a CPU uses his caches lines and optimize branch predictions...

At the end of the day it will make you a better software engineer, perhaps you will loose some immediate productivity but at the end of the day you will produce better code and be able to solve problems that are way more complex than simple logic flaws in your code path.

A stupid Gentoo tip I should have though about before ;)

To break this really annoying pattern on restricted packages :
You are root, you do your "emerge --keep-going --ask --update --newuse --deep --oneshot -t world"
and boom this dreaded restricted package pops in front of you:


Fetch instructions for dev-java/oracle-jdk-bin-1.7.0.5:
 * Please download jdk-7u5-linux-x64.tar.gz from:
 * http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1637583.html
 * and move it to /usr/portage/distfiles

I just hate this : I need to go to a browser (as my normal user of course), download it, come back to the CLI, do the move like "mv /home/gbin/downloads/jkd-blahblah /usr/portage/distfiles"

But when you think about it, as everything is sha/md5 in portage, the distfiles directory can be considered a public directory... so just : 

chmod 777 /usr/portage/distfiles 

Then straight at the warning before emerging, click, dl straight to it .. no more moving gymnastic between users.

Small gentoo tip : make eselect profile usable with an overlay

You can add profiles in an overlay but the selection has to be manual : you cannot use eselect profile out of the box.

# eselect profile list

Available profile symlink targets:
  [1]   default/linux/amd64/10.0
  [2]   default/linux/amd64/10.0/selinux
  [3]   default/linux/amd64/10.0/desktop
  [4]   default/linux/amd64/10.0/desktop/gnome
  [5]   default/linux/amd64/10.0/desktop/kde
  [6]   default/linux/amd64/10.0/developer
  [7]   default/linux/amd64/10.0/no-multilib
  [8]   default/linux/amd64/10.0/server
  [9]   hardened/linux/amd64
  [10]  hardened/linux/amd64/selinux
  [11]  hardened/linux/amd64/no-multilib
  [12]  hardened/linux/amd64/no-multilib/selinux

But you can make eselect think that your main portage directory is somewhere else:

# PORTDIR=/var/lib/layman/my-overlay eselect profile list 

Available profile symlink targets:
  [1]   awesome/linux/amd64/server/

# PORTDIR=/var/lib/layman/my-overlay eselect profile set 1

And if you check, yes indeed, it works !

# ll /etc/ma*
-rw-r--r-- 1 root root 2.2K Apr  6  2011 /etc/mailcap
-rw-r--r-- 1 root root  515 Feb 11 08:03 /etc/make.conf
-rw-r--r-- 1 root root  539 Feb 24  2011 /etc/make.conf.catalyst
lrwxrwxrwx 1 root root   40 Feb 11 08:05 /etc/make.globals -> ../usr/share/portage/config/make.globals
lrwxrwxrwx 1 root root   71 Feb 11 11:23 /etc/make.profile -> ../var/lib/layman/my-overlay/profiles/awesome/linux/amd64/server/
-rw-r--r-- 1 root root 4.6K Feb 24  2011 /etc/man.conf

You can of course alias it for the sake of userfriendliness :

alias ep='PORTDIR=/var/lib/layman/my-overlay eselect profile'

Small tip : combine eix and emerge to fix quickly your gentoo

This morning, as usual I forgot to rebuild x11 drivers after a minor update of xorg-server and boom, xdm starts, can't do anything even going back to a console. So reboot, pressed 'I' like a monkey during the openrc execution (yes, sub-trick, always enable Interactive boot) So here is the trick when you finally reached the console Here, to simply rebuild xf86 related stuff.

eix -I --only-names xf86 | xargs emerge -p1

-I = display only installed packages --only-names = make them emerge compatible in output format -p = pretend, so you can check if you won't make a big mistake -1 = don't change your world file : it avoids added unnecessary hard dependencies on your system Once it is ok, fire it for real.

eix -I --only-names xf86 | xargs emerge -1

How to apply User patches in portage

After years of tinkering with gentoo and feeding my custom overlay I discovered today the "user patches" feature of portage.

This is really awesome !

The Use Case :
You want to just try out a patch for something already in portage.

Instead of creating another ebuild in an overlay you can put your patch in :
/etc/portage/patches/{category}/{package_name}/

For example :

mkdir -p /etc/portage/patches/sys-libs/glibc-2.13-r3/
wget -O /etc/portage/patches/sys-libs/glibc-2.13-r3/fix1.patch "https://bugs.gentoo.org/attachment.cgi?id=279269"
emerge -1 glibc

To check if your patch applied you should see the following just after the standard batch of patches:

 * Applying user patches from /etc/portage/patches/sys-libs/glibc-2.13-r3 ...
 *   fix1.patch ...                                                                                 [ ok ]
 * Done with patching

poor man's backup and defragmentation process under linux for my desktops

No need to defragment linux ever ?! Try that and see for yourself.

As the title of this blog says I'm using gentoo. I just love this system. But compiling often has a drawback, it just spreads files around your filesystem all the time.

But there is no defragmenter yet under linux. So here is what I do to backup AND defrag my boxes.

First burn or even better, usb install the awesome SystemRescueCd.

Boot under systemrescueCD.

From there, mount back your main partition (here sda1) and do some cleanup prior to backup:

mount /dev/sda1 /mnt/gentoo
rm -Rf /mnt/gentoo/tmp/*
rm -Rf /mnt/gentoo/var/log/* # warning : if you don't care about your logs !
rm -Rf /mnt/gentoo/usr/portage/distfiles/* # you can always redownload them 
# etc ...

Important : umount the partition after cleanup

cd /
umount /mnt/gentoo

Now mount an external HD or your backup volume (here sdb1) :
Note : don't do that on FAT !! It must be something that we can name "filesystem" like ext4, xfs etc .. With at least the partition size you want to backup as free space of course.

mount /dev/sdb1 /mnt/backup

Dump an image of your main partition on it:

pv /dev/sda1 > /mnt/backup/sda1.img

You should see on the right side how many coffees you can drink during the dump.

Remount your backup read-only and check out quickly if the backup was successfull :

mount -o loop,ro /mnt/backup/sda1.img /mnt/custom
ll /mnt/custom

Now be extra careful, format your main partition (here sda1, the same as above), no typo are authorized here :

mkfs.ext4 /dev/sda1

Optional step for those who uses labels/UUID : restore them !

cat /mnt/custom/etc/fstab # to get whatever you use to mount your partition
tune2fs /dev/sda1 -U 4c7556bf-5c6c-4f85-bc45-d7fa55c8ca1d # for example for UUID

mount your all nice and clean main partition and restore the files from your backup.

mount /dev/sda1 /mnt/gentoo
rsync -av --progress /mnt/custom/ /mnt/gentoo/ #be careful the ending slashes ARE important

You'll notice how this is piece of cake for your main harddrive to cope with the incoming files and how painful it will be for your backup drive to be able to feed it from the files splashed all over the image (even from raid0 external disks on esata to a poor performing laptop drive !).

If your console slows down the process you can switch to another one and do df to see /dev/sda1 catching up the Use% of /mnt/backup/sda1.img.

Pack up, and enjoy the speed boost.

umount /mnt/gentoo
umount /mnt/custom
umount /mnt/backup
reboot

Mount Xoom/Android filesystem while in debug mode under gentoo

Step 1: Create a local overlay if you don't have one.
(if you already have one, skip to step 2)

mkdir -p /usr/local/portage/myoverlay

in /etc/make.conf:

source /var/lib/layman/make.conf # should be there already from layman
PORTDIR_OVERLAY="/usr/local/portage/myoverlay $PORTDIR_OVERLAY"

... if you have layman installed or :

PORTDIR_OVERLAY="/usr/local/portage/myoverlay"

Step 2: Put this ebuild in your local overlay in sys-fs/mtpfs

cd /usr/local/portage/myoverlay
mkdir -p sys-fs/mtpfs
cd sys-fs/mtpfs

mtpfs-0.9.ebuild

# Copyright 1999-2009 Gentoo Foundation
# Copyright 1999-2009 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $

EAPI="2"

DESCRIPTION="MTPFS is a Fuse filesystem based on libmtp"
HOMEPAGE="http://adebenham.com/mtpfs"
SRC_URI="http://adebenham.com/debian/mtpfs_${PV}.orig.tar.gz"

LICENSE="GPL-2"
SLOT="0"
KEYWORDS="~x86 ~amd64"
IUSE="debug"

DEPEND="sys-fs/fuse
      >=dev-libs/glib-2.18.4-r1
      media-libs/libmtp"
RDEPEND="${DEPEND}"

src_configure() {
   econf $(use_enable debug) || die 'econf failed'
}

src_compile() {
   emake || die 'emake failed'
}

src_unpack() {
    unpack ${A}
    mv ${P}.orig ${P}
}


src_install() {
   emake DESTDIR="${D}" install || die 'installation failed'
   dodoc AUTHORS INSTALL NEWS README ChangeLog
}

pkg_postinst() {
   einfo "To mount your MTP device, issue:"
   einfo " /usr/bin/mtpfs <mountpoint>"
   einfo
   einfo "To unmount your MTP device, issue:"
   einfo " /usr/bin/fusermount -u <mountpoint>"
} 

Step 3: Install it

ebuild mtpfs-0.9.ebuild digest
emerge mtpfs

Step 4: Mount and enjoy
Be sure your device is connected and in debug mode.

mkdir /mnt/xoom
mtpfs -o allow_other /mnt/xoom

Credits to thebytes, I merely improved what he was doing by fixing the ebuild from the original gentoo bugreport.

Speedup compilation with a distcc cross compilation on gentoo

This is a shortened version of the gentoo tutorial on crossdev & distcc.
I made it work between my Xoom running a chrooted gentoo and my i7 desktop machine.

On your slave(s) :

emerge sys-devel/crossdev

in make.conf set PORTDIR_OVERLAY to a your local custom overlay if is not done already like :

PORTDIR_OVERLAY="/usr/local/portage"

Compile the toolchain for the Xoom architecture:

crossdev -s4 --target armv7a-hardfloat-linux-gnueabi
emerge distcc

configure /etc/conf.d/distccd to accept incoming requests from the xoom for example, I also changed the log-level so I can see in the syslog (/var/log/messages) which compilation units are sent to my desktop:

DISTCCD_OPTS="${DISTCCD_OPTS} --allow 192.168.1.0/24"
DISTCCD_OPTS="${DISTCCD_OPTS} --log-level notice"

Start the daemon :

/etc/init.d/distccd start

On the master (Xoom) :
Before starting be sure you have sys-devel/binutils-2.21 or better or you will hit this bug : http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=595649

emerge ">=sys-devel/binutils-2.21"
emerge distcc

in /etc/distcc/hosts:
add the ip of your pc machine ie. 192.168.1.200
remove 127.0.0.1 (No need to run the daemon locally)

in /etc/make.conf add distcc feature:

FEATURE="distcc -userfetch"
MAKEOPTS="-j18" # something like nb of cpu in your farm x2

Don't forget to regenerate your environment :

env-update

emerge something (that compiles in C or C++ ;) ) on your Xoom/master, tail /var/log/messages on the slave to see if the compilation occurs correctly

Running Gentoo Linux (as a chroot) on the Motorola Xoom

Prerequisites

• a rooted xoom
• busybox installed
• a terminal emulator (Better Terminal Pro works perfectly)
• a large enough & preferably fast  microsd card. I used an 8Gigs class 6.
• check if you have nothing important to backup on the card before starting
• optional : a bluetooth keyboard

First step : prepare an sdcard


Plug your card with a reader / adapter on a standard linux.
It should appear like a /dev/mmcblk0

Wipe out the partition table and create one big linux fs:

fdisk /dev/mmcblk0

remove all partitions: d d d
re-add one, primary with the default linux type with the default size (all disk)
n then p then enter enter

Create an ext4 file system:

mkfs.ext4 /dev/mmcblk0p1

Mount the fs somewhere

mkdir -p /mnt/gentoo
mount /dev/mmcblk0p1 /mnt/gentoo
cd /mnt/gentoo

Second step : copy over the stage3 and portage

Browse to http://www.gentoo.org/main/en/mirrors2.xml and find a good mirror for you or if you have already a gentoo machine do :

emerge mirrorselect
mirrorselect -s 1 -o

You need to find a stage3 of armv7 arch with hardfp and undecompress it where you have mounted your sdcard :

cd /mnt/gentoo
wget http://mirror.ovh.net/gentoo-distfiles/releases/arm/autobuilds/current-stage3-armv7a_hardfp/stage3-armv7a_hardfp-20110503.tar.bz2
tar xjvf stage3*.tar.bz2
rm stage3*.tar.bz2

Then you need a portage snapshot and uncompress it in the right directory:

cd /mnt/gentoo/usr
wget http://mirror.ovh.net/gentoo-distfiles/releases/snapshots/current/portage-latest.tar.bz2
tar xjvf portage-latest.tar.bz2
rm portage-latest.tar.bz2

Third step : adjust some settings
in /mnt/gentoo/etc/make.conf adjust your mirrors and, very important disable the userfetch feature:

[...]
SYNC="rsync://rsync.europe.gentoo.org/gentoo-portage"
GENTOO_MIRRORS="http://mirror.ovh.net/gentoo-distfiles/"
FEATURES="-userfetch" # on android non-priviledged users like portage has no right to open a socket due to CONFIG_ANDROID_PARANOID_NETWORK in the kernel config

Note : you can also add MAKEOPTS="-j2" for parallel building

create /mnt/gentoo/root/.profile like this :

export TERM=xterm
env-update
cd

Unmount your card, put it in your Xoom and restart it.

Chroot in your gentoo
Here is a little script that you can put for example in /sdcard/gg.sh :

mkdir -p /sdcard/gentoo
mountpoint -q /sdcard/gentoo || mount -t ext4 -o noatime /dev/block/mmcblk0p1 /sdcard/gentoo
echo "nameserver `getprop dhcp.wlan0.dns1`" > /sdcard/gentoo/etc/resolv.conf
mountpoint -q /sdcard/gentoo/dev || mount -o bind /dev /sdcard/gentoo/dev
mountpoint -q /sdcard/gentoo/dev/pts || mount -o bind /dev/pts /sdcard/gentoo/dev/pts
mountpoint -q /sdcard/gentoo/sys || mount -o bind /sys /sdcard/gentoo/sys
mountpoint -q /sdcard/gentoo/proc || mount -t proc proc /sdcard/gentoo/proc
export HOME=/root
export TERMINFO=/etc/terminfo
chroot /sdcard/gentoo /bin/bash -l

then do from adb shell or your favorite terminal on the device :

su
source /sdcard/gg.sh

Enjoy your new gentoo environment !

---

Edit1:
- added a check to avoid remounts when you reexecute the script
- overrided the shell variable better terminal was putting

---

Edit2 2011-05-28: With the 3.1 update, the sdcard is mounted directly under /sdcard/external:
/dev/block/mmcblk0p1 /mnt/sdcard/external ext4 rw,relatime,barrier=1,data=ordered 0 0

So you can safely reuse it directly, here is the alternative script.

echo "nameserver `getprop dhcp.wlan0.dns1`" > /sdcard/external/etc/resolv.conf
mountpoint -q /sdcard/external/dev || mount -o bind /dev /sdcard/external/dev
mountpoint -q /sdcard/external/dev/pts || mount -o bind /dev/pts /sdcard/external/dev/pts
mountpoint -q /sdcard/external/sys || mount -o bind /sys /sdcard/external/sys
mountpoint -q /sdcard/external/proc || mount -t proc proc /sdcard/external/proc
export HOME=/root
export TERMINFO=/etc/terminfo
chroot /sdcard/external /bin/bash -l

Few tricks to sort out and present recovered pictures

Often after a recovery with photorec/testdisk or any other recovery tools, you just have a flat list of files named by their index where the tool found it on the image.
Some are corrupted, plenty are irrelevant.

Let's say you have a bunch of jpeg to sort out.
So first, a tool to check quickly which one are corrupted or not :

On Gentoo:

emerge media-gfx/jpeginfo

Then from the directory where you have your flat list of images :

mkdir ok_images
jpeginfo -c *.JPG | grep "\[OK\]" | cut -d " " -f 1 | xargs -I {} mv {} ok_images/

In plain English : This will check the integrity of the images, find back those OK in the list, isolate the name of the file and pass the file names one by one to the mv command.

Next step, produce a nice picture composed from the thumbnails of those images:
The tool is imagemagick :

emerge media-gfx/imagemagick

Be careful here, this can take a huge bunch of ram so don't run it with a root account or you risk to have your user processes killed :). Monitor the progress with for example htop.

montage -set label '%f' *.JPG -geometry 128x128+2+2 thumbnails.jpg &
htop

If it OOM, add swap or do it subset by subset.

A simple alias to be able to update your gentoo instance unattended

Starting a large update like KDE, going to bed and discovering that it failed on the not really important 3rd package the next morning is kind of irritating.

I have this defined in my /root/.bashrc (you need to have also the .bash_profile so it is triggered by the way).

alias up="emerge --keep-going --ask --update --newuse --deep world"

It means, update anything you need to but don't stop if one package fails.

As I have app-portage/eix installed, my update routine is :

eix-sync
up

Both with a nice feedback about what's new in the batch.