One liners for encoders

Multimedia capabilities of portable and non PC like devices is growing every day, but mostly limited to variations of MPEG standards variations. On the other hand on PCs there seems to appear a new format once a week.
Luckily there are a few good and free software for converting media formats.

My favourite A/V container is Matroska (.mkv) - can hold multiple .. everything (audio, video, subtitles tracks) :D.
Mkvtoolnix is a awesome toolkit for playing with A/V files - for example splitting & joining files, even from different formats.
GUI is easy to use, so here's some examples of shell tools usage

mkvextract  tracks       example.mkv  1:video.mkv  2:audio.aac 3:english.ass
mkvextract  attachments  example.mkv  1:cover.jpg  2:arial.ttf

For some reason I can't get my TV to play audio from DTS sources.
I used to use FLAC (lossless audio) encoder as a pass-thru for encoding audio to AAC. Here are batches for that - just put them somewhere %PATH% points :

1. Whatever (ffmpeg can read/decode) to FLAC

"C:\Program Files\mkvtoolnix\ffmpeg.exe" -i %1 -vn -sn -acodec flac %2

2. FLAC to HEv1 AAC

@ECHO OFF
cls
set tmp=%CD%
set temp=%CD%
"C:\Program Files\mkvtoolnix\flac.exe" -d -c %1 | "C:\Program Files\mkvtoolnix\neroAacEnc.exe" -he -q 1 -if - -of %2
@PAUSE

The set tmp=%CD% part is useful if you use ramdisk, without it NeroAacEnc uses default "c:\SomeWhere\Temp",
In both batches first parameter is input file, second points to where to write.

I've found a better way for doing that : direct decode whatever audio format to aac

"C:\Program Files\mkvtoolnix\ffmpeg.exe" -i %1 -vn -sn -acodec pcm_s16le -ar 48000 -f wav - | "C:\Program Files\mkvtoolnix\neroAacEnc.exe" -he -q 1 -if - -ignorelength -of %2

And there's also ffmpeg's built in AAC encoder
ffmpeg -i /media/funny.mp4 -vcodec copy -ab 512k -ac 6 -acodec aac -strict experimental /media/funny.mkv


There's a popular opinion around the Net that Nero AAC Encoder gives better results than free libfaac.
Just google for : nero ftp NeroAACCodec-1.5.1.zip - it has encoder version 1.5.4.0 from February 2010.

Help info (-help) says that HEv1/v2 is selected automatically, but I always got LC profile on my encodes, so I started forcing HEv1 (-he) - didn't notice quality difference, but size reduction is significant.

Here's a wine variant (or you can use Linux binaries and skip the "wine") :

wine flac.exe -d -c /media/example.flac | wine neroAacEnc.exe -he -q 1 -if - -of /media/example.aac

Other fun example : got lots of photos from a trip, take a lot of space, so what's the best way to compress them and save on storage space ?
2 pass slow encode from jpg to 1 FPS x264 - got 200MB movie from 900MB of photos, perfect for small SD phone card.

ffmpeg -f image2 -r 1 -i "%d.jpg" -vcodec libx264 -pass 1 -passlogfile /media/1pass -sameq -preset slow /media/temp.mkv

ffmpeg -f image2 -r 1 -i "%d.jpg" -vcodec libx264 -pass 2 -passlogfile /media/1pass -sameq -preset slow -vb 3400k /media/trip.mkv

MPEG (x264 is MPEG4) compression doesn't stores separate frames individually like MJPEG, but looks for similar blocks in neighboring frames and stores differences, plus other complicated size reduction methods of course.
Images have to be named like 1.jpg, 2.jpg .. 10.jpg, .. 999.jpg etc. While first pass is fairly fast, the second encoding can drop to 1-2 FPS with hi-res pics - but size/quality is awesome - or you can use -preset normal.
For bit rate (-vb) I use average file size from the pics list.


Observation : my Philips TV from 5000 series can only display JPEGs compressed without Progressive methode, also can only play mkv with one audio track.

Izismile has option to download their videos in MP4 or FLV format - if you need sources to experiment.

Wonderful world of RamDisks

Current market situation :
- HDD disks prices are almost twice that before floods in Asia
- SSD disks still very price, an still short writing life span
- RAM cheaper than ever

I've been trying to reduce wear of my office and home HDDs for years now, mostly by disabling usage of swap and tweaking mount options (Linux).

Example of one of my /etc/fstab

/dev/sda1 /   ext4 defaults,noatime,nodiratime 0 1
tmpfs  /media   tmpfs nodev,nosuid,size=3g  0 0

"noatime" disables updating time stamp of accessing file - "access" means reading file, not only modification time like on Windows. Not sure if nodiratime is needed - I've read few different info on the subject.

When RAM got cheap I've upgraded my trusty 6 years old personal notebook from 1 to 3 GB. System caching of often used files is great on both Linux and Windows, but I haven't found a way to prioritize which files I would prefer to buffer.
And what if you need to work on a file you will change many times in short time frame ? From what I read SSD disk based on SLC can withstand about 100k writes (MLC about 1/10 of that), seems like much but for a coder working on big project - not so much.

Ramdisk is virtual disk that uses part of system memory. System with UPS or note/netbook with healthy battery recommended, since dynamic memory looses data when not powered.

To no surprise setting up ramdisks on Linux is easy and doesn't need installation of any special software.
To types I know of :
- tmpfs - will use swap when needed, partition size limited at mount
- ramfs - none of the above

Mounting from shell

mount -t ramfs  ramfs  /ram
mount -t tmpfs -o size=3g  tmpfs /media

Which gives more or less this result

#mount
ramfs on /ram type ramfs (rw,relatime)
tmpfs on /media type tmpfs (rw,relatime,size=3145728k)

#df
ramfs          ramfs        0     0     0    - /ram
tmpfs          tmpfs     3.0G     0  3.0G   0% /media

I recommend tmpfs since its size can be limited, I've setup a 2.5GB ramdisk on a 3GB system - when it got full X crashed and system got stuck - live and learn :].


As for Windows, I've tried few free solutions - here's what I sticked too.

1. VSuite Free Edition
- simple/effective GUI
- a bit slow at mounting

2. My favorite IMDisk
- fast
- free, source code available

There's a neat GUI in Control Panel, but I prefer using batches for un-/mounting.
This mounts a FAT16 partition to a: , of size = first passed parameter

imdisk -a -t vm -s %1M -p "/fs:fat /y" -m a:

I use fat because I don't need journaling - ramdisk would disappear after reboot any how.

Forces dismounting a: ramdisk

imdisk -D -m a:

I'm heavily using IMDisk on 32b XP, but I encounter a small problem - I can't get ramdisk bigger than 1734 MB.
Besides awesome speed (at least 10x than HDD on DDR2), you can unmount ramdisk even if files are open - or something is holding a handle to them.

Keeping eth0 alive script

For some reason my servers at work are cursed with loosing connection to network - you have to connect with some app that send packets to keep network alive - for example Putty - or you won't be able to connect to server, without login in directly and pingin' some1. After little googlin' few years back, I thought it was related to net card driver. I've compiled few driver versions from vendors site but nothing changed. On second machine that I got later I had the same problem, and so did few of my co-workers with their machines. Apparently this behaviour can also mean that the router is dropping entries in arp tables for my machines. I don't have access to the router, so I've made a small hack/fix:

put this in file /etc/dhclient-eth0.conf

send dhcp-lease-time 1100;

This makes dhcp client renew lease every ~500 sec.

Of course I wasn't happy with this approach :). It depended on dhcp server to correctly work, could assign different ip etc. After changing to my eth0 configuration to static ip adress I had to think of other hack/fix.

Let's call it /usr/local/bin/keepalive.sh

#!/bin/bash

while [ -d /root ]
do
    if test -e /proc/net/nf_conntrack
    then
        cnt=`cat /proc/net/nf_conntrack | grep -v "192\.168" | grep EST | wc -l`
        if [ "${cnt}" -lt "3" ]; then
            ping -c 1 10.1.1.13  >> /dev/null 2>&1
        fi
    fi
    if [ ! -f /var/run/sshd.init.pid ]
    then
        break
    fi
    sleep 2m
done

Some explanation :
/proc/net/nf_conntrack - file containing information about network connection
grep -v "192\.168" - filters lines NOT including string "192.168", because I have virtual machines running on this server, with virtual network 192.168.1.0
grep EST - filter lines including connection status == ESTablished
wc -l - counts how many filtered lines were found

Instead of the whole line cnt=... you can just use :

cnt=`grep EST /proc/net/nf_conntrack | wc -l`

or if your system doesn't have /proc/net/nf_conntrack

cnt=`ss -an | grep -v "192\.168" | grep "ESTAB" | wc -l`

or

cnt=`netstat -an | grep -v "127\.0" | grep -v "192\.168" | grep "ESTABLISHED" | wc -l`

although ss is faster.

But to run it from crontab at boot I had to make a middle-man script to run my main script in background - scripts put in background directly from cron seem to time-out eventual and force cron to nag about it thou sendmail or syslog.

Let's call it /usr/local/bin/runbg.sh - not much to see here

/usr/local/bin/keepalive.sh &

And for the final touch, in /var/spool/cron/tabs/root add line

@reboot /usr/local/bin/runbg.sh >> /tmp/keepalive.log 2>&1

Kernel 3.3 and Xen 4.1.2 on openSuse 12.1

For over a year I used VMware Server 2 to run few virtual machines at work. But after every kernel update I had to look for a patch to build modules needed by VMS2. Eventually I got fed up with this procedure and started looking for a better alternative. Got XEN to run on Mandriva 2010, OS of choice at the time. Xen and kernel come as a team so no need to recompiling any thing, and it's a bit faster then VMS2. Sadly Mandriva 2011 didn't come with Xen full virtualization kernel, so I tried openSuse 12.1, after not so good experience with Fedora 16.
OpenSuse 12.1 comes with kernel 3.1.0 and xen 4.1.2. After updating to kernel 3.1.9, me and other users of virtual machines on my office server, noticed that net connection tends to break, for example when some1 starts his machine.

To allow every user to start VMs manually, I've added this to /etc/sudoers

%users  ALL=(ALL)         NOPASSWD: /usr/local/bin/xmst.sh

.. and /usr/local/bin/xmst.sh looks like this

#!/bin/bash

export LD_LIBRARY_PATH=/xen/lib64:/xen/lib
export LD_RUN_PATH=/xen/lib64:/xen/lib
export PYTHONPATH=/xen/lib64/python2.7/site-packages

/xen/sbin/xm start $1

I've renamed original xen init scripts

cd /etc/init.d/
rename -v xen z-xen xen*

1. Building kernel

# as root
cd /usr/src
curl -O http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.3.tar.bz2
tar xjf linux-3.3.tar.bz2
cd linux-3.3

a) configure
You can start with

make  allyesconfig

and just uncheck (in make menuconfig) what you don't need.

I've used .config from rpm of kernel-xen-3.1.9 as a base of my configuration.

DO NOT USE LZMA/XZ COMPRESSION FOR KERNEL/INITRD - Xen only supports gzip compression.

make menuconfig
make
make modules_install
make install

I've also copied the bare kernel image, just in case ;].

cp /usr/src/linux-3.3/vmlinux /boot/vmlinux-3.3

Building Xen

env PREFIX=/xen  make dist
make install

Now there are a few files to change. In /etc/init.d : xen-watchdog, xencommons, xend, xendomains (this one I don't use)

Here's what to add after INIT INFO block:

export LD_LIBRARY_PATH=/xen/lib64:/xen/lib
export LD_RUN_PATH=/xen/lib64:/xen/lib
export PYTHONPATH=/xen/lib64/python2.7/site-packages
export PATH=/xen/bin/xen/sbin:$PATH

I've prepended PATH with my build bins because I don't want to use those installed from rpm.

And added something like this /boot/grub/menu.lst

title Xen - openSUSE 12.1 - 3.3.0
    root (hd0,0)
    kernel /boot/xen-4.1.2.gz vga=mode-0x31a dom0_mem=4488M noreboot
    module /boot/vmlinuz-3.3.0-xen root=/dev/sda1 showopts console=tty7 xencons=tty8
    module /boot/initrd-3.3.0-xen

I've copied vms config.sxp to /xen/vm and set path in /xen/etc/xen/xend-config.sxp

(xend-domains-path /xen/vm/)

You can see that process running my vm has opened vnc access.

ps aux | grep xen

/xen/lib/xen/bin/qemu-dm -d 1 -domain-name dom1 -videoram 4 -k pl-pl -vnc 0.0.0.0:2,password -parallel none -vcpus 1 -vcpu_avail 0x1L -boot c -localtime -serial pty -acpi -usb -usbdevice tablet -net nic,vlan=1,macaddr=00:16:3e:3c:48:65,model=rtl8139 -net tap,vlan=1,ifname=tap1.0,bridge=virbr0 -M xenfv

I've added paths to my xen libs in /etc/ld.so.conf.d/xen.conf like so

/xen/lib
/xen/lib64

After running ldconfig I could probably comment-out those exports in /etc/init.d/my-xen*, but I did it mostly so I could run xentop without typing freaking long paths like this

env LD_LIBRARY_PATH=/xen/lib64:/xen/lib  PYTHONPATH=/xen/lib64/python2.7/site-packages  /xen/sbin/xentop

Since openSuse 12.1 includes custom confs for ld first, my libs will be used before those installed from rpm - in /usr/lib64.

ldd xenconsoled xenstored xenwatchdogd
xenconsoled:
        linux-vdso.so.1 =>  (0x00007fff3c288000)
        libxenctrl.so.4.0 => /xen/lib64/libxenctrl.so.4.0 (0x00007f6a35c82000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f6a35a7e000)
        libxenstore.so.3.0 => /xen/lib64/libxenstore.so.3.0 (0x00007f6a35874000)
        libutil.so.1 => /lib64/libutil.so.1 (0x00007f6a35671000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f6a35469000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f6a350d9000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6a34ebc000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f6a35ea5000)
xenstored:
        linux-vdso.so.1 =>  (0x00007fff64d85000)
        libxenctrl.so.4.0 => /xen/lib64/libxenctrl.so.4.0 (0x00007fe712f30000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fe712d2c000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fe71299c000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe71277f000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fe713153000)
xenwatchdogd:
        linux-vdso.so.1 =>  (0x00007fff6a149000)
        libxenctrl.so.4.0 => /xen/lib64/libxenctrl.so.4.0 (0x00007f5b01bb5000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f5b019b1000)
        libxenguest.so.4.0 => /xen/lib64/libxenguest.so.4.0 (0x00007f5b0178b000)
        libxenstore.so.3.0 => /xen/lib64/libxenstore.so.3.0 (0x00007f5b01581000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f5b011f1000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f5b00fd4000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f5b01dd8000)
        libz.so.1 => /lib64/libz.so.1 (0x00007f5b00dbc000)

After checking that everything works, I've added system wide path to my xen bins
-in /etc/profile.local

export PATH=/xen/bin:/xen/sbin:$PATH

Here's a example how to use xl - the replacement for xm

xl vcpu-pin dom1 0 0
xl vcpu-pin dom1 1 1
xl vcpu-pin dom2 0 2
xl vcpu-pin dom2 1 3

- pins physical cpu (cores) 0 & 1 to dom1s virtual cpu 0 & 1, and physical cpu 2 and 3 to dom2 virtual cpu 0 & 1.

Xen 4.2 is due this or next month, hope it will be as easy to install ;].

UPDATE

Here's .config from my xen kernel 3.3.
Weird thing is - I've tried same .config with kernel 3.3.1 when it came out - can't get the damned thing to work.
Kernel on it's on boot fine, but with xen - ends with blank screen and reboots (removed noreboot from grub) - kernel 3.3 & xen works fine, so I guess I'll take a crack at next kernel version (3.3.2 / 3.4).


UPDATE 2
First thing I did at work on monday morning was compile kernel 3.3.2 ... and Xen works again, woo hoo :D

Fun with TLSv1.1 and TLSv1.2

Over a half a year ago there was a big fuss over the BEAST attack method against SSLv3/TLSv1, since then ... basically nothing changed :D. As far as I know Opera is only browser supporting better/newer TLSv1.1 and TLSv1.2 - IE on Win7 and Win2008 Server supposedly also supports those protocols, will check at work and write when I can.

OpenSSL (CVS changes log) with version 1.0.1 will introduce support for improved TLS, GnuTLS had it for few years now.

Here are few "simple" steps to install and play around with the latest software without interfering in system.
I'm doing everything in /tmp dir since it's running on tmpfs (ramdisk). Zlib is used to present a fun way to use env and LD_LIBRARY_PATH, and maybe for those interested with really old systems.

1. Downloading sources

mkdir /tmp/src /tmp/opt
cd /tmp/src

wget http://zlib.net/zlib-1.2.6.tar.gz
aria2c http://ftp.gnu.org/gnu/gnutls/gnutls-3.0.15.tar.xz
curl -O ftp://ftp.openssl.org/snapshot/openssl-1.0.1-stable-SNAP-20120311.tar.gz

tar xzf zlib-1.2.6.tar.gz
tar xJf gnutls-3.0.15.tar.xz
tar xzf openssl-1.0.1-stable-SNAP-20120311.tar.gz

Hint : tar xjf some.tar.bz2 for bz2 compressed tars.

2. Compiling zlib

cd zlib-1.2.6
./configure --prefix=/tmp/opt/zlib --64
make && make install

3. Compiling openssl

cd ../openssl-1.0.1-stable-SNAP-20120311
./config --prefix=/tmp/opt/ssl --with-zlib-lib=/tmp/opt/zlib/lib --with-zlib-include=/tmp/opt/zlib/include zlib threads shared
env LD_LIBRARY_PATH=/tmp/opt/zlib/lib make && make install

4. Compiling gnutls

cd ../gnutls-3.0.15
./configure --prefix=/tmp/opt/tls --with-libz-prefix=/tmp/opt/zlib --disable-openssl-compatibility
env LD_LIBRARY_PATH=/tmp/opt/zlib/lib make && make install

Sure, you could create my.conf in /etc/ld.so.conf.d/ containing path /tmp/opt/zlib/lib and run ldconfig, but this way may cause a problem.

# ldd /tmp/opt/ssl/lib/libcrypto.so.1.0.0
 linux-vdso.so.1 =>  (0x00007ffff2da8000)
 libdl.so.2 => /lib/libdl.so.2 (0x00007f9a81116000)
 libz.so.1 => /usr/lib/libz.so.1 (0x00007f9a80f00000)
 libc.so.6 => /lib/libc.so.6 (0x00007f9a80b5e000)
 /lib/ld-linux-x86-64.so.2 (0x00007f9a81711000)

# env LD_LIBRARY_PATH=/tmp/opt/zlib/lib  ldd /tmp/opt/ssl/lib/libcrypto.so.1.0.0
 linux-vdso.so.1 =>  (0x00007fffe35ff000)
 libdl.so.2 => /lib/libdl.so.2 (0x00007f062222c000)
 libz.so.1 => /tmp/opt/zlib/lib/libz.so.1 (0x00007f0622016000)
 libc.so.6 => /lib/libc.so.6 (0x00007f0621c74000)
 /lib/ld-linux-x86-64.so.2 (0x00007f0622827000)

Depending on content /etc/ld.so.conf custom confs from /etc/ld.so.conf.d/ may be used prior or after paths with standard system libraries. This means either every binary linked against libz.so.1 will use system library or that build in /tmp/opt/zlib/lib (apps I just compiled would use /usr/lib/libz.so.1, or everything would use my libz) - and I didn't want that.

5. We will need keys/certs

export  LD_LIBRARY_PATH=/tmp/opt/zlib/lib
cd  /tmp/opt
ssl/bin/openssl req -new -nodes -out req.pem -keyout key.pem
ssl/bin/openssl  rsa -in key.pem -out new.key.pem
ssl/bin/openssl  x509 -in req.pem -out server.pem -req -signkey new.key.pem -days 3650

Remember that export variable=value expires after login out or exiting mc if you're using Ctrl+o.

6. Run gnutls as a simple http server

env LD_LIBRARY_PATH=/tmp/opt/zlib/lib  tls/bin/gnutls-serv -p 443 --http -g --x509certfile=server.pem --x509keyfile=key.pem

Ctrl+C to exit.

In Opera : Tools > Preferences > Advanced > Security > Security Protocols ... - leave only [Enable TLS 1.2] checked (1.1). You have to close and open Opera anew for the changes to take effect. Now open https://localhost/ - you have to [Approve] the Security Issue.

7. Run openssl as a simple http server
You can force TLS version by using -tls1_2, -tls1_1, -tls1 or disable like this -no_tls1_2 - you can have all {Security Protocols} in Opera enabled.

cd /tmp/opt
env LD_LIBRARY_PATH=/tmp/opt/zlib/lib  ssl/bin/openssl s_server -accept 443 -key new.key.pem  -tls1_2 -www

Ctrl+C to exit.

Well that was fun, right ;]. More fun with LD_LIBRARY_PATH soon.

SSL pubkeys to keystore update script

From what I've been told for Jboss to use other services (SMS, Smtp) over SSL, their public keys have to be added to keystore.
So i whipped out this piece of code.

#!/bin/bash

CRDIR=/opt/https
KEYTL=/opt/java/bin/keytool
KYSTR=/opt/test.keystore
LOG=/opt/cron_cert.log
LST=(smtp.gmail.com:465  ssl.hqsms.com:443)

date >> ${LOG}
echo "--== List count : ${#LST[*]}" >> ${LOG}

if [ ! -d ${CRDIR} ];then
    echo "!!! ERROR: Cert dir missing: ${CRDIR}" | tee -a ${LOG}
    exit 1
fi

cd ${CRDIR}
rm -f ${CRDIR}/*.tmp

for item in ${LST[*]}
do
    if [ ! -f ${item} ];then echo "dummy" > ${item}
    fi
done

CERTS=(*)

if [ "${CERTS[0]}" = "*" ];then
    echo "??? WARNING : No certs in ${CRDIR}" | tee -a ${LOG}
    if [ ${#LST[*]} -eq 0 ];then
        echo "!!! ERROR : Certs list is empty" | tee -a ${LOG}
        exit 1
    fi
else
    echo "--== Count of certs in ${CRDIR} : ${#CERTS[*]}" >> ${LOG}
fi

if [ ! -f ${KYSTR} ];then
    ${KEYTL} -genkey -noprompt -dname "CN=s, OU=s, O=s, L=s, ST=s, C=s" -alias fooxyz -keystore ${KYSTR} -storepass Somepass -keypass Somepass
    ${KEYTL} -delete -alias fooxyz -keystore ${KYSTR} -storepass Somepass
fi

#------------------------------------------
for CRT in ${CERTS[*]}
do
    echo "--== CERT :: ${CRT}" >> ${LOG}
    echo "Q" | openssl s_client -connect ${CRT} 2>>${LOG} | grep -B 100 "END CERTIF" | grep -A 100 "BEGIN CERTIF" > ${CRT}.tmp
    # gnutls-cli  --print-cert  -p 443  some.ip

    if [ -s ${CRT}.tmp ];then
        diff -qs ${CRT} ${CRT}.tmp >> ${LOG}
        rtn=`echo $?`
        if [ ! ${rtn} == 0 ]; then
            echo "--== Updating ${CRT}" >> ${LOG}
            mv -f ${CRT}.tmp ${CRT}
            ${KEYTL} -delete -alias ${CRT} -keystore ${KYSTR} -keypass Somepass -storepass Somepass >> ${LOG} 2>&1
            ${KEYTL} -importcert -noprompt -alias ${CRT} -file ${CRDIR}/${CRT} -keystore ${KYSTR} -storepass Somepass >> ${LOG} 2>&1
    fi
    else
        echo "!!! ERROR : Couldn't get cert from ${CRT}" >> ${LOG}
    fi
    rm -f ${CRDIR}/*.tmp
done

Still got one problem with this script - when adding a new cert to keystore, keytool will report false error about removing old key before update.

HTTPS redirecting with socat

Long story short : got jboss app (port 8443) that goes off-line for few minutes at night - data migration. For that time I wanted to redirect users to apache (port 443) page with proper info. I've tried two netcat processes with redirecting stdin & stdout, but it was awfully slow. I've found socat as a feature richer netcat alternative.

#!/bin/bash

/etc/init.d/jboss stop

dflt=`cat /proc/sys/net/ipv4/tcp_fin_timeout`
dflr=`cat /proc/sys/net/ipv4/tcp_tw_reuse`

echo ${dflt}
echo ${dflr}

echo 5 > /proc/sys/net/ipv4/tcp_fin_timeout
echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse

for i in {1..9}
do
    sclog=/tmp/socat${i}.log
    #sleep 1
    (nohup /opt/socat/bin/socat TCP-LISTEN:8443,fork tcp4-connect:127.0.0.1:443 > ${sclog} 2>&1)&
    sleep 5
    cnta=`grep -Ei "adres|alrea|use" ${sclog} | wc -l`
    if [ ! -s ${sclog} ] && [ ${cnta} == 0 ];then
        break
    else
        killall /opt/socat/bin/socat
    fi
    rm -f ${sclog}
done

rm -f ${sclog}

echo ${dflt} > /proc/sys/net/ipv4/tcp_fin_timeout
echo ${dflr} > /proc/sys/net/ipv4/tcp_tw_reuse

If you got commercial ssl cert only in keystore format - here's how to convert it to apache comptible format

/opt/java/bin/keytool -importkeystore -srcstoretype JKS -srcstorepass SomePass -srckeystore https.keystore -deststoretype PKCS12 -deststorepass SomePass -destkeystore https.pk12.der
openssl pkcs12 -in https.pk12.der -nodes -out apache.pem

Send mail with attachment - python script

I've needed a shell script to daily send a log of some cron job. So after some googlin' and mixing some examples I got this :

#!/usr/bin/env python
# -*- coding: utf_8 -*-

import sys
from os import path
from smtplib import SMTP
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from email.MIMEBase import MIMEBase
from email import Encoders

fromad = "UserName@gmail.com"
toaddr = "AdminLogin@gmail.com"

msg = MIMEMultipart('mixed')
msg['Subject'] = sys.argv[1]
msg['From'] = fromad
msg['To'] = toaddr

text = "Some text that says details are in attached log or something"
body = MIMEText(text, 'plain')
msg.attach(body)

part = MIMEBase('application', "octet-stream")
fp = open(sys.argv[2], 'rb')
part.set_payload( fp.read() )
fp.close()
Encoders.encode_base64(part)
part.add_header('Content-Disposition', 'attachment; filename="%s"' % path.basename(sys.argv[2]))
msg.attach(part)

sndm = SMTP()
sndm.set_debuglevel(0)
sndm.connect('smtp.gmail.com', '587')
sndm.starttls()
#sndm.ehlo()
try :
    #sndm.esmtp_features['auth'] = 'LOGIN DIGEST-MD5 PLAIN'
    sndm.esmtp_features['auth'] = 'LOGIN'
    sndm.login('UserName', 'PassWord')
    sndm.sendmail(fromad, toaddr, msg.as_string())
    sndm.quit()
except Exception, e:
    print e

Usage : ./script.sh "Some email subject" /tmp/cron.log

Works great on openSuse 12.1, on Arch Linux 2011 - not so much - have to change 1st line to :

#!/usr/bin/python2

To use SSL instead of StartTLS switch to SMTP_SSL and change to proper port - for gmail SSL it's 465

from smtplib import SMTP_SSL
#from smtplib import SMTP
.
.
sndm = SMTP_SSL()
#sndm = SMTP()
.
.
sndm.connect('smtp.gmail.com', '465')
#sndm.connect('smtp.gmail.com', '587')
.
.
#sndm.starttls()

On Arch Linux 2011 line

#!/usr/bin/env python

runs script under Python 3.2, not 2.7.

Here's a working version for Python 3.2 on Arch Linux.

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import sys
from os import path
from smtplib import SMTP_SSL
#from smtplib import SMTP
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from email.mime.base import MIMEBase
from email import encoders

fromad = "UserName@gmail.com"
toaddr = "AdminLogin@gmail.com"

msg = MIMEMultipart('mixed')
msg['Subject'] = sys.argv[1]
msg['From'] = fromad
msg['To'] = toaddr

text = "Some text that says details are in attached log or something"
body = MIMEText(text, 'plain', 'utf-8')
msg.attach(body)

part = MIMEBase('application', "octet-stream")
fp = open(sys.argv[2], 'rb')
part.set_payload( fp.read() )
fp.close()
encoders.encode_base64(part)
part.add_header('Content-Disposition', 'attachment; filename="%s"' % path.basename(sys.argv[2]))
msg.attach(part)

sndm = SMTP_SSL()
#sndm = SMTP()
sndm.set_debuglevel(1)
sndm.connect('smtp.gmail.com', '465')
#sndm.starttls()
#sndm.ehlo()
try :
    #sndm.esmtp_features['auth'] = 'LOGIN DIGEST-MD5 PLAIN'
    sndm.esmtp_features['auth'] = 'LOGIN'
    sndm.login('UserName', 'PassWord')
    sndm.sendmail(fromad, toaddr, msg.as_string())
    sndm.quit()
except Exception as e:
    print(e)

Biggest problem I found with Python 3.2 version is you can't use non-ascii letters for email subject - if you do script will stop with something like this

'ascii' codec can't decode byte 0xc4 in position 0: ordinal not in range(128)

To check which auth methods servers support you can either use

telnet plus.smtp.mail.yahoo.com 25
ehlo me
quit

or for SSL

openssl s_client -connect smtp.zoho.com:465
ehlo me
Q  -to quit connection

Arch Linux 2011 - post installation

Every now and then I like to fiddle with a new Linux distro or a pre-release version of one o few distros I'm familiar with. After a quick look at http://distrowatch.com I've picked Arch Linux, because :

- I use openSuse at work
- tried Fedora some months ago
- Debian is overall outdated
- I hate *buntu
- I've used Mandriva at work and home for few years

I usually don't bother to read instructions/documentation on distros I want to play with, until thing go really wrong - where's the fun in doing things by the book, right :D

Instalation process went fast and smoove, so lets skip to what I had to do after that.
Everything done loged in to shell as root.

1. Get IP adress by dhcp

dhcpcd -b eth0

2. Select pacman repos

nano /etc/pacman.conf

and uncoment few lines like this

[core]
#SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist

[extra]
#SigLevel = PackageOptional
Include = /etc/pacman.d/mirrorlist

[community]
#SigLevel = PackageOptional
Include = /etc/pacman.d/mirrorlist

- to save and exit from nano : Ctrl o , Enter , Ctrl x
- or : Ctrl x , y , Enter

3. Select pacman mirror

nano /etc/pacman.d/mirrorlist

and uncomment one link like so :

Server = http://mirrors.kernel.org/archlinux/$repo/os/$arch

4. Update system

pacman -Syu

5. Install few usefull apps
- mc - midnightcommander - file manager working in shell
- xfce4 - light X Desktop Environment
- gnome-packagekit - package manager for X
- wpa_supplicant - for connecting to wireless networkes
- htop - manage runing processes

pacman -S mc xfce4 wpa_supplicant gnome-packagekit htop

After starting X

startxfce4

You can run graphical package manager with

Alt F2
gpk-application

6. Tweak few bash commands - for lazy ppl

mcedit /etc/bash.bashrc

and add this lines

alias ll="ls -al --color=yes"
alias grep="grep --color"
alias df="df -aTh"

Press F2 to save and F10 to exit mcedit
Try 'll' or other command after re-loging in to shell - some distros have this neat feature by default.

7. Picking boot runlevel 3

mcedit /etc/inittab

# Boot to console
id:3:initdefault:
# Boot to X11
#id:5:initdefault:

While editing inittab you can stop unnecessary terminals from starting at boot, I've rarely needed more than 2

#c3:2345:respawn:/sbin/agetty -8 -s 38400 tty3 linux
#c4:2345:respawn:/sbin/agetty -8 -s 38400 tty4 linux
#c5:2345:respawn:/sbin/agetty -8 -s 38400 tty5 linux
#c6:2345:respawn:/sbin/agetty -8 -s 38400 tty6 linux

8. If you have wifi router (or can connect to one), here's a helpfull hints

a) make a backup of config file

cp  /etc/wpa_supplicant.conf  /etc/wpa_supplicant.conf~

b) generate psk from ssid and passphrase

wpa_passphrase yourSSIDhere
# reading passphrase from stdin
TypeYourPassphraseHere

copy the result - psk - will be needed for config file

c) I've removed all network examples from my conf (starting from then line : # Example blocks: )

mcedit /etc/wpa_supplicant.conf

and whipped out something like this - for most secure setup I can pick for my WRT160N, which is WPA2 with AES

network={
 id_str="MY"
 disabled=0
 mode=0
 ssid="yourSSIDhere"
 scan_ssid=1
 proto=WPA2
 key_mgmt=WPA-PSK
 auth_alg=OPEN
 pairwise=CCMP
 group=CCMP
 psk=d123456789e4ba9b3b34c8085d39ee5d6e69ff625c6198b726b1e1acabcdef
 priority=9
}

d) and here's a script to start and stop wifi connection

mcedit /wifi.sh

Paste in this code

#!/bin/bash

if [ -e /var/run/dhcpcd-wlan0.pid ];then
    if [ -e /var/run/wpa-wlan0.pid ];then
        PIDF=`cat /var/run/wpa-wlan0.pid`
        kill ${PIDF}
    fi
    dhcpcd -x wlan0
    ifconfig wlan0 down
else
    ifconfig wlan0 up
    wpa_supplicant -q -B -Dwext -iwlan0 -P/var/run/wpa-wlan0.pid -c/etc/wpa_supplicant.conf
    dhcpcd -b wlan0
fi

press F2 to save and F10 to exit mcedit

chmod 700 /wifi.sh

Run by

/wifi.sh

Will start necessary apps and create pid files for them, if pids already exist - will stop those apps.

It's been a week and so far I'm loving Arch Linux. I get latest software & kernel and it's all running smoking fast.