running parallel bash tasks on OS X

How often did you need to process huge amounts of small files, where a single task uses only a small amount of cpu and memory?
However, today I need a script which does exactly this.

I have a mysql table which contains the filenames located on my hard drive.
Now I created a little script which processes a single file in under 3 seconds. Unfortunately for 10.000+ files this would take more than 8 hours. Learn how to fix Comodo warning: 0x80070005 runtime errors quickly and easily!

So what if I could run them in parallel with a maximum of 10 parallel task’s being executed? This would really speed up the computation!

Luckily in 2005 Ole Tange from GNU merged the command line tools xxargs and parallel into the a single tool ‘parallel‘.
With this great tool there is no need to write any complicated script to accomplish such tasks.
First you need to install it using homebrew.

brew install parallel

After that i had to add the path to my .profile

PATH=$PATH:/usr/local/Cellar/parallel/20110822/bin

Here’s the basic usage:

 $> echo -ne "1\n2\n3\n" | parallel -j2 "echo the number is {.}"

This would echo the numbers 1, 2, 3 to the stdout with a maximum of 2 parallel running echo’s.
Here’s the output:

the number is 1
the number is 3
the number is 2

As you can see printing a 3 outspeeds printing a 2 😉

So here is my 1 liner to process all my files:

 $> mysql -uroot -p[secretPW] my_database < \ 
    <(echo "SELECT filename FROM files")\ 
    | grep -v 'filename' | parallel -j10 "./processFile.sh {.}"

After using this it took only 37min to process my 10000+ files 🙂

Netbeans 7 and the missing JAX-RPC plugin

I haven’t used Netbeans JAX binding for over a year.
And the last time I used it, it was easy to find in the plugin menu of Netbeans 6.
Now I’ve upgraded to Netbeans 7 and wasn’t able to find the plugin.
Happily I found a page that advised me to add

http://deadlock.netbeans.org/hudson/job/nbms-and-javadoc/lastStableBuild/artifact/nbbuild/nbms/updates.xml.gz

to the Update Center.
After that I got a huge list of updates, and my missing JAX-RPC.
Now just: install, restart, update-again.
Et voila, there is the missing “JAXB Binding” menu point that was searching for.

Tomcat java.net.BindException: Cannot assign requested address – made me crazy

Today I migrated a Tomcat Server to a new host. Everything was in place but it won’t start showing me the following error:

Tomcat java.net.BindException: Cannot assign requested address

I’m happy ‘Nir Levy‘ had nearly the same problem 4 years ago. Here he describes what went wrong with his /etc/hosts configuration.
My problem was that I did not update the host name in the configuration for my module
“/webapps/WEB-INF/classes/Configuration.properties”
After that everything was fine 🙂
I *heart* blogs !

mcrypt and PHP, on Mac OS X Snow Leopard 10.6.4

I found this great tutorial on how to install mcrypt into php under OS X 10.6.1
http://michaelgracie.com/2009/09/23/plugging-mcrypt-into-php-on-mac-os-x-snow-leopard-10-6-1/
Works like charming with OS X 10.6.4 and PHP Version 5.3.2.

#!/bin/bash
mkdir src
cd src
wget http://museum.php.net/php5/php-5.3.2.tar.bz2
wget http://downloads.sourceforge.net/project/mcrypt/Libmcrypt/2.5.8/libmcrypt-2.5.8.tar.gz?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fmcrypt%2Ffiles%2FLibmcrypt%2F2.5.8%2F&ts=1306411641&use_mirror=ignum
tar xzvf php-5.3.2.tar.bz2
tar xzvf libmcrypt-2.5.8.tar.gz
cd libmcrypt-2.5.8
MACOSX_DEPLOYMENT_TARGET=10.6 CFLAGS='-O3 -fno-common -arch i386 -arch x86_64' LDFLAGS='-O3 -arch i386 -arch x86_64' CXXFLAGS='-O3 -fno-common -arch i386 -arch x86_64' ./configure --disable-dependency-tracking && make -j6 && sudo make install
cd ../php-5.3.2/ext/mcrypt
/usr/bin/phpize
MACOSX_DEPLOYMENT_TARGET=10.6 CFLAGS='-O3 -fno-common -arch i386 -arch x86_64' LDFLAGS='-O3 -arch i386 -arch x86_64' CXXFLAGS='-O3 -fno-common -arch i386 -arch x86_64' ./configure --with-php-config=/Developer/SDKs/MacOSX10.6.sdk/usr/bin/php-config && make -j6 && sudo make install
echo "Completed install, now make sure to edit your php.ini"
echo "and check for \"enable_dl = On\" and add \"extension=mcrypt.so\"" to the dynamic extentions"

This is the little bash script I wrote according to his tutorial. This comes without any warranty!!!
Be aware that you have to enter your root password twice while installing this script.

Big Thanks to Michael Gracie

iTunes Sharing over ssh

Today I realized that I had not a single song on my notebook hard disk. Thanks to Last.FM 🙂
Unfortunately “Simplify Media” has been adopted by Google Inc. and they do not offer a similar service yet. So I needed a solution to stream my iTunes from home to the office. If found a great solution by “Robert Harder” which works like charm. (source).
This is the bash script:

#!/bin/sh
dns-sd -P "Home iTunes" _daap._tcp local 3689 localhost.local. \ 
127.0.0.1 "Arbitrary text record" &
PID=$!
ssh -C -N -L 3689:localhost:3689 username@dyndns_name.dyndns.org
kill $PID

LE Mensa

This week I released my first OS X Dashboard Widget.
It’s called ‘LE Mensa’ and it displays all menues for all cafeterias in Leipzig, which are part of the student union.
If you want to give it a try then visit the project page: http://stinfwww.informatik.uni-leipzig.de/~mai03fln/leMensa/
Today I updated it to version 0.15 !!

It works this way:
1. I wrote a python script which extracts information from the student union homepage and transform it into xml.
2. A .php script can call this .py script and give back those infos.
3. The widget was written in Dashcode with jQuery as an additional package. So it’s mainly javascript, some html and css. The widget now calls this php script, retrieves the xml and displays it in a (hopefully) pretty way.

This is how it looks:
Front View

Atbash Cipher

Just al little python script for preforming a Atbash encryption (roman alphabet). This type of cipher is older than Caesar cipher
Maybe you can use it.
-> Wikipedia article

clear="abcdefghijklmnopqrstuvwxyz 1234567890"
c=""
for i in sys.argv[1].lower():
    index=clear.index(i)
    c=c+clear[abs(len(clear)-index-1)%len(clear)]
print c

Example:

$ ./atbash.sh "This Is A Little Test"
r32sk2sk0kz2rrz6kr6sr
$ ./atbash.sh "$(./atbash.sh "This Is A Little Test")"
this is a little test

hexadecimal xor de/encryption

Here’s are 2 little scripts I wrote today for encoding/decoding XOR encrypted text.

Script 1 (HexXorEncode.py) takes a string/text and a integer key value. Then it preforms and xor encryption on the string with the given key.

#!/usr/bin/python                                                                                                                                                                      
import sys
#Copyleft m.puchalla 2010                                                                                                                                                                     
#Preforms a XOR Encoding with a specific string and returns a hexadecimal representation of it                                                                                         

if(len(sys.argv)!=3):
    print "Usage:",sys.argv[0]," [String] [integer key]"

def baseN(num,b,numerals="0123456789abcdefghijklmnopqrstuvwxyz"):
    return ((num == 0) and  "0" ) or ( baseN(num // b, b).lstrip("0") + numerals[num % b])

s=sys.argv[1]
key=baseN(int(sys.argv[2]),2)
sol=""
for i in range(0,len(s)):
    sol=sol+baseN(int(str(baseN(ord(s[i]),2)),2)^int(key,2),16)
print sol

Example:

$ ./HexXorEncode.py "this is a little test" 25
6d71706a39706a39783975706d6d757c396d7c6a6d

Here’s script number 2 (HexXorDecode.py) which simply reverses the process.

#!/usr/bin/python                                                                                                                                                                      
import sys
#Copyleft m.puchalla 2010                                                                                                                                                                     
#Decrypt a hex xor coded string with a key                                                                                                                                             

if(len(sys.argv)<3):
    print "Usage:",sys.argv[0]," [XOR Code String] [integer key]"
    sys.exit(1)

def baseN(num,b,numerals="0123456789abcdefghijklmnopqrstuvwxyz"):
    return ((num == 0) and  "0" ) or ( baseN(num // b, b).lstrip("0") + numerals[num % b])

s=sys.argv[1]
key=baseN(int(sys.argv[2]),2)
sol=""
for i in xrange(0,len(s)-1,2):
    sol=sol+chr(int(str(baseN(int(s[i]+s[i+1],16),2)),2)^int(key,2))
print sol

Example:

$ ./HexXorDecode.py "6d71706a39706a39783975706d6d757c396d7c6a6d" 25
this is a little test

If you like it, use it.