test proxy speed with bash and wget

I need to test the speed of some proxy server. Here’s a little script on how I achieved this.
I have a text-file ‘proxy.list’ which looks like: (I took out the last two digits of the ip).

...[lot's of ip's]...
193.196.*.*:3124       Germany
143.205.*.*:3127      Austria
64.161.*.*:3128         United States

Here is the script which will run through the list of all proxy and will download 5 test pages from a specific site. Then it will determine the period of time which is needed for execution. It’s create/append to a file ‘time.list’ which will contain the needed Information to determine the best proxies. You also need to create a subdirectory called ‘raw_proxy’ where the raw html code is save that you retrieve from the proxies. The files are named ”raw_proxy/$ip.$port.$i.tmp’ where $i is the i.th test page you downloaded. I need to keep those files to determine if the proxy send me the right file or e.g. a login page .

size=$(cat proxy.list | wc -l)
while read proxy
    #determine the first parameter (IP:Port)
    ad=$(echo $proxy | awk '{print $1}')
    ip=${ad%:*}   #extract ip
    port=${ad#*:} #extract port
    #set and export the proxy settings
    http_proxy=$ip:$port && HTTP_PROXY=$http_proxy && export http_proxy HTTP_PROXY
    #save start timestamp
    start=$(date +%s)
    #download 5 pages #(yes I know 'seq' but I'm on Mac and I needed a sth. quick&dirty)
    for i in $(echo "1 2 3 4 5")
        #use wget to retrive the page. We want to try 1 time and set some specific timeouts. + we force to use a Mozilla User agent to hide that we are using wget.
    	wget -O "raw_proxy/$ip.$port.$i.tmp" --tries=1 --dns-timeout=10 --connect-timeout=8 --read-timeout=15 -U "Mozilla/5.0 (compatible; Konqueror/3.2; Linux)" "http://www.yourTestPage.com/$i.txt" &> /dev/null
    #save end timestamp
    end=$(date +%s)
    #calculate the difference
    diff=$(( end - start ))
    #append this info to time.list
    echo -e "$ip:$port\t$diff" >> time.list
    #to have a nice and shiny output I use figlet, this is optional, if you don't want it comment out next 3 lines or just remove ' | figlet'
    echo "PC: #"$size" - "$diff"s" | figlet
    sleep 1
    size=$(( size-1 ))
done < proxy.list

If you used figlet your output looks like this:

 ____   ____       _  _    __  _____ _           ____   ___      
|  _ \ / ___|_   _| || |_ / /_|___ // |         |___ \ / _ \ ___ 
| |_) | |   (_) |_  ..  _| '_ \ |_ \| |  _____    __) | | | / __|
|  __/| |___ _  |_      _| (_) |__) | | |_____|  / __/| |_| \__ \
|_|    \____(_)   |_||_|  \___/____/|_|         |_____|\___/|___/

It shows how much proxies need to be checked and shows the last execution time.

After the script has finished you need to get a list of which proxy was best.
This is the command line which evaluates everything and gives me back a list of ip's sorted by access time. It also removes all proxies where the downloaded page had a size of 0B.

#command line to list proxy with lowest time to download
clear && while read line; do ip=${line% *};time=$(echo $line | awk '{print $2}');ip=${ip%:*};echo -e $ip"\t"$time"\t"$(ls -alshr raw_proxy/ | grep 1.tmp | grep $ip | awk '{print $6}'); done < <(tr -s ' ' < time.list | sort -n -r -k2 | cut -d' ' -f3) | grep -v "0B"

This is the output:

201.22.*.*	43	52K
196.213.*.*	43	13K
147.102.*.*	1	2,1K
132.227.*.*	1	2,1K
130.75.*.*	1	52K

If you know the filesize the you can append a

 | grep "52K"

to the last command to show only files which have the right size.
This is it 😉

I know that out there are better and fast implementations to do this but ...
but it was fun

OS X – AppleScript to open Apps Hidden

I need to open Safari with a specific URL on command line. The problem is that this has to happen completly in background and should be started through an existing bashscript.
Here is my solutiuon, it should apply to any other app in OS X SnowLeopard. You need to call the bash schript like this:

./open_safari_with_url.sh "http://www.site_i_want_to_visit.com"

“open_safari_with_url.sh” looks like this:

osascript -e "
tell application \"Safari\" 
	tell application \"Finder\"
		set visible of process \"Safari\" to false
	end tell
	set the URL of document 1 to \"$1\"
end tell"

[EDIT] It seems that ‘launch’ in AppleScripts starts the app in background. So we don’t need to tell Finder to hide Safari anymore.

tell application "Safari" 
	set the URL of document 1 to "http://whatever..."
end tell"

awk and word frequencys

I have a list that I use for making gnuplots. The structure is the following

wordId wordWeight

e.g. 101 34.342 = Word with id 101 has a calculated weight of 34.342 times.

The bad thing is that this list is unordered. Now I wan’t to order this list getting the greatest weight and their corresponding word id. Bash doens’t seem to be the best solution for this so I made up my first awk script.

Here is it, it prints the most common word from file toPlot.stats.

awk ‘ BEGIN{ max=0; w=-1 } { if ( $2 >= max) { max=$2; w=$1 } } END{ print “id”,w,”has  most often with count of”, max; }’ toPlot.stats

and the output:

id 1545 has greates weight with  28199.40186090438

My file has 1.722.913 lines, execution time:

real    0m1.618s
user    0m1.576s
sys     0m0.032s

open terminal here

Marc Liyanage wrote a nice Apple Script which easily allows you to switch from the Finder to the Terminal.
Just download it here and drag the script to the upper free space in the Finder.

Clicking it results in a new Terminal window which location is the same as the Finder location.
new Finder icon

Here’s how it looks.

P.S. this works under OS X 10.6 😉

first look at javafx

Today I had my first look at javafx. Real nice but you must get used to the syntax.
After 1/2 hour of reading and 1/2 hour of modifying a nice netbeans example (original version from Michal Skvor) i end up with this. Not perfect but it has shown me how the javafx basics work.

winner screen


And after I read this ‘Deploy A Rich Internet Application Developed With JavaFX Technology‘ (another 10 minutes) i was able to deploy it to the web.

Start it

If you like it let me know

You can cheat by pressing ‘r’=remove single block. ‘s’=remove more 😉

Source Code can be foundhere

sort a hashmap, fast

Imagine you have an hash map where key is an String and value is a integer.
Now you want to sort based on the value of each entry in this hashmap in descanding order.

Here is how I do it:

List<Map.Entry<String, Integer>> list = new Vector<Map.Entry<String, Integer>>(myHashMap.entrySet());
java.util.Collections.sort(list, new Comparator<Map.Entry<String, Integer>>() {
      public int compare(Map.Entry<String, Integer> e, Map.Entry<String, Integer> e1) {
          return (e.getValue().equals(e1.getValue()) ? 0 : (e.getValue() > e1.getValue() ? 1 : -1));

Now the list contains the sorted entrys from hashmap ‘THashMap myHashMap<String,Integer>’.
Iterate over it or copy it back to ‘myHashMap’ if you need to.

Hope this helps …