Automating connecting with ssh to a series of hosts

Posted on 2008-12-02 by ken_fallon

I sometimes need to a large number of servers one after the other to run a command. For example if we want to see how full the home partition is on all our servers. The command we want to run is the disk free command in human readable format, df -h /home. The first step is to have a text file with a list of servers that you want to connect to.

cat servers.txt
server1
server2
server3
...

Then create a loop to cycle through the list and connect to each in turn and run  the command you want.

for i in `cat servers.txt`;do 
  echo $i;ssh $i "df -h /home"
done

Sometimes when one of your servers is down or disconnected there is a long timeout in establishing the connection ass ssh waits for the tcp connection to fail. You can get around that by using the ConnectTimeout ssh option. This usually run this type of command all on one line.

for i in `cat servers.txt`;do echo $i;ssh -o ConnectTimeout=3 $i "df -h /home";done

Keep in mind that this will connect to the servers in sequence so another approach might be Parallel ssh http://www.theether.org/pssh/ mentioned on David O’Yeats’s podcast http://www.lottalinuxlinks.com, although that is probably not going to be installed by default.

Posted in Podcasts | Leave a comment

Initial ssh connections slow – Is GSSAPI to blame ?

Posted on 2008-12-02 by ken_fallon

I’ve had several occasions in the past where the initial ssh connection has been slow to setup. Once you’re logged in then the connection is fast. I’ve always looked to dns resolving issues for these type or problems. Either disabling UseDNS in the sshd.conf file or adding the line GSSAPIAuthentication no to your ~/.ssh/config file.

[Edit]Excellent article here explaining the issue http://injustfiveminutes.com/2013/03/13/fixing-ssh-login-long-delay/

Posted in General | Leave a comment

HPR episode on using a squid proxy server locally

Posted on 2008-11-14 by ken_fallon

This month my HPR episode featured using a local squid proxy. You might want to to run your own proxy server to provide yourself with a secure web connection when you are out and about by tunneling your traffic over ssh. Another good reason is to find out which urls your browser is going to. On some sites url’s are deliberatly hidden or you may be interested in exactly where you are sending traffic to. All is explained.

Posted in Podcasts | Tagged , , | Leave a comment

Converting an entire spreadsheet to text files.

Posted on 2008-11-06 by ken_fallon

I have often the need to convert spreadsheets to text. The file -> save as trick is fine but will only save one sheet at a time. Often I get a spreadsheet with multiple workbooks and need to export the entire thing. For that I use xls2tab which is a perl program that cycles through the sheets and dumps them to comma seperated text files.

You’ll need to install a perl module but that is easy enough to do using cpan.

$ sudo perl -MCPAN -e shell
Terminal does not support AddHistory.

cpan shell -- CPAN exploration and modules installation (v1.7602)
ReadLine support available (try 'install Bundle::CPAN')

cpan>  install Spreadsheet::ParseExcel::Simple

It may ask you a load of questions if this is the first time you ran cpan but the defaults have been sensible enough for me in the past. Once you’re finished type quit to exit. You can then run the program using the command

$ xls2tsv.pl test.xls

After that you’ll have a whole directory full of tab files.

Posted in General | Leave a comment

Taking screenshots of your home movies

Posted on 2008-10-24 by ken_fallon

So by now you are finished Archiving your DV tapes and also finished converting your DVDs to mp4 (h264) . So now you have a big hard disk full of movies and you have no idea what’s in what video.

I would never leave you in such a state so today we will be generating screen shots of the movies so you can browse the images to see what the recordings are about. I got most of the work from Screenshots of a DVD with ‘ffmpeg’ from Peter Forret’s blogg. Then a quick jump to the ffmpeg FAQ How do I encode movie to single pictures? and we were done.

The script will search for all avi files and it uses ffmpeg (our friend) to take a snapshot every 20 seconds.

for i in *.avi;do
  ffmpeg -i ${i} \r
  -r .05 \r
  -s 360x288 \r
  -an \r
  -f image2 \r
  -vcodec mjpeg \r
  ${i}_%04d.jpg;
done

-r .05 (Frames per seconds so this is every 20 secs)
-s Sets the frame size.
-an Disables the audio recording
-f image2 Forces an image format
-vcodec  Force video codec to mjpeg

The files will be named the same as the parent movie but will have a number tagged on in the range 0001 -> 9999 then .jpg. The option %04d is used to do the numbering.

Posted in General | Leave a comment

HPR: thisrunslinux.org

Posted on 2008-10-15 by ken_fallon

My HPR episode for October was released today. This one’s all about 3:30’s idea to get video material to promote linux. It’s still in very very soft launch beta so if you want to offer help or suggestions please have a look over at http://www.thisrunslinux.org. . . . . . . .And in other news. The chat with Dave also came out today. I’ve spoken before about how surprising it is to hear yourself speaking in your own podcast list and this was even worse. That said it was great to talk to Dave and had the battery on his phone not run out, I imagine I’d have kept talking for the rest of his drive home.

Posted in Podcasts | Leave a comment

I spoke to Dave !!!

Posted on 2008-10-12 by ken_fallon

In episode 91 Dave Sexy Yeats announced his first ever call-in show. Fortunately this was at a time that I should be awake I thought it would be too good an opertunity to miss. For some reason my land line wouldn’t call grand central so I had to resort to using skype out. I got through and had about an hour of chat with Dave. I’m not sure how interesting the conversation was for Dave or anyone else for that matter but I realy enjoyed chatting with him. In other news I’ve started a new job and I’ve recorded a HPR episode but I still need to edit it.

Posted in Podcasts | Tagged | Leave a comment

faster: a bashpodder plugin

Posted on 2008-09-22 by ken_fallon

I’ve been asked to supply the script that I use to speed up podcasts. So here it is:

#!/bin/bash
# v0.3 20080919 https://kenfallon.com
if [ -d "${*}" ]; then
  cd "${*}"
  ls -1 2>/dev/null | while read FILE;do
    FILENAME=${FILE##*/}
    sox "${FILENAME}" "${FILENAME}-faster.ogg" -V9 tempo 1.5
    if [ "$?" != "0" ]; then
      faad -o - "${FILENAME}" | lame - - | sox -t mp3 - "${FILENAME}"-faster.ogg -V9 tempo 1.5
    fi
    rm "${FILENAME}"
  done
fi

The script will change to a directory if one is given and then will try to speed up all the files in that directory. If the sox program fails to process the file then the faad program will try to convert it to mp3 using lame. This will then be piped to sox again to speed it up and convert it to an ogg file.

Yes I know this is not fantastic and it will also try and speed up text files or anything else that is in the directory. However this is supposed to be used as a plugin to bashpodder. It seems everything has a plugin architecture so why should bashpodder be left out. The plugin aspect is achieved by adding this line to the end of the bashpodder script.

if [ "`basename ${0}`" = "bp" ];then fast ${datadir};fi

The idea is that you can still run bashpodder normally you just call bashpodder.shell as normal. To use the script you can call it with bp. That is nothing more than a symbolic linc to bashpodder.shell

ln -s bashpodder.shell bp
Posted in Podcasts | Tagged , , , , | 2 Comments

HPR Episode 3 tips posted.

Posted on 2008-09-16 by ken_fallon

Another of my episodes just appeared on Hacker Public Radio. This time some tips that I use very often.
Tip 1: Repeating the same command using an infinite loop and a wait timer

while [ "x" = "x" ]; do ls -al ; sleep 5; done

If you find yourself repeating the same command over and over again then this tip may help. The example I gave in the episode was watching a file grow during a FTP transfer but this morning I used the same idea but did an ifconfig while I was waiting for a UMTS connection to establish.

About five minutes after posting the episode I noticed discovered that logic could be improved using while true instead. As in:

while true; do ls -al ; sleep 5; done

Tip 2: Speeding up podcasts
As Linc said on the Linux Linc Tech Show, there are a lot of podcasts out there and it’s difficult to get to them all. So this tip allows you to speed up the tempo of the podcast without affecting the pitch. So to make a long story short the podcasts are spead up without everyone sounding like Chipmunks. It takes a while to adjust to it but for spoken word it’s fine; for music it’s not so good.

sox in.mp3 out.ogg tempo 1.5

I’ve been using this tip since 2008-04-28. The reason I know this because that’s when I filed a bug with with Ubuntu beacuse their version of sox can convert from mp3 but not to mp3. It turns out this is also a problem with Debian. Unfortunately the ability to fix this is outside my skill set. My solution was just to convert everything to ogg and this works for me and my Samsung YP-U2.

Tip 3: Making an identical copy of a directory

Very often I need to copy a directory from one location to another but want to preserve things permissions, attributes, weird file names etc. The best tool for this job it tar. Rather than making a tar file, copying it and then extracting it as three separate steps, you can combine all the steps into one using.

tar -cf - . | ( cd /media/backupdisk; tar -xvf - )
Posted in Podcasts | Tagged , , , , , | 4 Comments

DVD to mp4 (h264) conversion

Posted on 2008-09-09 by ken_fallon

Following on on from what’s becoming a mini series on data archiving from the Linux command line, I want to talk about backing up DVD’s via the command line. I’m not going to insult your intelligence with the lecture about copying copyrighted material so let’s get to it.

The command to use is dvdbackup and from the package description “Description: tool to rip DVD’s from the command line.  dvdbackup will extract all (or optionally only selected) titles as found on the dvd.  It will structure the extracted files in format suitable for burning at a later time with genisoimage and dvdrecord.  Has the advantage of being very easy to use, small, and fast.”. I just use it to dump the entire DVD onto the hard disk.

dvdbackup -Mv

From there you can encode the movie to MP4 format. I looked into a lot of encoding options and spent a lot of time trying to tweak the best options for video size versus quality. In the end I’ve found that the quality of the video produced following the ffmpeg settings on the gentoo forum are superb.

cat VTS_01_1.VOB  VTS_01_2.VOB  VTS_01_3.VOB | \r
ffmpeg \r
-map 0:0 -map 0:1 \r
-i - \r
-threads 2 \r
-b 604k \r
-vcodec h264 \r
-ac 1 \r
-ab 256k \r
-ar 44100 \r
-acodec mpeg4aac \r
-coder 1 \r
-flags +loop \r
-cmp +chroma \r
-partitions +parti4x4+partp8x8+partb8x8 \r
-me hex \r
-subq 5 \r
-me_range 16 \r
-g 250 \r
-keyint_min 25 \r
-sc_threshold 40 \r
-i_qfactor 0.71 \r
-vol 500 \r
my-wedding-video.mp4

I did get an error about “Codec type mismatch” but following a post on the YouMakeMedia blog I changed the settings to:

cat VTS_01_1.VOB  VTS_01_2.VOB  VTS_01_3.VOB | \r
ffmpeg -i - \r
-threads 2 \r
-b 604k \r
-vcodec h264 \r
-ac 1 \r
-ab 256k \r
-ar 44100 \r
-acodec mpeg4aac \r
-coder 1 \r
-flags +loop \r
-cmp +chroma \r
-partitions +parti4x4+partp8x8+partb8x8 \r
-me hex \r
-subq 5 \r
-me_range 16 \r
-g 250 \r
-keyint_min 25 \r
-sc_threshold 40 \r
-i_qfactor 0.71 \r
-vol 500
my-wedding-video.mp4
Posted in General | Tagged | 1 Comment