Cron Update

Regular readers may remember that I've been trying to automate the checking of podcasts using Armangil's Podcatcher. Back in November, I wrote this post explaining my frustration. A comment from my riding buddy Steve gave me some good things to try, which helped me get the script going. Unfortunately, no progress was made on the cron front.

Here's what gets me. The command, or a script containing the command, works fine from the command line. The job takes several minutes just to check all of the feeds I watch, plus additional time for any downloads. I can see the ruby interpreter chewing up CPU in the top display. However, when I put either the command or script in cron, ruby runs just for a second.

I'm wondering if there is a file permission or visibility issue. Both seem to run under my user - not as root. Unfortunately, ruby doesn't tell me why it gives up so quickly.

I checked the syslog, but there are no entries from ruby - just the line from cron saying that it's starting the task. Is there another log where I can find more detail, or can I ask ruby to be more open with its issues?

In the meantime, I'll keep running the script from a command line every day or so.

Cron?

OK, I've done some research and tried a few things, but I can't seem to get my podcatcher to run automatically. The command is:

ruby /home/dgentry/podcatcher/bin/podcatcher /home/dgentry/podcast_subs.opml -D /export/podcasts -s 0 -S back_catalog -H 2007.10.01 -p -v -r 4 -d /home/dgentry/podcatcher -v

When I put this command into cron, it only runs the ruby command without any of the parameters. Also, I tried building a script file, but was not successful.

Anyone have any advice?

Armangil's Podcatcher - Part 5

(See part 4 here)

OK, so I've moved this program to the production server, but not after a little more testing on the play machine.

After looking into file security issues, I've decided to run this under my user, instead of using root. It's better for everyone. This makes all of the settings much easier. Here's the command to run the job:

ruby bin/podcatcher ../podcast_subs.opml -D /export/podcasts -s 0 -S back_catalog -H 2007.10.01 -p -v -r 4

It's running now. We'll see how it goes.

UPDATE: Went pretty well, except the existing directory was called "Podcasts", and I directed the output to "podcasts". In Windows, I could only see the new folder, and I thought that it had deleted all of my saved content. Turns out, Windows (via Samba) handles names that differ only by capitalization by only seeing one of them. I've changed the folder name to all lowercase, so this should be solved.

Armangil's Podcatcher - Part 4

(See part 3 here).

With everything tested, my next step was to setup a configuration similar to my production server.

I installed the Podcatcher package in the /opt directory. For the application files, I created /var/opt/podcatcher. The OPML file was stored in /etc/opt/podcatcher. And the podcast files were directed to /export/podcasts.

Here is the command used to run the program:

ruby /opt/podcatcher/bin/podcatcher
/etc/opt/podcatcher/podcast_subs.opml
-d /var/opt/podcatcher
-D /export/podcasts
-s 0
-S back_catalog
-H 2007.08.15
-p
-v

Seems to be working OK. The next step is to set it up to run twice a day unattended.

Armangil's Podcatcher - Part 3

(See part 2 here)

OK, so I think I've figured out why files are being deleted somewhat randomly. The SIZE option determines how much disk space will be used for the content cache, and the default is set to 512MB. With all of the testing I've been doing, I filled up that space pretty quickly. I can set it to zero to indicate no limit, or pick another size.

Juice, my windows based podcatcher, has a handy option that handles the deletion of old content on a per feed basis, where the cutoff is stated in days, not MB. I use this for podcasts I that only listen to occasionally. The latest version is always available, but I don't have to worry about piling up old episodes. Since the SIZE option looks at the entire cache to decide what to delete, I probably won't use if for this purpose.

So I've dropped the --no-empty option, and I'll have to add a SIZE option. Also, I'll probably use 'new' for the strategy rather than 'chron'. The difference seems to be which is downloaded first - old or new, so it doesn't really matter too much if I have enough space allotted.

My next experiments will be to try directing the cache to another folder, and using my actual OPML file in place of the demo.

More later...

UPDATED: I take it all back! The back_catalog strategy works better than new. In fact, I'm not sure of the difference between new and one.

Armangil's Podcatcher - Part 2

(See part 1 here)

I've been experimenting the past few nights with some options for the podcatcher.

-S chron

The strategy option controls the order of and amount of downloads. I used 'chron' to grab any new content

-H 2007.08.01

Horizon is used to set a cutoff date for the earliest download. I wanted to limit the number of downloads (and the time it would take) when working with long running or prolific podcasts

-p

Creates a subfolder for each feed

--no-empty

I've been having the problem of old content being deleted when new content is downloaded, but this option didn't really seem to help.

-v

The verbose mode let me see what the program was doing as it happened

In this snippet of the output, we can see files being deleted.

Deleting: /home/dgentry/podcatcher/cache/PodTech.net: Technology and Entertainment Video Network/Podtech_sameer_zook.mp3
Deleting: /home/dgentry/podcatcher/cache/Rocketboom RSS 2.0 WINDOWS WMV/rb_07_aug_14.wmv
Deleting: /home/dgentry/podcatcher/cache/Rocketboom RSS 2.0 WINDOWS WMV
Fetching: http://media.libsyn.com/media/seradio/seradio-episode65-introToEmbeddedSystems.mp3 (42257055 bytes)
#EXTINF:-1,[Software Engineering Radio] Episode 65: Introduction to Embedded Systems
/home/dgentry/podcatcher/cache/Software Engineering Radio/seradio-episode65-introToEmbeddedSystems.mp3
Fetching: http://www.rocketboom.net/video/rb_07_aug_15.wmv (1000 bytes)
#EXTINF:-1,[Rocketboom RSS 2.0 WINDOWS WMV] rb_07_aug_15
/home/dgentry/podcatcher/cache/Rocketboom RSS 2.0 WINDOWS WMV/rb_07_aug_15.wmv
Deleting: /home/dgentry/podcatcher/cache/Photos from everyone tagged green/1120072640_cac0ed247e_o.jpg
Deleting: /home/dgentry/podcatcher/cache/dawn and drew tv/dawnanddrewtv-73620-08-05-2007.mp4
Fetching: ...

More later...

Armangil's Podcatcher

I've been looking for a way to have my server download podcasts, rather than my desktop machine. The main reason is that I'm running Juice on Windows, and I don't have Windows running that often. I have to reboot into XP to grab the latest; then I switch back to Linux. Sure, I could run the Linux version of Juice/iPodder, but I hope to get this function off the desktop completely.

I downloaded version 3.0.0 here, and extracted it to a folder in my home folder - just to play. I also had to install Ruby on the machine.

The extract command is: tar zxvf podcatcher-3.0.0.tar.gz

To install Ruby: sudo apt-get install ruby

To give it a try, I ran it using the demo OPML file:

ruby bin/podcatcher demo/subscriptions.opml


The screen displays #EXT3MU, and then nothing else for a long while. Be patient!

(I should point out that I'm running this test on my play LAMP server, which is a 300MHz powerhouse with 196MB of RAM. It should be better on the production file server.)

After some time, the latest file in each feed was downloaded to the cache folder. Not bad.

There are a lot of parameters/switches available to fine tune the behavior. I can have it grab more than just the latest, get all since a date, direct to other folders, and much more. Looks like I have some experimentation ahead.