philwilson.org

Downloading from the BBC streaming iPlayer is hard

17 December, 2007

(this was mostly written on Friday 14th December, although no-one else seems to have stepped up to the plate yet)

The new Flash-based BBC iPlayer streams its content over Adobe’s proprietary RTMP, making it very hard to download the stream and save it for later.

There are some native, commercial, applications to do this on Windows like Replay Media Catcher but as far as I can tell, nothing on Linux. There has been a first-pass MythTV integration but this seems to use an embedded browser window.

snagged iplayer content

Red5 is a Java-based open-source Flash server which also comes bundled with some code for a sample client (download the tarball and then look in src/org/red5/samples/client). It might be possible to use this to download the stream; I haven’t yet tested. There also appear to be some code samples in Ruby and Python lying around, but these look very incomplete.

The URL for the stream is along the lines of this:

rtmp://217.243.192.52:1935/ondemand?_fcs_vhost=cp41752.edgefcs.net&auth=SECRET_KEY&aifp=v001&slist=STREAM_NAME

where both SECRET_KEY and STREAM_NAME come from a file of the name http://www.bbc.co.uk/mediaselector/3/stream/check/iplayer?pid=PID where PID is a BBC Programme ID such as b008h3zq as found on http://www.bbc.co.uk/programmes/ – the base URL for this address is found in the config.xml file linked to from every iPlayer page: http://www.bbc.co.uk/iplayer/emp/xml/config.xml

So this looks hackable, with the most promising chance probably being with Java at the moment.

See other posts tagged with bbc general iplayer real red5 and other posts made in December 2007.

Comments

Mark
18 December, 2007 at 12:11

OK I’m tempted to take the bait and have a look at doing this. I’d already got as far as working out the URL but was then kind of stumped for a way of downloading the rtmp stream (without using something like Orbit Downloader). Often you can just change the rtmp:// to http:// and the stream will work using the RTMPT protocol, but unfortunately this doesn’t seem to work.

Hopefully Red5 will provide the missing link, although from a quick glance at the source code I’m not entirely sure how to do it.

Java definitely seems the way to go though, as it would allow a signed applet to be added to the pages via greasemonkey which could provide a download button for the flv files.

Now I just have to work out how Red5 works!

Mark

Arnar Birgisson
28 December, 2007 at 15:10

Hey there. I’m somewhat affiliated with RTMPy and although you are correct in that RTMP implemenation is incomplete yet, there is active work being done on it. So far, we have alpha support for AMF through [PyAMF](http://pyamf.org/) which is being used as the basis for the RTMP implementation.

If you would be interesting in helping out with a client for downloading from BBC in the near future, that would be an intersting test for RTMP client functionality. Please stop by our mailing list if you feel so inclined 🙂

Iain
28 December, 2007 at 22:57

Is there any kind of forum or wiki where people are working on this? I see MythTV working on this, and I was linked from a thread on the XBox Media Center forums: http://www.xboxmediacenter.com/forum/showthread.php?t=27063

I’ve gotten as far as decompiling the flash player to see what it’s doing and deciding that it’s going to take considerably longer than a dull Saturday afternoon to hack something together.

How are you getting the URLs? I’ve found the stream traffic was encrypted.

Gnash Developer
02 January, 2008 at 04:40

Check the gnash-dev mailing list archives for recent progress on RTMP support. Join the list to help contribute to making this work with the BBC web site and streams.

Hubert
21 January, 2008 at 14:17

There’s another (free) app for capturing RTMP Flash streams called Orbit Downloader – http://www.orbitdownloader.com

Alas, it is also only available for Windows, but it does the job marvellously and you can’t fault the price.

lofty
27 July, 2008 at 06:37

I have been trying to do this as well, and got about as far as you did.

A few extra bits I found out:

The SECRET_KEY changes every time you request the mediaselector file.

The reply when you request the mediaselector file includes the header ‘Vary: Cookie’, which indicates to caching proxies that they should get a fresh copy every time the cookies change.

Brian
01 August, 2008 at 17:34

Sorry but none of the versions of the downloader work for me, on any platform. Resulting file is 0 bytes in size, finds MP4 file but dfails to download it. Given up trying to make it work.

Phil
01 August, 2008 at 18:27

Apparently there were some changes on the 30th July which stopped the downloader working for a while. It’s been updated now – get the latest from http://po-ru.com/projects/iplayer-downloader/

MattC
08 August, 2008 at 08:55

from a very quick casual look my understanding was that the iplayer-downloader only handled the RealAudio over RTSP streams (using mplayer), not the MP3 over RTMP streams?

Phil
08 August, 2008 at 09:04

Yes, I think you’re right although I do believe an RTMP version is coming.

Ding Jiang Zhu
30 September, 2008 at 08:21

Yes this is true, now there is MP3 support
see this clipping from the Authors Website:

Paul Battley Wrote at 2008-09-23 21:06 UTC (6 days ago)
using Firefox 3.0.1 on Linux:

I’ve just uploaded a new version with proper radio download support—which just meant detecting radio programmes and changing the extension to mp3!

That’s version 0.1.12 of the library and command-line tools, and version 1.8 of the GUI.

Hurra !

andersson
17 March, 2009 at 19:40

I had this problem today. Since i use mplayer in firefox, i just typed “ps lax | grep mplayer” in a shell and got this:

0 0 17665 17464 20 0 174336 8488 – Ss ? 0:00 mplayer -wid 0x220080a -vf scale=640:-3 -osdlevel 1 -nojoystick -noconsolecontrols -cookies -softvol -slave -user-agent NSPlayer -nomouseinput -cache 512 rtsp://rmv8.bbc.net.uk:554/radio4fmcoyopa/radio_4_fm_-_monday_2100.ra?timestamp=1237240768&pid=b00j3vn9&BBC-UID=64998765964d0de8f7947d6ad1740198c717267800d0205bdc6dc842b46f86d8&SSO2-UID=

Then just take the URL and download it with mplayer. Here is the result of the nice radio program
http://www.bbc.co.uk/iplayer/episode/b00j3w8k/Sound_Architecture_The_Spaces_That_Speak/
i got today:

http://andersson.is-a-geek.org/aac/

Video seems to be harder since you probably need to find a nice proxy in the UK.
If you succed – put the result on http://thepiratebay.org

Greetings from Uppsala, Sweden

Latest posts: