philwilson.org

How to export Firefox's history to a text file

04 January, 2005

Note that this doesn’t currently work in Firefox 1.5

Specifically, serialised as RDF/XML.

Early last year, Jamie Zawinski asked about how to extract information from Firefox’s history.dat file and ended up having to write a Perl script to do it.

He actually got further than most – most people who try to access the data not via Firefox itself just give up. This is because Firefox’s history is stored in a Mork file, which is an incredibly over-complicated and under-documented proprietary file format. You’ll learn more about it by reading the comments on Jamie’s post that you would by reading the docs.

Mork is used as a file format all over Mozilla, in both Firefox and Thunderbird in various points, but there are plans to replace it with SQLite in Mozilla 2.0 (see Unified Storage on the official Mozilla wiki) possibly with some interfaces made public via RDF (but I’m not 100% sure on this).

As it stands, there’s an open bug asking for history.dat to be easier to parse (which will probably be fixed by the move to SQLite rather than anything else) which actually has a very useful piece of Javascript attached which will convert the Mork history.dat into an RDF/XML history file, which I personally use as a bookmarklet from my toolbar: [export history.dat](<javascript:(function(){netscape.security.PrivilegeManager.enablePrivilege(%22UniversalXPConnect%22);var rdfhis =Components.classes[%22@mozilla.org/rdf/rdf-service;1%22].getService(Components.interfaces.nsIRDFService).GetDataSource(%22rdf:history%22);var serializer=Components.classes[%22@mozilla.org/rdf/xml-serializer;1%22] .createInstance(Components.interfaces.nsIRDFXMLSerializer);serializer.init(rdfhis);var outputStream = { data: %22%22, close : function(){}, flush : function(){},write: function (buffer,count){this.data += buffer; return count; }, writeFrom : function (stream,count){}, isNonBlocking: false};serializer.QueryInterface(Components.interfaces.nsIRDFXMLSource).Serialize(outputStream);outputStream.close();var summ=outputStream.data;var fp =Components.classes[%22@mozilla.org/filepicker;1%22].createInstance(Components.interfaces.nsIFilePicker);fp.init(window,%22%22,Components.interfaces.nsIFilePicker.modeSave);fp.defaultString = %22history.rdf.txt%22;fp.show();var filoutputStream = Components.classes[%22@mozilla.org/network/file-output-stream;1%22].createInstance( Components.interfaces.nsIFileOutputStream );filoutputStream.init( fp.file, 0x04 | 0x08, 420, 0 );filoutputStream.write(summ, summ.length);filoutputStream.close();})() >). Just drag it to your toolbar, load a local file in Firefox to ensure you’re in a trusted context and click it like you would any other bookmarklet. It’ll give you a security warning, which you can click “yes” to (if you trust my link – if not, you can get it from the bug linked to above, or look at the code yourself). It’ll take a few seconds to kick in (or longer, depending on the size of your history) but will eventually ask you to specify a location to save the file to. Give it a name, and you’re free to parse it with whatever you like (although preferably an actual RDF parser 🙂

See other posts tagged with general and all posts made in January 2005.

Comments

Acting True
04 March, 2005 at 20:24

Hi, I’d like more information about this — I went to the link and it wasn’t clear or wasn’t there.

Pip
04 March, 2005 at 23:49

If you followed the link to bugzilla (i.e. ) then yes, it’s all very unclear.

What exactly would you like more information about?

Anonymous
17 May, 2005 at 06:52

I tried to open the export history.dat-link but it doesn’t work.

i get the following error in the firefox javascript-console:

Fehler: uncaught exception: Einem Skript von “http://philwilson.org” wurden UniversalXPConnect Sonderrechte verweigert.

in english this stands for, that firefox denied the priviliges to run this script.

anyone has an idea how to get this one running?

greetings

michael

Pip
17 May, 2005 at 07:07

Hi Michael.

Type “about:config” in your Firefox address bar and type “xpconnect” into the text box marked “filter” and check that the preference “security.xpconnect.plugin.unrestricted” is marked as “true”. I *think* that’s the problem you’re having.

Anonymous
12 June, 2005 at 13:56

I’m also having trouble with export history.dat-link on this page. The Java Console reports, “Error: uncaught exception: A script from “http://philwilson.org” was denied UniversalXPConnect privileges.”

security.xpconnect.plugin.unrestricted is true in about:config.

Any other ideas?

Pip
12 June, 2005 at 14:05

Yes – you need to viewing a local HTML page in Firefox before you click the bookmarklet.

Just view any webpage, save it to disk (or create a new page on your hard drive), load it in Firefox then click the link. Loading the local page puts Firefox into a lower security mode which lets the bookmarklet access the files on the hard drive.

GJ
07 July, 2005 at 12:00

Is it not possible to export firefox history when the URL is remote (not a local file) ? If yes, then what do I have to do extra?

I would like to export the history irrespective of the URL (local, remote or about:blank)

Please do help.

Regards,
Gautam Jain

Pip
07 July, 2005 at 13:09

I don’t think it’s possible to do it unless you’re in the local context because of security constraints. Obviously being able to dump a file to hard drive, named as you want it, via JavaScript, would be a massive hole.

Jimmy Carlyle
01 September, 2005 at 13:28

Hi

I’ve spent some of today looking for something like this and it worked great. Only trouble is how can I import the file onto another FireFox installation.

The user got a new PC is the reason.

Cheers

Jimmy

Pip
01 September, 2005 at 13:38

If you need to move the Firefox history between machines, you should just be able to copy the history.dat file from the profile directory:

* close Firefox on both computers
* find the profile directory on each computer (on Windows this will be something like C:Documents and SettingsUsernameApplication DataMozillaFirefoxProfilessdfkjsh4.default)
* copy the history.dat from the source profile directory to the destination profile directory

I believe that should work just fine.

wom
03 September, 2005 at 14:50

Hi, for a university project I need a way to track user surf paths. The history.dat as RDF is a already a big help. But it is incomplete in the way that it does not recognize back button actions and does not distiguish beetween link following and typing in an URL in the adress field. Is there a way to enrich the history with that data?

Pip
03 September, 2005 at 15:44

Hi wom. I seem to remember that the Mozilla suite’s history.dat is far richer in data than Firefox’s and does at least some (if not all of) the things you’re asking about.

Anonymous
10 November, 2005 at 22:10

WOM: Try writing a firefox extension.

Anonymous
13 December, 2005 at 12:55

hello Phil,

a few months ago i told you that i have problems to use the javascript. because of much work i had no time to look at your blog earlier. your advice to solve my problem did work very nice and now i have a full functional xml file.

thx for this phil.

now i am working on a script to convert the xml file to a clear listing with the visited sites.

i will finish this script until end of the week probably and post a link to my website where everyone is allowed use it.

would be nice if some of you guys would test my script.

greetings from germany

Anonymous
13 December, 2005 at 12:57

the prior comment was written by me (Michael) 🙂

Pip
13 December, 2005 at 14:14

Excellent news, Michael, I’m glad I could be of use. I look forward to seeing your script!

Anonymous
13 December, 2005 at 16:43

Writing the script was more easier than i had imagined.

i made a little webpage for this script. would be interesting if it works with other history files than mine.

in the future i will optimize the script and perhaps add some enhancements.

you can find the script here: http://www.universe-hero.de/kortas_net/projects/ffhistory/index.php

for comments, reportings bugs or some special ideas feel free to write me an email: m(.at.)kortas(.dot.)net

greetings

Michael

sari gordon
22 December, 2005 at 17:39

Wow, I’m a total dunce when it comes to any of the abbreviations you’re talking about, but I managed to export a list of the sites I’ve visited today. My plan is to blog my travels in an attempt to illustrate my goal of reading the entire web (). If you could develop this into an extension, I would be a happy blogger! Thanks for your efforts!

Pip
22 December, 2005 at 18:58

wow, I’m glad I could help! This stuff doesn’t work in the latest version of Firefox, so I’ll have to do something better anyway!

Anonymous
25 January, 2006 at 19:39

Phil, please help! I accessed your file about moving Firefox’s history to text, and it worked great. The only problem is, the history.dat file shows a whole bunch of files that the history.rtf.txt does not (the history.rtf.txt only shows the last nine days). I want to find out the dates accessed for all of these files in history.dat. Is there any way for me to do that?

Also, if a user goes into Firefox and goes to the history, and deletes a specific site (just right-clicks and hits delete, like that), will that site still show up in history.dat, or in the text version?

Please help – I’m really desperate to figure this out. Thanks in advance! I’m at ubergottin @ herzeleid . net if you want to email me directly, or post here. (I tried to follow the link on the “contact you” part of this site, but it didn’t work for me.)

Lindsay

Anonymous
03 February, 2006 at 05:46

Phil, Thank you so much for doing this! It worked great – until I updated to ver 1.5.

Hope to see your new version soon. I wish I could find an old version of FireFox to install so I could continue to use your bookmarklet for now.

Sival

leo4must
06 February, 2006 at 20:32

The export history.dat is no longer working with FF 1.5.0.1 on Windows XP Pro SP2. It worked fine earlier (FF versions on WinXP SP2), but now Javascript console says:
uncaught exception: [Exception… “Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIRDFXMLSource.Serialize]” nsresult: “0x80004005 (NS_ERROR_FAILURE)” location: “JS frame :: javascript:(function(){netscape.security.PrivilegeManager.enablePrivilege(%22UniversalXPConnect%22);var rdfhis =Componens.classes (… too much to type)

Pip
07 February, 2006 at 12:32

Thanks leo – I did say at the beginning of the post “Note that this doesn’t currently work in Firefox 1.5”.

When I get a chance, I really will look at an alternative that works in FF 1.5!

Anonymous
08 March, 2006 at 22:41

The extended history manager only lists the URLs when you copy and paste — it does not include page titles or view date/times.

Anonymous
17 March, 2006 at 18:14

FYI There’s a utility at
http://www.sunturbine.com/dork/
that exports the HISTORY.DAT file to a tab delimited text file. Free/source code included.

Anonymous
09 June, 2006 at 08:28

Hi,
Does anybody know here if it’s possible to force history.dat outside the profile like the brower.bookmarks.file staff?

I try to have this for my network users whom can’t get the whole profile roaming bu I could’nt find any clue.

Thanks.

Pip
09 June, 2006 at 09:52

Well, you can force the entire Firefox profile to a network drive, for example, using something along the lines of:

browser.cache.disk.parent_directory = X:wherever

Other than that, no, I don’t think so.

Anonymous
09 June, 2006 at 11:02

I guess I have no choice but doing this 🙁

Thank for your help.

Anonymous
27 June, 2006 at 07:21

Thank you very much for your efforts, I was able to extract my foxfire history into a text file: it worked Great 🙂

By using the warning you gave about the file being overwritten without a notice, I renamed the file before running another dat file through and was happy to not lose anything..

Thanks again
rhage2

Anonymous
28 June, 2006 at 03:55

I know most people haven’t heard of Google, but Google Browser-Sync is out for FF. (Check in the labs section). It is supposed to synchronize Passwords, Bookmarks, & History. I think there is a bug in the initial release, as when I try and sync one of my PCs with a large history or bookmark file, it quits. BUT, it’s a step in the right direction.

Anonymous
05 July, 2006 at 13:53

I would like and many of my customers would like, a way to remove all sexual terms and links from the history file, without clearing the history.
People get so embarrassed when you type a l in the browser and get lesbian something.. most just have me turn off history all together

Pip
05 July, 2006 at 16:22

You and me both!

Anonymous
24 July, 2006 at 01:14

Hi, I’m just a blue collar guy who likes to turn his history off for the same reason as mentioned before. I don’t no how though. Can someone help me with this? Much appreciated

Anonymous
16 November, 2006 at 19:28

Can someone send a solution for a mac user. I have students playing in bad sites and do not want to accuse the wrong one. You can email an answer to mmorse@hotmail.com

Phil Wilson
10 March, 2007 at 00:11

Hi Ben, thanks very much for that useful update!

Phil Wilson
03 June, 2007 at 21:59

Ace, thanks for that Ed.

Jim-BobH
22 July, 2007 at 17:53

Thanks to Ed Mullen for enabling me to access Dork and to its author Keith Anderson.

Suggestions to Keith and other users:

1. Every time you run it it will OVERWRITE any history.txt file it previously created, so if you don’t want to lose your “history history” you had better move or rename the existing one.

(The above fact is in the Dork readme.txt but the following warning is not.)

2. If you run it with Firefox running you will get:

ERROR: Could not open file. Error 32

which apparently means the file is in use by another application (in this case, I closed Firefox and Dork seems to work perfectly).

IHTH

Jim

Anonymous
05 December, 2007 at 12:23

if you get the “uncaught exception: Einem Skript von “http://philwilson.org” wurden UniversalXPConnect Sonderrechte verweigert.” error, this is a security issue. Try adding the following lines to your user.js file located in your profile dir (create it, if it does not exist):

user_pref(“capability.principal.philwilson.id”, “http://philwilson.org”);
user_pref(“capability.principal.philwilson.granted”, “UniversalXPConnect”);

Save the file, restart Firefox. This should work. But be careful with granting rights to pages you don’t know and trust!

dglp
06 December, 2007 at 23:31

Interesting stuff here. I am looking for a way to graph my browsing history and haven’t had any joy finding explanations of history.dat. And here we are at Firefox 2.0 and Mork is still in place.

Are there any straightforward ways of converting the mork file into a format that will preserve the number of visits per site?

Anonymous
09 December, 2007 at 06:06

In the latter case you don’t have to shutdown Firefox but you can just create a copy of the history.dat file and use the copy with DORK.
Thanks for this wonderful tool.