Twitter Archiving

Been asked a few times about how to get all a user’s tweets off twitter and onto a local file.

Activestate have a great python recipe to do it that works well. You just have to get python and BeautifulSoup onto your PC.

First install Python:

  • Linux, it’s easy, just check your package manager and get python.
  • Windows, it’s easy, go to activestates website a download and install ActivePython

Then install BeautifulSoup:

  • Download BeautifulSoup tar
  • Unzip it – you may need to untar it too. On Windows you can just use 7-zip or WinZip on the tar file as if it was a zip file. You will end up with a folder called BeautifulSoup-X.x.x.
  • Open a terminal or Command Prompt and cd to the BeautifulSoup-X.x.x folder
  • Type python setup.py install and hit return
  • BeautifulSoup is installed

Now setup the program that gets the data and saves it for you.

  • Open the activestate recipe page.
  • Create a new text file in your Documents folder called as twitterArchive.py (make sure you get rid of the .txt extension)
  • Edit the file and copy paste the recipe into the file:
    #!/usr/bin/python
    
    import time
    from urllib2 import urlopen
    from BeautifulSoup import BeautifulSoup
    
    # Replace USERNAME with your twitter username
    url = u'http://twitter.com/USERNAME?page=%s'
    tweets_file = open('tweets', 'w')
    
    for x in range(10*10000):
        f = urlopen(url % x)
        soup = BeautifulSoup(f.read())
        f.close()
        tweets = soup.findAll('span', {'class': 'entry-content'})
        if len(tweets) == 0:
            break
        [tweets_file.write(x.renderContents() + '\n') for x in tweets]
        # being nice to twitter's servers
        time.sleep(5)
    
    tweets_file.close()
    
  • Update USERNAME as instructed in the recipe and save it
  • Run the program. On Windows you can double-click it or run from a Command Prompt. On linux you’ll need to run it through python: python -c twitterArchive.py
  • It will create a file called tweets in the same folder. After a while (it can take some time) the file will be filled with your tweets.

Easy Group Twittering?

@danielbower was wondering whether there is a way to address multiple twitter users in a reply easily. There are a couple of ways that spring to mind:

  1. You create another twitter account specially for the group. Let’s call it @keystone. Everyone in the group friends up @keystone and posts group messages to @keystone. Simple.
  2. Decide on a keyword for the group twittering. Let’s call ours grpKeyStone. Have a crontab watching each of the group members twitter rss. When a post is on the rss starting with grpKeyStone , the script re-tweets to each of the other members of the group with the usual @name.
  3. Option 2 doesn’t address your other friends getting tweets with the incomprehensible grpkeyStone on the beginning of them, so may be worth combining the above options. Have a separate twitter account to target the group messages at, and a script watching and re-tweeting the messages to the specific members of the group.

Clear as mud 🙂

Eurovision Madness

Russia won. Gutted.

Ukraine should have had it. Turkeys wasn’t a Eurovision song, way too good. Ditto Britains Andy Abrahams who was cruelly, but totally predictably, ignored to the tune (ahem) of joint last on about 16 points with Polands answer to Ken‘s Dad’s prayers (sorry, that was evil comment, but blimey Poland). Those three were the only songs that you could actually remember 10 mins after hearing them. Stood out from the miasma of ear-assault that is so much a feature of Eurovision.

My favourite for it’s utter Eurovision nuttiness was Bosnia and Hertzegovina… no recollection of the tune but the show was mental and not obvious.

Was going to try to live blog it, but it interrupts watching too much (see last years attempts if you like). Found that Twittering, linked to Facebook status, is an acceptable alternative, esp if others on Twitter are watching too.