Twitter Archiving

Been asked a few times about how to get all a user’s tweets off twitter and onto a local file.

Activestate have a great python recipe to do it that works well. You just have to get python and BeautifulSoup onto your PC.

First install Python:

  • Linux, it’s easy, just check your package manager and get python.
  • Windows, it’s easy, go to activestates website a download and install ActivePython

Then install BeautifulSoup:

  • Download BeautifulSoup tar
  • Unzip it – you may need to untar it too. On Windows you can just use 7-zip or WinZip on the tar file as if it was a zip file. You will end up with a folder called BeautifulSoup-X.x.x.
  • Open a terminal or Command Prompt and cd to the BeautifulSoup-X.x.x folder
  • Type python setup.py install and hit return
  • BeautifulSoup is installed

Now setup the program that gets the data and saves it for you.

  • Open the activestate recipe page.
  • Create a new text file in your Documents folder called as twitterArchive.py (make sure you get rid of the .txt extension)
  • Edit the file and copy paste the recipe into the file:
    #!/usr/bin/python
    
    import time
    from urllib2 import urlopen
    from BeautifulSoup import BeautifulSoup
    
    # Replace USERNAME with your twitter username
    url = u'http://twitter.com/USERNAME?page=%s'
    tweets_file = open('tweets', 'w')
    
    for x in range(10*10000):
        f = urlopen(url % x)
        soup = BeautifulSoup(f.read())
        f.close()
        tweets = soup.findAll('span', {'class': 'entry-content'})
        if len(tweets) == 0:
            break
        [tweets_file.write(x.renderContents() + '\n') for x in tweets]
        # being nice to twitter's servers
        time.sleep(5)
    
    tweets_file.close()
    
  • Update USERNAME as instructed in the recipe and save it
  • Run the program. On Windows you can double-click it or run from a Command Prompt. On linux you’ll need to run it through python: python -c twitterArchive.py
  • It will create a file called tweets in the same folder. After a while (it can take some time) the file will be filled with your tweets.
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s