Sunday, October 4, 2015

The Force Awakens ... on Twitter


TL;DR

Last week I wrote about Markov Chains in a post titled, "Using Python to Sound Like a Wine Snob". Yesterday morning, while laying in bed, I thought it would be a fun exercise to take this idea further. How about creating a Python script that automatically generates, and posts, humorous tweets in the voice of Yoda, combined with the wisdom of Zen masters? I created and tested the code, and thus the creation of the Yoda parody twitter handle, @YodaUncut.

A day later I felt it would be valuable to write a blog post sharing what I coded. This code is like Burger King, "have it your way, but don't go crazy". Please, don't abuse this with Twitter (see Twitter's Terms of Service). I'm sharing this as educational material that hopefully inspires some people to pursue, or continue learning, coding ... or at least provide some answers for anyone who's stuck with their existing code, searching the web for some answers ... else, for your amusement?!?

Be original

In my example I blend quotes from Yoda and various Zen masters. If you go through this exercise, pick something different. For example, how about combining quotes from Warren Buffett and Darth Vader, or 50 Shades of Grey and Mother Teresa?

Criteria

Here's the initial criteria I came up with:

  1. Purpose: Generate amusing Yoda'esque tweets using Markcov Chains
  2. The tweets would blend the wisdom of Yoda with the koans of various Zen masters
  3. The generated tweets would need to be 140 characters, or less, in length
  4. Create a random latitude and longitude in North America for Yoda's geolocation
  5. Post the tweet on Twitter with Yoda's geolocation

Requirements

In order for the Python script to work, I used the following third party libraries:

  • oauthlib==1.0.3
  • requests==2.7.0
  • requests-oauthlib==0.5.0

The above libraries were installed with:

-----
$ easy_install pip
Searching for pip
Best match: pip 6.0.8
Adding pip 6.0.8 to easy-install.pth file
Installing pip script to /tmp/test/bin
Installing pip3.4 script to /tmp/test/bin
Installing pip3 script to /tmp/test/bin
Using /tmp/test/lib/python2.7/site-packages
Processing dependencies for pip
Finished processing dependencies for pip

$ pip install requests oauthlib requests-oauthlib
Collecting requests
Using cached requests-2.7.0-py2.py3-none-any.whl
Collecting oauthlib
Using cached oauthlib-1.0.3.tar.gz
Collecting requests-oauthlib
Using cached requests_oauthlib-0.5.0-py2.py3-none-any.whl
Installing collected packages: requests-oauthlib, oauthlib, requests
Running setup.py install for oauthlib
Successfully installed oauthlib-1.0.3 requests-2.7.0 requests-oauthlib-0.5.0
-----

The code was written for, and tested with, Python version 2.7.9. It hasn't been tested in with Python 3.

-----
$ python --version
Python 2.7.9
-----

Generate Twitter API and token keys/secrets

You will need to get your Twitter API and token keys/secrets. If you haven't already:

  1. Go to https://apps.twitter.com/
  2. Create a new app
  3. Once created, scroll down to "Application Settings" and click on "manage keys and access tokens"
  4. Copy you "Consumer Key (API Key)" and "Consumer Secret (API Secret)"
  5. Scroll down to "Token Actions" and click on "Create my access token"
  6. Copy your "Access Token" and "Access Token Secret"

Save your keys/secrets. Don't share them. Don't upload them to your git repository.

For the sake of this example, put them in a file called "keys.txt". The syntax of keys.txt will look like the following:

-----
$ cat keys.txt
client_key:
client_secret:
token:
token_secret:
-----

Imports

Starting our Python code with the imports:

  • We want "urllib" for proper encoding of our tweets
  • We import "choice" from "random" so that we can randomly choose text as part of the Markcov Chain
  • We also want to import "randint" from "random" for generating some random numbers for geolocation (latitude and longitude)

The two remaining imports require third party libraries to first be installed:

  • To post our tweet over HTTPS to Twitter's API, we'll need "requests"
  • We also need "OAuth1" from "requests_oauthlib" for authentication with Twitter

Let's start with the Python code...

# Local libraries
import urllib
from random import choice, randint
# Third party libraries
import requests
from requests_oauthlib import OAuth1

Taking about geolocation...

In order for the latitude and longitude geolocation to show up in the tweets, geolocation needs to be enabled for the Twitter account. To enable geolocation sharing:

  1. Go to your Twitter account settings
  2. On the left side, click on "Security and privacy"
  3. Scroll down to "Privacy" > "Tweet location" and check "Add a location to my Tweets"
  4. Click on "Save changes"

Preparing the environment

Adjust the following filenames to whatever you're using. In my example, I have a file that has Yoda quotes, and a second file that has various Zen master quotes.

# Quotes filenames
quote_filename1 = 'yoda_quotes.txt'
quote_filename2 = 'zen_quotes.txt'

The format of both quote files looks something like the following. Each quote is on a new line. For example:

-----
$ tail -n 25 yoda_quotes.txt | head -n 5
size matters not. look at me. judge me by my size do you?
so certain are you. always with you it cannot be done. hear you nothing that I say?
strong am I with the Force, but not that strong. twilight is upon me, and soon night must fall. that is the way of things... the way of the Force.
that is why you fail.
the boy you trained, gone he is, consumed by Darth Vader.

$ tail -n 25 zen_quotes.txt | head -n 5
no yesterday, no tomorrow, and no today.
nothing is exactly as it seems, nor is it otherwise.
one falling leaf is not just one leaf; it means the whole autumn.
the Force is not some kind of excitement, but merely concentration on our usual everyday routine.
the Force is selling water by the river.
-----

Twiter's API base URL

Here we define the base URL for Twitter's API endpoints.

# Base URL for Twitter calls
base_twitter_url = "https://api.twitter.com/1.1/"

Create a function to get your Twitter API and token keys/secrets from keys.txt

def get_twitter_credentials():
    """
    This function reads in the Twitter credentials and organizes them in a dictionary
    :return:  Return the twitter credentials as a dict
    """
    f = open('keys.txt', 'rb')
    raw_credentials = f.readlines()
    f.close()
    credentials = {}
    for credential in raw_credentials:
        key = credential.split(':')[0]
        value = credential.split(':')[1].strip('\n')
        credentials[key] = value
    return credentials

Reading in the quotes

We also need a function that will read in the quotes from our quotes files.

def get_quotes(filename):
    """
    This function read the quotes in from a file
    :param filename:  The filename of the file that contains the quotes
    :return:  A string with the quotes concatenated together. Newlines are removed.
    """
    f = open(filename, 'rb')
    quotes = f.readlines()
    f.close()
    quotes_list = []
    for quote in quotes:
        quotes_list.append(quote.strip('\n'))
    return " ".join(quotes_list)

Create a dictionary for use with the Markcov Chain

This is the function used last week in "Using Python to Sound Like a Wine Snob".

def create_markcov_dict(original_text):
    """
    This function takes the quotes and creates a dictionary with the words
    in Markcov chunks
    :param original_text:  The plaintext to chunck up
    :return:  Return the dictionary with Markcov chunks
    """

    original_text = original_text
    split_text = original_text.split()
    markcov_dict = {}
    for i in xrange(len(split_text) - 2):
        key_name = (split_text[i], split_text[i+1])
        key_value = split_text[i+2]
        if key_name in markcov_dict:
            markcov_dict[key_name].append(key_value)
        else:
            markcov_dict[key_name] = [key_value]
    return markcov_dict

Generating the tweet, 140 characters, or less, using the Markov Chain

This is a modified version of the function used from last week's wine snob example.

def create_markcov_tweet(markcov_dict):
    """
    This function creates the Tweet using the Markov Chain
    :param markcov_dict:  The dictionary with the text in Markcov chunks
    :return:  Return the tweet
    """
    # Pick a random starting point
    selected_words_tuple = choice(markcov_dict.keys())
    markcov_tweet = [selected_words_tuple[0].capitalize(), selected_words_tuple[1]]
    # Generate the Markcov text, ending the Markcov text when we create a "key" that doesn't exist
    while selected_words_tuple in markcov_dict:
        next_word = choice(markcov_dict[selected_words_tuple])
        if len(" ".join(markcov_tweet) + " " + next_word) < 140:
            if (markcov_tweet[-1]).endswith('.'):
                markcov_tweet.append(next_word.capitalize())
            else:
                markcov_tweet.append(next_word)
            selected_words_tuple = (selected_words_tuple[1], next_word)
        else:
            tweet = " ".join(markcov_tweet).strip()
            if tweet.endswith(","):
                tweet = tweet.rstrip(',') + '.'
            elif not (tweet.endswith(".") or
                      tweet.endswith("!") or
                      tweet.endswith("?") or
                      tweet.endswith(";") or
                      tweet.endswith(":")):
                tweet = tweet + '.'
            return tweet

Create a random latitude and longitude in North America

This is dirty, but works fine our purpose.

def get_geolocation():
    """
    This function generates a random latitude and longitude, somwhere
    in Northern America.
    :return:  Returns a dict with the lat and long
    """

    latitude = "{}.{:07d}".format(randint(28, 69), randint(0, 999999))
    longitude = "{}.{:07d}".format(randint(-128, -64), randint(0, 999999))
    return {'lat':latitude, 'long':longitude}

Create function to post our tweet via HTTPS using Twitter's API

This is how we'll upload our tweet to Twitter

def post_tweet(markcov_tweet):
    """
    This function posts the tweet to Twitter
    :param markcov_tweet:  The text to tweet
    :return:  Return the response from requests.post()
    """

    # Setup authentication
    credentials = get_twitter_credentials()
    client_key = credentials['client_key']
    client_secret = credentials['client_secret']
    token = credentials['token']
    token_secret = credentials['token_secret']
    oauth = OAuth1(client_key, client_secret, token, token_secret)
    # Get latitude and longitude for the tweet
    coordinates = get_geolocation()
    # Make the URL
    api_url = "{}statuses/update.json".format(base_twitter_url)
    api_url += "?status={}".format(urllib.quote(markcov_tweet))
    api_url += "&lat={}&long={}".format(coordinates['lat'], coordinates['long'])
    api_url += "&display_coordinates=true"
    # tweet
    response = requests.post(api_url, auth=oauth)

    return response

Tying it all together

Now that we have everything in place, let's take this out for a test drive. We're going to generate a tweet and post it to Twitter!

We begin by reading in, and concatenating, all of the quotes from both quote files.

quotes = get_quotes(quote_filename1) + " " + get_quotes(quote_filename2)

For the sake of this example, let's do a quick test to see the first 1000 characters loaded:

print quotes[0:1000]
I cannot teach him. the boy has no patience. I hear a new apprentice you have, Emperor. or should I call you Darth Sidious? Master Obi-Wan, not victory. the shroud of the dark side has fallen. begun, the Clone War has! Qui-Gon's defiance I sense in you. need that you do not. agree with you, the Council does. your apprentice young Skywalker will be. Yoda I am, fight I will. a labyrinth of evil, this war has become. a trial of being old is this: remembering which thing one has said into which young ears. and well you should not! for my ally is the Force. and a powerful ally it is. life creates it, makes it grow. its energy surrounds us... and binds us. luminous beings are we, not this... crude matter! you must feel the Force around you. here, between you, me, the tree, the rock... everywhere! even between the land and the ship. around the survivors, a perimeter create! at an end your rule is... and not short enough it was. awww, cannot get your ship out...eh-heheheh! careful you must be

We'll feed text into our Markcov dictionary creation function

markcov_dict = create_markcov_dict(quotes)

Let's also see how many chunks of text we have in our newly created dictionary as a quick test.

print len(markcov_dict)
1595

Generate the tweet

We'll generate the tweet with our Markcov function and quotes:

markcov_tweet = create_markcov_tweet(markcov_dict)
# Let's see what we'll be tweeting and the number of characters of the tweet
print("[+] Tweet ({}): \"{}\"".format(len(markcov_tweet), markcov_tweet))
[+] Tweet (140): "Of evil, this war has become. A trial of being old is this: remembering which thing one has said into which young ears. And well you should."

Ready ... aim ... FIRE!

Finally, let's post the tweet and get a status of whether or not the tweet was posted successfully!

response = post_tweet(markcov_tweet)
if response.status_code == 200:
    print("[+] Tweet posted successfully")
else:
    print("[-] Tweet post failed")
[+] Tweet posted successfully

Back to this world...

The tweet has now been posted on Twitter. Good job!

I hope you found this useful. Please share suggestions, improvements, and any additional features you've created to enhance the code.

Afterthoughts

  • If you want to run this via cron, you'll want to use absolute paths for the filenames (both quotes files and the keys.txt file).
  • If you get an "InsecurePlatformWarning" when running the script, no worries. The tweets will still post. If you want to get rid of the warning:

    $ sudo apt-get install python-dev libffi-dev libssl-dev
    $ pip install requests[security]
  • Sample Code: I wrote some sample code and put it in a Github repository called Twoda. I added the ability to Tweet animated GIFs from Giphy that pertain to the general theme of your tweets.

No comments: