15.6.3.5. Using MySQL and memcached with Python

The Python memcache module interfaces to memcached servers, and is written in pure Python (that is, without using one of the C APIs). You can download and install a copy from Python Memcached.

To install, download the package and then run the Python installer:

python setup.py installrunning installrunning bdist_eggrunning egg_infocreating python_memcached.egg-info...removing 'build/bdist.linux-x86_64/egg' (and everything under it)Processing python_memcached-1.43-py2.4.eggcreating /usr/lib64/python2.4/site-packages/python_memcached-1.43-py2.4.eggExtracting python_memcached-1.43-py2.4.egg to /usr/lib64/python2.4/site-packagesAdding python-memcached 1.43 to easy-install.pth fileInstalled /usr/lib64/python2.4/site-packages/python_memcached-1.43-py2.4.eggProcessing dependencies for python-memcached==1.43Finished processing dependencies for python-memcached==1.43

Once installed, the memcache module provides a class-based interface to your memcached servers. When you store Python data structures as memcached items, they are automatically serialized (turned into string values) using the Python cPickle or pickle modules.

To create a new memcache interface, import the memcache module and create a new instance of the memcache.Client class. For example, if the memcached daemon is running on localhost using the default port:

import memcachememc = memcache.Client(['127.0.0.1:11211'])

The first argument is an array of strings containing the server and port number for each memcached instance to use. To enable debugging, set the optional debug parameter to 1.

By default, the hashing mechanism used to divide the items among multiple servers is crc32. To change the function used, set the value of memcache.serverHashFunction to the alternate function to use. For example:

from zlib import adler32memcache.serverHashFunction = adler32

Once you have defined the servers to use within the memcache instance, the core functions provide the same functionality as in the generic interface specification. The following table provides a summary of the supported functions:

Python `memcache` Function	Equivalent Generic Function
`get()`	Generic `get()`.
`get_multi(keys)`	Gets multiple values from the supplied array of `keys`. Returns a hash reference of key/value pairs.
`set()`	Generic `set()`.
`set_multi(dict [, expiry [, key_prefix]])`	Sets multiple key/value pairs from the supplied `dict`.
`add()`	Generic `add()`.
`replace()`	Generic `replace()`.
`prepend(key, value [, expiry])`	Prepends the supplied `value` to the value of the existing `key`.
`append(key, value [, expiry[)`	Appends the supplied `value` to the value of the existing `key`.
`delete()`	Generic `delete()`.
`delete_multi(keys [, expiry [, key_prefix]] )`	Deletes all the keys from the hash matching each string in the array`keys`.
`incr()`	Generic `incr()`.
`decr()`	Generic `decr()`.

Note

Within the Python memcache module, all the *_multi()functions support an optional key_prefix parameter. If supplied, then the string is used as a prefix to all key lookups. For example, if you call:

memc.get_multi(['a','b'], key_prefix='users:')

The function retrieves the keys users:a and users:b from the servers.

Here is an example showing the storage and retrieval of information to a memcache instance, loading the raw data from MySQL:

import sysimport MySQLdbimport memcachememc = memcache.Client(['127.0.0.1:11211'], debug=1);try:    conn = MySQLdb.connect (host = "localhost",                            user = "sakila",                            passwd = "password",                            db = "sakila")except MySQLdb.Error, e:     print "Error %d: %s" % (e.args[0], e.args[1])     sys.exit (1)popularfilms = memc.get('top5films')if not popularfilms:    cursor = conn.cursor()    cursor.execute('select film_id,title from film order by rental_rate desc limit 5')    rows = cursor.fetchall()    memc.set('top5films',rows,60)    print "Updated memcached with MySQL data"else:    print "Loaded data from memcached"    for row in popularfilms:        print "%s, %s" % (row[0], row[1])

When executed for the first time, the data is loaded from the MySQL database and stored to the memcached server.

shell> python memc_python.pyUpdated memcached with MySQL data

Because the data is automatically serialized using cPickle/pickle, when you load the data back from memcached, you can use the object directly. In the example above, the information stored to memcached is in the form of rows from a Python DB cursor. When accessing the information (within the 60 second expiry time), the data is loaded from memcached and dumped:

shell> python memc_python.pyLoaded data from memcached2, ACE GOLDFINGER7, AIRPLANE SIERRA8, AIRPORT POLLOCK10, ALADDIN CALENDAR13, ALI FOREVER

The serialization and deserialization happens automatically. Because serialization of Python data may be incompatible with other interfaces and languages, you can change the serialization module used during initialization. For example, you might use JSON format when you store complex data structures using a script written in one language, and access them in a script written in a different language.