Thursday, February 25, 2010

Stupid Boto Tricks #2 - Reliable Counters in SimpleDB

As a follow-up to yesterday's article about the new consistency features in SimpleDB, I came up with a handy little class in Python to implement a reliable integer counter in SimpleDB.  The Counter class makes use of the consistent reads and conditional puts now available in SimpleDB to create a very Pythonic object that acts like an integer object in many ways but also manages the synchronization with the "true" counter object stored in SimpleDB.

The source code can be found in my bitbucket.org repo.  I have copied the doc string from the class below to give an example of how the class can be used.  Comments, questions and criticisms welcome. As with all Stupid Boto Tricks, remember the code is hot off the presses. Use with appropriate skepticism.


    A consistent integer counter implemented in SimpleDB using new
    consistent read and conditional put features.

    Usage
    -----
    To create the counter initially, you need to instantiate a Counter
    object, passing in the name of the SimpleDB domain in which you wish
    to store the counter, the name of the of the counter within the
    domain and the initial value of the counter.

    >>> import counter
    >>> c = counter.Counter('mydomain', 'counter1', 0)
    >>> print c
    0
    >>>

    You can now increment and decrement the counter object using
    the standard Python operators:

    >>> c += 1
    >>> print c
    1
    >>> c -= 1
    >>> print c
    0

    These operations are automatically updating the value in SimpleDB
    and also checking for consistency.  You can also use the Counter
    object as an int in normal Python comparisons:

    >>> c == 0
    True
    >>> c < 1
    True
    >>> c != 0
    False

    If you have multiple processes accessing the same counter
    object it will be possible for your view of the Python to become
    out of sync with the value in SimpleDB.  If this happens, it will
    be automatically detected by the Counter object.  A ValueError
    exception will be raised and the current state of your Counter
    object will be updated to reflect the most recent value stored
    in SimpleDB.

    >>> c += 1
    ---------------------------------------------------------------------------
    ValueError                                Traceback (most recent call last)
    ...
    ValueError: Counter was out of sync
    >>> print c
    2
    >>>

    In addition to storing the value of the counter in SimpleDB, the
    Counter also stores a timestamp of the last update in the form of
    an ISO8601 string.  You can access the timestamp using the
    timestamp attribute of the Counter object:

    >>> c.timestamp
    '2010-02-25T13:49:15.561674'
    >>>

1 comment:

  1. I got counter always out of sync
    think it is bug somewhere during expected_values comparison

    \boto\contrib\counter.py", line 158, in update
    raise ValueError, 'Counter was out of sync'
    ValueError: Counter was out of sync

    sdberr looks like:
    Conditional check failed. Attribute (current_value) value exists

    ReplyDelete