Inconsistent cache values using Zend Cache with AWS ElastiCache across multiple servers

Question

We are using Zend Cache with a memcached backend pointing to an AWS ElastiCache cluster with 2 cache nodes. Our cache setup looks like this:

$frontend = array(
    'lifetime' => (60*60*48),
    'automatic_serialization' => true,
    'cache_id_prefix' => $prefix
);
$backend = array(
    'servers' => array(
        array( 'host' => $node1 ),
        array( 'host' => $node2 )
    )
);
$cache = Zend_Cache::factory('Output', 'memecached', $frontend, $backend);

We have not noticed any problems with the cache in the past when using a single EC2 server to write and read from the cache.

However, we have recently introduced a second EC2 server and suddenly we're seeing issues when writing to the cache from one server and reading from another. Both servers are managed by the same AWS account, and neither server has issues writing to or reading from the cache individually. The same cache configuration is used for both.

Server A executes $cache->save('hello', 'message');

Subsequent calls to $cache->load('message'); from Server A return the expected result of hello.

However, when Server B executes $cache->load('message');, we get false.

As far as my understanding of ElastiCache goes, the server making the read request should have no bearing on the cache value returned. Can anyone shed some light on this?

I would assume this is a latency issue, have you tried to sleep(xxxx) and then perform the $cache->load from B? — Julius F, Sep 17 '12 at 22:33
Unfortunately, this is not the case. Even hours later a value set from A is not readable from B. — michaelxor, Sep 17 '12 at 22:49
Which version of PHP are you using? I think the serialization is what is in play here. Try disabling the auto serialization and see what happens. The unfortunate side effect is that you have to serialize everything manually that is not a string. — Rich H., Oct 16 '12 at 21:06
Additionally is the cache_id_prefix the same or a generated value? — Rich H., Oct 16 '12 at 21:08
I would try writing some test code using PHP's [Memcache](http://php.net/manual/en/book.memcache.php), removing Zend_Cache from the equation. I never much liked the Memcache extension and have always favored [Memcached](http://php.net/manual/en/book.memcached.php) (but don't get me started on why the hell they added a 'd' to the client). For one it supports compare-and-swap operations. — ficuscr, Oct 17 '12 at 18:35
Hey guys, thanks for the feedback. Unfortunately we are back down to 1 server (we were migrating from A to B at the time), so I haven't had a chance to test this any further in the last several days. I will have some more info soon. @Rich PHP 5.3.3, Zend 1.11.9, and the $prefix is a pre-determined value we use to 'partition' the cache (it was the same for both servers). ficuscr, I may take your advice. We _are_ using ZendCache's Memcached backend as [Elasticache](http://docs.amazonwebservices.com/AmazonElastiCache/latest/UserGuide/Introduction.html) is 'protocol compliant with Memcached.' — michaelxor, Oct 18 '12 at 19:21

score 1 · Answer 1 · answered Jan 02 '13 at 11:09

1

Can you tell what hash_strategy you are using for memcache? I've had problems in the past using the default standard but everything has been fine since changing to consistent:

http://php.net/manual/en/memcache.ini.php#ini.memcache.hash-strategy

answered Jan 02 '13 at 11:09

Jason

487
3
9

score 0 · Answer 2 · edited Jun 20 '20 at 09:12

With a normal hashing algorithm, changing the number of servers can cause many keys to be remapped to different servers resulting in huge sets of cache misses.

Imagine you have 5 ElastiCache Nodes in your cache Cluster, adding an sixth server may cause 40%+ of your keys to suddenly point to different servers than normal. This activity is undesirable, may cause cache misses and eventually swamping your backend DB with requests. To minimize this remapping it is recommended to follow consistent Hashing model in your cache clients.

Consistent Hashing is a model that allows for more stable distribution of keys given addition or removal of servers. Consistent Hashing describes methods for mapping keys to a list of servers, where adding or removing servers causes a very minimal shift in where keys map to. Using this approach, adding an eleventh server should cause less than 10% of your keys to be reassigned. This % may vary in production but it is far more efficient in such elastic scenarios compared to normal hash algorithms. It is also advised to keep memcached server ordering and number of servers same in all the client configurations while using consistent Hashing. Java Applications can use “Ketama library” through spymemcached to integrate this algorithm into their systems. More information on consistent hashing can be found at http://www.last.fm/user/RJ/journal/2007/04/10/rz_libketama_-_a_consistent_hashing_algo_for_memcache_clients

Inconsistent cache values using Zend Cache with AWS ElastiCache across multiple servers

2 Answers2