python - Redis-py Sentinel ConnectionError race condition during failover -


i'm having issues redis-py sentinel in high-availability asynchronous work queue project design. starters, here's code (simplified):

from redis.sentinel import sentinel  sentinel = sentinel(sentinels=sentinel_hosts, socket_timeout=0.1) r = sentinel.master_for('redis-sentinel-cluster')  while true:     item = r.rpop('work-queue')     if item none:          break     else:         # work on item      continue 

now, stands, redis-py has retry logic in determining redis connection use.

basically, if socket fails, try , new one:

  1. https://github.com/andymccurdy/redis-py/blob/master/redis/sentinel.py#l44-l45

  2. https://github.com/andymccurdy/redis-py/blob/master/redis/sentinel.py#l99-l108

and requests current master existing sentinels:

  1. https://github.com/andymccurdy/redis-py/blob/master/redis/sentinel.py#l204-l223

this works fine long outage (and new master election) completed before next redis call attempt.

however, if redis failure occurs—and sentinels have 5 second timeout before elect new one—the redis call fail, , sentinel connection request new master existing master gets returned! results in connectionerror.

obviously, can wrap redis calls in connectionerror try/excepts , sleep or retry appropriately, there cleaner way using existing connection params of redis-py? i've played around several different combinations no avail (and i'm still not 100% sure each 1 does).

ideally, sentinel redis connection throw connectionerror if period has elapsed, , still no masters available.


Comments

Popular posts from this blog

java - UnknownEntityTypeException: Unable to locate persister (Hibernate 5.0) -

python - ValueError: empty vocabulary; perhaps the documents only contain stop words -

ubuntu - collect2: fatal error: ld terminated with signal 9 [Killed] -