python - Redis-py Sentinel ConnectionError race condition during failover -
i'm having issues redis-py sentinel in high-availability asynchronous work queue project design. starters, here's code (simplified):
from redis.sentinel import sentinel sentinel = sentinel(sentinels=sentinel_hosts, socket_timeout=0.1) r = sentinel.master_for('redis-sentinel-cluster') while true: item = r.rpop('work-queue') if item none: break else: # work on item continue
now, stands, redis-py has retry logic in determining redis connection use.
basically, if socket fails, try , new one:
https://github.com/andymccurdy/redis-py/blob/master/redis/sentinel.py#l44-l45
https://github.com/andymccurdy/redis-py/blob/master/redis/sentinel.py#l99-l108
and requests current master existing sentinels:
this works fine long outage (and new master election) completed before next redis call attempt.
however, if redis failure occurs—and sentinels have 5 second timeout before elect new one—the redis call fail, , sentinel connection request new master existing master gets returned! results in connectionerror.
obviously, can wrap redis calls in connectionerror try/excepts , sleep or retry appropriately, there cleaner way using existing connection params of redis-py? i've played around several different combinations no avail (and i'm still not 100% sure each 1 does).
ideally, sentinel redis connection throw connectionerror if period has elapsed, , still no masters available.
Comments
Post a Comment