2

I'm iterating over several JSON objects at several files. I will make Twitter API calls with some parameters from such JSON objects.

However, Twitter API got a limit and I can't exceed more than 180 calls per a 15 minutes period.

API calls are executed using a library, let's call api the object which makes the calls.

Having one, I would use a sleep every time I get to the limit and after the wait I'd continue. However, as I can use more than one api object for different Twitter apps, I'd like to assign them the call statements and make them wait if necessary but not making the other api stop if not necessary or the whole execution.

Let's say I have:

api1, api2, api3, api4, api5 ... api10

This is the code where I iterate and get to the point of API call:

for f in os.listdir('tweets'):
            #print f
            with open('tweets/'+f,'r', buffering=1) as jsonQuery:
                twitterJSON = json.load(jsonQuery)
                for category in twitterJSON:
                    for trend in twitterJSON[category]:
                        j = 0
                        for t in twitterJSON[category][trend]:
                            self.dist=[]
                            values = []

                            # I'd make the call here for 
                            # any of the api variables which gets its turn.

How can I thread the task at loop and round robin to any of the api's?

diegoaguilar
  • 7,496
  • 12
  • 62
  • 120
  • Do you need 'round robin'? This confuses me. One can parallelize your code. If you want a special sheduler this becomes more difficult. – User Apr 25 '14 at 12:43
  • I think is more simple than what I maybe suggested. I want n api instances doing queries at same time and waiting when they have to (180 queries per second). Why did I mentioned round robin? Because at loop they receive an argument for querying, I thought by round robin any api instance would be picked for the task – diegoaguilar Apr 25 '14 at 12:54
  • How does [this](http://stackoverflow.com/questions/9786102/how-do-i-parallelize-a-simple-python-loop) relate to your question? – User Apr 25 '14 at 13:07

0 Answers0