2

My script grabs the content of an rss page gets the urls in that page saves them to a list then it grabs the content of each url and it emails the contents of the page to me. Everything is working very well accept I can't send every link in the list. Typically about 22 links in the list. I don't want to combine the contents of multiple links into one email. If I don't add a timeout I get an over quota error like this

<class 'google.appengine.runtime.apiproxy_errors.OverQuotaError'>: The API call mail.Send() required more quota than is available.

After I added "time.sleep(9)" to slow it down it gives me this error.

<class 'google.appengine.runtime.DeadlineExceededError'>: 
Traceback (most recent call last):

Here is my code.. Any thoughts?

size = len(my_tabletest)
a=2 
while a < size:
  url = my_tabletest[a].split('html</link>')[0] + "print"
  url_hhhhhh = urlfetch.fetch(url)
  my_story = url_hhhhhh.content
  my_story = my_story.split('<div class="printstory">')[1]
  my_story_subject = my_story.split('<h1>')[1]
  my_story_subject = my_story_subject.split('</h1>')[0]
  my_story =  ''.join(BeautifulSoup(my_story).findAll(text=True))
  message = mail.EmailMessage(sender="me<me@someplace.com>",
  subject=my_story_subject)
  message.to = "Jim <me@someplace.com>"
  message.body = my_story
  message.html = my_story_html
  message.send()
  time.sleep(9)
  a=a+1
jim
  • 71
  • 1
  • 7
  • You're probably hitting the [per minute quota](http://code.google.com/appengine/docs/quotas.html#Per-minute_Quotas). I suppose the Tasks API could be a workaround here but I haven't used those before. On a side note, consider using [Feed Parser](http://www.feedparser.org/) if you're doing a lot of manual RSS parsing. – hyperslug May 01 '11 at 01:07

1 Answers1

6

Welcome to Stack Overflow!

The task queue is built to solve this problem. You can leverage it with minimal change to your existing code using the deferred library:

Instead of calling message.send(), do something like this:

def send_email(message):  
  message.send()

deferred.defer(send_email, message)

This will create a batch of ad-hoc tasks that send your emails in the background, after your main request handler has returned. Some of these tasks will probably fail on the first try as your app hits short term quota limits for outbound mail. That's OK; failed tasks will back off and retry automatically until they succeed.

Edit: Oh, and take the sleep out of your code. =)

Edit #2: You can speed things up further by moving the urlfetch into the task, so each task fetches one URL and then sends one email. Fetching 22 URLs in one request handler could be enough to cause timeouts, independent of sending mail.

Drew Sears
  • 12,652
  • 1
  • 28
  • 39