6

I have a projects deployed on Google App Engine having Google API (Python). Every request to any of API make a database connection , execute a procedure and return data and close the connection. I was not able to access any of API as it was showing

"Process terminated because the request deadline was exceeded. (Error code 123)" and "This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application." error.

Database is also on cloud (Google Cloud SQL). As I checked there was 900 connection and more than 150 instances were up but no api request was getting handled. This happens frequently. So I restart database server and deploy API code again to solve this issue. What is the issue and how can I solve this permanently ? Here is my python code for database connectivity :-

import logging
import traceback
import os
import MySQLdb
from warnings import filterwarnings

filterwarnings('ignore', category = MySQLdb.Warning)

class TalkWithDB:
    def callQueries(self,query,req_args):
        try:
            if (os.getenv('SERVER_SOFTWARE') and os.getenv('SERVER_SOFTWARE').startswith('Google App Engine/')):
                db = MySQLdb.connect(unix_socket = UNIX_SOCKET + INSTANCE_NAME, host = HOST, db = DB, user = USER ,charset='utf8',use_unicode=True)
            else:
                db = MySQLdb.connect(host = HOST, port = PORT, db = DB, user = USER, passwd = PASSWORD,charset='utf8',use_unicode=True)

            cursor = db.cursor()
            cursor.connection.autocommit(True)
        try:
        sql = query+str(req_args)
        logging.info("QUERY = "+ sql )
        cursor.execute(sql)     
        procedureResult = cursor.fetchall();
        if str(procedureResult) == '()':
            logging.info("Procedure Returned 0 Record")
            procedureResult = []
            procedureResult.append({0:"NoRecord", 1:"Error"})
            #procedureResult = (("NoRecord","Error",),)
        elif procedureResult[0][0] == 'Session Expired'.encode(encoding='unicode-escape',errors='strict'):
            procedureResult = []
            procedureResult.append({0:"SessionExpired", 1:"Error"})                     
            except Exception, err:
        logging.info("ConnectDB.py : - Error in Procedure Calling :  " + traceback.format_exc())
        #procedureResult = (('ProcedureCallError','Error',),)
        procedureResult = []

        procedureResult.append({0:"ProcedureCallError", 1:"Error"})         

        except Exception, err:
            logging.info("Error In DataBase Connection : " + traceback.format_exc())
           #procedureResult = (('DataBaseConnectionError','Error',),)
        procedureResult = []

        procedureResult.append({0:"DataBaseConnectionError", 1:"Error"})       
# disconnect from server
        finally:
            try:
                cursor.close()
                db.close()
            except Exception, err:
                logging.info("Error In Closing Connection : " + traceback.format_exc())
            return procedureResult
Sunil Garg
  • 10,122
  • 12
  • 92
  • 133
  • *how can I solve it in future ?* haven't you already solved it? – Tim Jun 18 '15 at 10:51
  • 1
    this is solved for the time. But I don't think so this is due to changes in app.yaml for idle instances. And what is reason for this ? – Sunil Garg Jun 18 '15 at 11:36
  • you've increased number of idle instances -> more chances that you have an idle instance for a request -> decreased number of times when new instances needs to be started – Igor Artamonov Jun 18 '15 at 12:25
  • so that means in this case instances were not up? Then How can deployment solve the purpose ? – Sunil Garg Jun 18 '15 at 12:31
  • I deployed the application again with changes in app.yaml . After successful deployment everything was working fine . But If a changed idle instances for one project then how other API project started working ? – Sunil Garg Jun 18 '15 at 12:35
  • i'm talking about your app.yaml config. if it's for one project only, it helps for this project. i don't know about other projects – Igor Artamonov Jun 18 '15 at 12:41
  • The request that take 1-2 sec to serve, it starts taking more than 70 sec when there are huge traffic of requests. what is the problem ? – Sunil Garg Sep 09 '15 at 06:59

1 Answers1

4

Two possible improvements :

  • your startup code for instances may take too long, check what is the startup time and if possible use warmup requests to reduce startup times. Since increasing your idle instances seems to help, your startup time may take too long.
  • A better approach would be to call external services (e.g. talk to Google Calendar) in a Task Queue outside of the user request scope. This gives you a 10-min deadline instead of the 60s deadline for user requests
koma
  • 6,257
  • 2
  • 21
  • 48
  • But everything was working fine. Instances were up. Then how is this possible ? – Sunil Garg Jun 19 '15 at 08:57
  • Enable cloud monitoring on your project and verify what is taking so long, you get a nice breakdown of all calls and their timings. Metrics avoid having to guess.... – koma Jun 19 '15 at 12:49
  • warmup requests are not working and TaskQueus are not best fit in my situations – Sunil Garg Sep 09 '15 at 06:57