no matching index found - GAE

Question

I'm deploying a little backend with some methods. One of them makes a simple query to retrieve a list of objects. This is the method:

@ApiMethod(path = "getMessagesByCity", name = "getMessagesByCity", httpMethod = ApiMethod.HttpMethod.POST)
    public MessageResponse getMessagesByCity(@Named("City_id") Long city) {
        MessageResponse response = new MessageResponse();
        List<Message> message = ofy().load().type(Message.class).filter("city", city).list();
        response.response = 200;
        return response;
    }

And this is the Message class:

@Entity
public class Message {
    @Id 
    private Long id;
    private String name;
    @Index
    private Long city;
    ...
}

I've read a lot of posts and all of them are mentioning that probably is caused because datastore-indexes.xml are not being updated automatically. However, Google doc says this (https://cloud.google.com/appengine/docs/standard/python/config/indexconfig):

Every Cloud Datastore query made by an application needs a corresponding index. Indexes for simple queries, such as queries over a single property, are created automatically.

So,following that, I think that index related files are not necessary for me.

If I execute the method "getMessagesByCity" with the simple query:

List<Message> message = ofy().load().type(Message.class).filter("city", city).list();

The backend returns me an error 503 with this log message:

"com.google.appengine.api.datastore.DatastoreNeedIndexException: no matching index found. An index is missing but we are unable to tell you which one due to a bug in the App Engine SDK. If your query only contains equality filters you most likely need a composite index on all the properties referenced in those filters."

Any idea? How can I solve it?

score 1 · Answer 1 · answered Jun 13 '18 at 16:58

1

You need to upload index configs in, so Datastore will start to accept your queries with custom projections with this command.

gcloud app deploy index.yaml

See https://cloud.google.com/datastore/docs/concepts/indexes for more information about Datastore queries handling and indexes.

answered Jun 13 '18 at 16:58

QuestionAndAnswer

2,963
4
23
44

but, index.yaml is needed? I've created it, but it is not working yet =( However, in firestore panel, appears a message like this: "The unique fields found in all the documents in a collection are automatically indexed in order to be able to sort them and execute simple queries." My query is simple, so I understand that I should not create any file about indexing – adlagar Jun 14 '18 at 10:10
Are you using Firestore or Datastore? – A.Queue Jun 15 '18 at 09:33
I'm using Firestore. I've tried to create indexes in a manual way and it does not work either. – adlagar Jun 18 '18 at 08:02
The link in this answer is related to Datastore which is different from Firestore. – A.Queue Jun 18 '18 at 14:00
Make it clear that you are using Firestore and not Datastore in the main question. – A.Queue Jun 18 '18 at 14:02

score 1 · Answer 2 · answered Jun 14 '18 at 12:27

1

Every time you use a new datastore query in your code with a different set of filters / orders etc. your index.yaml should automatically update, (might need to run that logic at least once in the local dev server for it to add the new index to the file)

On local dev, the first time you hit it, it should work, HOWEVER when deploying new indexes, there is a lag time before it becomes available in production / on the appspot server. We have run into this a lot, and from the google console you can actually see if its in progress or not, by going to Datastore > Indexes (https://console.cloud.google.com/datastore/indexes) for the project in question.

If all indexes have a green tick and the issue persists then this is not the issue, and can debug further, however if some have spinners next to them this means this index is still being made and cannot be used until its finished.

If this is your problem, you can avoid it in the future by deploying the index.yaml first through gcloud and then only deploying your application.

alternatively make sure you have run the new method / function on your local, and make sure that the index.yaml did in-fact get changed, if you use Git or something the file should have popped up as modified after the local server ran the function / method.

answered Jun 14 '18 at 12:27

Richard Hazelden

88
6

Thanks so much for your comment! In localhost it is working fine... I can make the query and there is not any error, however when I deploy the backend to App Engine, it is still with the same 503 error – adlagar Jun 15 '18 at 08:38
index.yaml is 100% needed, and no query will work in GAE to Datestore if there isnt a matching entry in your index.yaml index.yaml should autogenerate, and you should never edit it manually or delete it, No matter how simple your query is, the fundamentals of datastore require it. – Richard Hazelden Jun 16 '18 at 17:34
its not like a traditional DB where an index is an optional thing to optimise a specific look up, Datastore indexes are literally the only query your datasote can run, to the point where a query to "Table/Collection" X where Y < 10 would require 1 entry in the index.yaml, and then a query to the same "Table/Collection" X where Z > 10, would create a whole new entry in the index.yaml, so for 1 "Table/Collection" you might have many rules in the yaml, hence why GAE autogens based on what you use in your code. – Richard Hazelden Jun 16 '18 at 17:39
Also its probably important to note that datastore on GAE deployed VS connection to your local datastore "simulator" do have slight differences, nothing too major, I haven't tested it but it is very possible that on your local, with or without the index.yaml, your queries would work fine, and only in production you would experience the issue that you are having. – Richard Hazelden Jun 16 '18 at 17:43
You're right. From localhost all is working fine. I've tried with @QuestionAndAnswer instruction ("gcloud app deploy index.yaml"), but it is still not working. I don't know what can I try... – adlagar Jun 18 '18 at 11:31
That is very strange, so you didn't delete your index.yaml? maybe as a test, change the filter in your query, run it on local, and check that the index.yaml file is auto updating, by using a different query ( different checks and filters ) this would be a good place to continue debugging, Is the index.yaml actually updating correctly based on what you are using in your code. – Richard Hazelden Jun 19 '18 at 12:16
Another suggestion would be to try get the data from NDB like using this syntax : Message.query(Message.city == city).fetch() Thats how we run queries and the auto updates to index.yaml work perfect, it might be worth testing it like that, seeing if that creates the index.yaml entry and then changing it back to how your code looks now, this is just a shot in the dark, but as im not sure how the auto updating of the undex.yaml works, would be worth a try. – Richard Hazelden Jun 19 '18 at 12:18
I will try that, thanks so much for the suggestion. At the moment, it is still not working. – adlagar Jun 22 '18 at 09:48
Just wanted to follow up to find out if you ever managed to figure out what the problem was ? – Richard Hazelden Jun 28 '18 at 12:36

no matching index found - GAE

2 Answers2