I want to search a specific word in my custom google search engine and then filter the output of that search to get the headline

Question

I can return the output but can not filter the return data using python 3.x

it returns a bunch of result with a unique style and i just only want to get htmlSpinnet and htmlTitle values from the result

from googleapiclient.discovery import build
import pprint

my_api_key = "xxx"
my_cse_id = "xxx"


def google_search(search_term, api_key, cse_id, **kwargs):
    service = build("customsearch", "v1", developerKey=api_key)
    res = service.cse().list(q=search_term, cx=cse_id, **kwargs).execute()
    return res['items']


results = google_search(
    'mehkeme', my_api_key, my_cse_id, num=10)

# this is the htmlSpinnets and also htmlTitle
newDict = dict()
# Iterate over all the items in dictionary and filter items which has even keys
for (key, value) in results.items():
    if key == 'htmlSpinnet':
        newDict[key] = value

print('Filtered Dictionary : ')
print(newDict)

# for result in results:
#     pprint.pprint(result)

it returns this error

/Users/valizadavali/PycharmProjects/webScrape/venv/bin/python /Users/valizadavali/PycharmProjects/webScrape/googleCustomSearch.py
Traceback (most recent call last):
  File "/Users/valizadavali/PycharmProjects/webScrape/googleCustomSearch.py", line 20, in <module>
    for (key, value) in results.items():
AttributeError: 'list' object has no attribute 'items'

it returns this without filtering, and I need to get the values which are bolded

{'cacheId': 'fGQCNF9pc6cJ', 'displayLink': 'azvision.az', 'formattedUrl': 'https://azvision.az/.../mehkeme-huquq-sisteminde-islahatlar-derinlesdirilir-- ' 'ferman--.html', 'htmlFormattedUrl': 'https://azvision.az/.../mehkeme-huquq-sisteminde-islahatlar-derinlesdirilir-- ' 'ferman--.html', 'htmlSnippet': '3 Apr 2019 ... Prezident İlham Əliyev məhkəmə-hüquq ' 'sistemində islahatların dərinləşdirilməsi
\n' 'haqqında fərman imzalayıb.', 'htmlTitle': 'Məhkəmə-hüquq sistemində islahatlar dərinləşdirilir -', 'kind': 'customsearch#result', 'link': 'https://azvision.az/news/174505/mehkeme-huquq-sisteminde-islahatlar-derinlesdirilir--ferman--.html', }

Where does `Dict` class come from? Did you import it? If you copy this code from somewhere else, can you provide a link? — ExplodingGayFish, Sep 29 '19 at 08:28
Your `results` variable is a `list` so you can not iterate it like a `dict`. Can you print it out? — ExplodingGayFish, Sep 29 '19 at 10:31
And by the way you should not post your API_KEY and ID to public site like this (if they are legit in this case), someone might take advantage of that. — ExplodingGayFish, Sep 29 '19 at 10:33
thanks for advice and i make it like that, and i share the result withour i am filtering(please look at the question i update it) — Vali Valizada, Sep 29 '19 at 10:57
Hmm the stuff you printed out does indeed look like a `dict` but you stil have the `AttributeError: 'list' object has no attribute 'items'` error? — ExplodingGayFish, Sep 29 '19 at 11:35
If you manage to get a `dict` object like the printed stuff then you can extract the information you want by replacing `if key == 'htmlSpinnet':` (line 21) with `if key in '[htmlSpinnet','link','htmlTitle']:` — ExplodingGayFish, Sep 29 '19 at 11:37
same error happening, let's say results=googlesearh is not a dictionary how can i filter then? — Vali Valizada, Sep 29 '19 at 12:33
If `results` is a list of dictionary then you can use another loop to get the result (or just use `results[0]` if there is only 1 element :D), else I may need you to print `type(results)` to help you resolve this — ExplodingGayFish, Sep 29 '19 at 15:01
it returns that it is a list. Is it means list of dictionaries? — Vali Valizada, Sep 29 '19 at 15:21
it is a list of dictionaries, so can you help me to solve this problem? — Vali Valizada, Sep 29 '19 at 15:57
Have you try replacing `for (key, value) in results.items():` with `for (key, value) in results[0].items():`? — ExplodingGayFish, Sep 29 '19 at 16:03
it works but there is an issue that it only returns one filtered dictionary not the other ones — Vali Valizada, Sep 29 '19 at 16:43
Nice, now to get all the stuff create a new list and use a loop to append each newDict to it, like this: https://pastebin.com/0MHYhisz — ExplodingGayFish, Sep 29 '19 at 16:54
Traceback (most recent call last): File "/Users/valizadavali/PycharmProjects/webScrap/googleCustomSearch.py", line 38, in for (key, value) in results.items(): AttributeError: 'list' object has no attribute 'items' — Vali Valizada, Sep 29 '19 at 17:05
oh my bad, replace `for (key, value) in results.items()`: with `for (key, value) in result.items():` (remove the `s` in `results`) — ExplodingGayFish, Sep 29 '19 at 17:08
woow it works thanks you, is there any way to instert that data to csv file? — Vali Valizada, Sep 29 '19 at 17:10
Yes very simple, you can see some example here https://stackoverflow.com/questions/3086973/how-do-i-convert-this-list-of-dictionaries-to-a-csv-file — ExplodingGayFish, Sep 29 '19 at 17:12

score 1 · Answer 1 · answered Oct 09 '19 at 09:32

1

You can also change your request to google to ask specifically for 'htmlSnippet' and 'htmlTitle' only by using 'fields' parameter:

...&fields=items(htmlTitle,htmlSnippet)...

That might make the returned results easier to parse?

See this link for more info: fields

answered Oct 09 '19 at 09:32

pipja

181
12

i will try this out, could you please look at my unasnwered question?https://stackoverflow.com/questions/58218685/using-http-request-get-output-of-each-manually-search-automatically – Vali Valizada Oct 09 '19 at 10:25
1

I'm not really well versed in python so can't really help you, I just spotted that you only want 2 properties out of the search results so I suggested the solution (which we are using extensively in production and works perfectly). – pipja Oct 09 '19 at 23:13
thanks anyway, what about searching multiple keywords, like we are creating searhing list and for each of them it returns outputs for each search as a dicttionaries of list. did you get what i mean? – Vali Valizada Oct 10 '19 at 04:32
didn't quite get you there, if you have multiple words then maybe you could run 1 api query per keywords then aggregate the results into a dictionary? – pipja Oct 11 '19 at 00:53

I want to search a specific word in my custom google search engine and then filter the output of that search to get the headline

1 Answers1