header_pic
Back to blog home

Python Function to Return up to 100 Google Custom Search Results

by Adam on 2021-02-28

Below is a Python function I wrote as an extension of the function in the 4th step of this answer https://stackoverflow.com/a/37084643/42346 to return up to 100 results from the Google Search API. This is also posted as an SO answer here: https://stackoverflow.com/a/66404428/42346 It increases the start parameter by 10 for each API call, handling the number of results to return automatically. For example, if you request 25 results the function will induce 3 API calls of: 10 results, 10 results, and 5 results.

Background information:
For instructions on how to set-up a Google Custom Search engine: https://stackoverflow.com/a/37084643/42346
More detail about how to specify that it search the entire web here: https://stackoverflow.com/a/11206266/42346

from googleapiclient.discovery import build
from pprint import pprint as pp
import math

def google_search(search_term, api_key, cse_id, **kwargs):
    service = build("customsearch", "v1", developerKey=api_key)

    num_search_results = kwargs['num']
    if num_search_results > 100:
        raise NotImplementedError('Google Custom Search API supports max of 100 results')
    elif num_search_results > 10:
        kwargs['num'] = 10 # this cannot be > 10 in API call 
        calls_to_make = math.ceil(num_search_results / 10)
    else:
        calls_to_make = 1

    kwargs['start'] = start_item = 1
    items_to_return = []
    while calls_to_make > 0:
        res = service.cse().list(q=search_term, cx=cse_id, **kwargs).execute()
        items_to_return.extend(res['items'])
        calls_to_make -= 1
        start_item += 10
        kwargs['start'] = start_item
        leftover = num_search_results - start_item + 1
        if 0 < leftover < 10:
            kwargs['num'] = leftover

    return items_to_return 

And here's an example of how you'd call that:

NUM_RESULTS = 25
MY_SEARCH = 'why do cats chase their own tails'
MY_API_KEY = 'Google API key'
MY_CSE_ID = 'Custom Search Engine ID'

results = google_search(MY_SEARCH, MY_API_KEY, MY_CSE_ID, num=NUM_RESULTS)

for result in results:
    pp(result)


tags: python google-custom-search

Post thanked 0 time(s).

Related posts: