How to Use Google Custom Search Engine API in Python?

Google is the >most popular search engine, and compared to other search engines, it is fast and returns the best results based on the search query.

In this tutorial, we will discuss how to use Google Custom Search Engine API in Python.

Let's say you want to fetch the results shown by the Google Search engine for the search query "Python." To do so, you have the option of web scraping. However, web scraping is not efficient.

Moreover, it is slow, and for some reason, Google also might block your request considering that you are a bot. So Python web scraping for fetching Google Search results is a big no!

So what other option do we have? Google itself provides the Custom Search Engine ( CSE ) API, which can be used to search results from the entire web or some specific websites.

Here, in this Python tutorial, we will walk you through the Google Custom Search Engine API, writing a Python program to interact with Google CSE, and fetching Google Search results for TechGeekBuzz.

How to Use Google Custom Search Engine API in Python?

First of all, we will be setting the Google Custom Search Engine (CSE), and here our main objective will be to create the API Key.

Next, we will set the search option to the specific website, which, in this case, is techgeekbuzz.com. As here we are using a Google API, you must have a Gmail ID for the Google Developer Console.

Set Up the Google CSE API for Python

Step 1 - Create a New Project

Visit the Google Developer Console Dashboard and log in with your Gmail ID. Once you do that, you will see a Dashboard similar to the one shown below. Here, you need to create a New Project.

Click on the My First Project, then click on New Project .

Give a name to your Project. For this tutorial, we are creating a Project with the name, MyNewProject .

Step II - Set Up the API

After creating the project, select it and search for the Custom Search API using the Dashboard Search Bar, and click on the first option.

Now, click on the ENABLE button, and this will enable the Custom Search API for our current project.

After Enabling the API, you need to go to Credentials . There, click on CREATE CREDENTIALS , and select the Help me choose option.

Next, click on the API key link to create a new API key.

Now, select API restrictions . Select Custom Search API, and hit the Create button.

It will create a New API for you with a key. Copy the key in your clipboard because we will be using it in the Python program.

Step III - Create a Custom Search Engine

Now we are done with setting the API. Next, we need to create a Custom Search Engine, and to create a Custom Search Engine, we need to visit https://cse.google.com/cse/all . Now, click on the Add button.

After clicking the ADD button, we need to specify the Sites to Search option. As for this tutorial, we will be searching for the posts on techgeekbuzz.com. Therefore, in the Site to search column, we will be filling in our website address.

Also, we will be giving our custom search engine the name of MyFirstCustomEngine. After filling in the Name and Sites to search, click the create button.

This will create a Custom Search Engine for you. You will get a screen similar to this:

From this newly created Custom Search Engine, we need its Search Engine ID . To get the same, visit https://cse.google.com/cse/all , and click on the newly created Custom Search Engine. Look for the Search Engine ID, and copy it on the clipboard because we will be using it in the Python program.

Now we are all done with setting up and creating the Google Custom Search Engine API. Next, we can write the Python script to use Google Custom Search Engine API, but before that, let's install the required library.

Install the Required Library

When it comes to Google APIs, we need to install a Python library provided by Google to interact with Google APIs. Run the following Python pip install command to install the google-api-python-client library:

pip install google-api-python-client

Now we are all set to use the Google Custom Search Engine with Python. Next, open your best Python IDE or text editor and start coding.

Create Google Custom Search Engine With Python

Let's start with importing the required modules and credentials, API_KEY , and Search Engine ID .

#google api modules
from apiclient.discovery import build

#credentials 
api_key = "AIzaSyCkaDGV3Ca_jndjdhbswMgPE3dMuciuYKw" 
search_engine_id= "42nskjsh426jsn63d2" #here put your own credential

Now, we need to create a Resource object, which will send the request to the Custom Search Engine. To build a resource, we will use the build() model and call the cse() method on it.

#build resouce
resource = build("customsearch", "v1", developerKey=api_key).cse()

The build module accepts three mandatory parameters. Here, we are using the Custom Search Engine, and that's why we specify the customsearch service with version v1 and its API key.

Now, we need to create a request and execute it with the resource object. The request will contain the query q and the Search Engine ID cx . After creating the query, we need to execute it using the execute() method.

#create a request
request =resource.list(q="Python", cx=search_engine_id)

#execute request
result = request.execute()

Normally, the web results are sent and received in JSON format, but the execute() method will return the result in a dictionary format, which is a similar data structure to JSON. The result would be a dictionary, where the items key holds all the values for all the search results. We can print len of result["items"] to see the total number of results found on the first page.

print("Total",len(result['items']), "results found on 1st page")

By default, the Custom Search Engine shows only ten results per page. Now, we can loop through the items and grab the link, title, and description for all the result items.

for item in result["items"]:
    print("|-----------------------------------------|")
    print("Title:", item["title"])   #access title
    print("Link:", item["link"])     #access link
    print("Description:",item["snippet"])    #access description
    print("|-----------------------------------------|\n")

Finally, put all the code together and execute.

#A Python Program to Use Google Custom Search Engine API

#google api modules
from apiclient.discovery import build

#credentials
api_key = "AIzaSyCkaDGV3Ca_jndjdhbswMgPE3dMuciuYKw"
search_engine_id= "42nskjsh426jsn63d2" #here put your own credential

#build resouce
resource = build("customsearch", "v1", developerKey=api_key).cse()

#create a request
request =resource.list(q="Python", cx=search_engine_id)

#execute request
result = request.execute()

print("Total",len(result['items']), "results found on 1st page")
print("---------------------------")

for item in result["items"]:
    print("|-----------------------------------------|")
    print("Title:", item["title"])   #access title
    print("Link:", item["link"])     #access link
    print("Description:",item["snippet"])    #access description
    print("|-----------------------------------------|\n")

Output

Total 10 results found on 1st page
---------------------------
|-----------------------------------------|
Title: Python Global, Local and Nonlocal Variables - TechGeekBuzz
Link: https://www.techgeekbuzz.com/python-global-local-nonlocal-variables/
Description: Nov 1, 2020 ... Python Global Variables. All those variables in python which are declared 
outside the function in a global scope are known asa global variable. In ...
|-----------------------------------------|

|-----------------------------------------|
Title: Python Lambda Function - TechGeekBuzz
Link: https://www.techgeekbuzz.com/python-lambda-function/
Description: Nov 1, 2020 ... Python know as lambda function aka anonymous function. we will discuss how 
and where to use it. so let's discuss Python Lambda Function.
|-----------------------------------------|

|-----------------------------------------|
Title: Python Nested Dictionary
Link: https://www.techgeekbuzz.com/python-nested-dictionary/
Description: Feb 23, 2020 ... In this tutorial we will discuss nested dictionaries in python, we will learn how can 
we create and excess nested dictionaries and its elements.
|-----------------------------------------|

|-----------------------------------------|
Title: Python Variables, Constants & Literals - TechGeekBuzz
Link: https://www.techgeekbuzz.com/python-variables-constants-literals/
Description: Oct 31, 2020 ... In this tutorial, we are going to learn what are the Python variables, Constants & 
Literals with their use. In Python, variables are those.
|-----------------------------------------|

|-----------------------------------------|
Title: Best Way to Learn Python | Tips for Learning Python Programming
Link: https://www.techgeekbuzz.com/best-way-to-learn-python/
Description: May 16, 2020 ... Readability counts. Python's syntax is relatively simple and unique than other 
languages which can be learned easily. Python is used for server- ...
|-----------------------------------------|

|-----------------------------------------|
Title: Learn Python Programming Language - TechGekkBuzz
Link: https://www.techgeekbuzz.com/learn-python-programming/
Description: Nov 3, 2020 ... Python is slower than other programming languages such as C++ and Java, but 
still, it is the number one choice for the Data Science and ...
|-----------------------------------------|

|-----------------------------------------|
Title: Python if, if...else, if...elif...else & Nested if Statement
Link: https://www.techgeekbuzz.com/python-if-elif-else-statements/
Description: Nov 1, 2020 ... we are learn the concept of Flow Control and how to use the if, elif and else 
statement statements. so let's discuss Python If else elif.
|-----------------------------------------|

|-----------------------------------------|
Title: Best Python Compiler You Need to Know In 2020 [Ranked]
Link: https://www.techgeekbuzz.com/best-python-compiler/
Description: May 23, 2020 ... Though by default, python termed as an interpreter, there are many software on 
the internet, which are known as Python compilers.
|-----------------------------------------|

|-----------------------------------------|
Title: 5 Best Python Interpreter You Need to Know in 2020 [Updated]
Link: https://www.techgeekbuzz.com/best-python-interpreter/
Description: Nov 20, 2020 ... Computer can translate and execute the python high-level code. When we install 
python in our system, we apparently install the python ...
|-----------------------------------------|

|-----------------------------------------|
Title: Python Operators: Arithmetic, Comparison, Logical and more.
Link: https://www.techgeekbuzz.com/python-operators/
Description: Dec 12, 2020 ... From web development to artificial intelligence Python is everywhere. As like 
other programming languages, Python also has many operators.
|-----------------------------------------|

From the above output, you can see that all the search results are from techgeekbuzz.com for the Python query.

Conclusion

In this Python tutorial, you learned how to set up Custom Search Engine API on the Google API console, how to create a Custom Search Engine, and how to write a Python program to interact with the Google Custom Search Engine API. To Interact with the Google CSE, you need the CSE API and the Custom Search Engine ID, and you all are good to go. While creating the API, do not share the API key or ID with anyone.

We have shared the key and ID to explain this tutorial. After writing this tutorial, we deleted the API key and ID for obvious security reasons.

People are also reading:

• Delete Emails in Python
• Python Numpy Array Tutorial
• Numpy Dot Product
• Python Check If File or Directory Exists
• Online Python Compiler
• Python Developer Salary
• 10 Python Tips and Tricks
• Python Assert Statement
• Shallow Copy and Deep Copy in Python
• Python List Comprehension