Google is the >most popular search engine, and compared to other search engines, it is fast and returns the best results based on the search query.
In this tutorial, we will discuss how to use Google Custom Search Engine API in Python.
Let's say you want to fetch the results shown by the Google Search engine for the search query "Python." To do so, you have the option of web scraping. However, web scraping is not efficient.
Moreover, it is slow, and for some reason, Google also might block your request considering that you are a bot. So Python web scraping for fetching Google Search results is a big no!
So what other option do we have? Google itself provides the Custom Search Engine ( CSE ) API, which can be used to search results from the entire web or some specific websites.
Here, in this Python tutorial, we will walk you through the Google Custom Search Engine API, writing a Python program to interact with Google CSE, and fetching Google Search results for TechGeekBuzz.
How to Use Google Custom Search Engine API in Python?
First of all, we will be setting the Google Custom Search Engine (CSE), and here our main objective will be to create the API Key.
Next, we will set the search option to the specific website, which, in this case, is techgeekbuzz.com. As here we are using a Google API, you must have a Gmail ID for the Google Developer Console.
Set Up the Google CSE API for Python
Step 1 - Create a New Project
Visit the Google Developer Console Dashboard and log in with your Gmail ID. Once you do that, you will see a Dashboard similar to the one shown below. Here, you need to create a New Project.
Click on the My First Project, then click on New Project .
Give a name to your Project. For this tutorial, we are creating a Project with the name, MyNewProject .
Step II - Set Up the API
After creating the project, select it and search for the Custom Search API using the Dashboard Search Bar, and click on the first option.
Now, click on the ENABLE button, and this will enable the Custom Search API for our current project.
After Enabling the API, you need to go to Credentials . There, click on CREATE CREDENTIALS , and select the Help me choose option.
Next, click on the API key link to create a new API key.
Now, select API restrictions . Select Custom Search API, and hit the Create button.
It will create a New API for you with a key. Copy the key in your clipboard because we will be using it in the Python program.
Step III - Create a Custom Search Engine
Now we are done with setting the API. Next, we need to create a Custom Search Engine, and to create a Custom Search Engine, we need to visit https://cse.google.com/cse/all . Now, click on the Add button.
After clicking the ADD button, we need to specify the Sites to Search option. As for this tutorial, we will be searching for the posts on techgeekbuzz.com. Therefore, in the Site to search column, we will be filling in our website address.
Also, we will be giving our custom search engine the name of MyFirstCustomEngine. After filling in the Name and Sites to search, click the create button.
This will create a Custom Search Engine for you. You will get a screen similar to this:
From this newly created Custom Search Engine, we need its Search Engine ID . To get the same, visit https://cse.google.com/cse/all , and click on the newly created Custom Search Engine. Look for the Search Engine ID, and copy it on the clipboard because we will be using it in the Python program.
Now we are all done with setting up and creating the Google Custom Search Engine API. Next, we can write the Python script to use Google Custom Search Engine API, but before that, let's install the required library.
Install the Required Library
When it comes to Google APIs, we need to install a Python library provided by Google to interact with Google APIs. Run the following Python pip install command to install the
google-api-python-client
library:
pip install google-api-python-client
Now we are all set to use the Google Custom Search Engine with Python. Next, open your best Python IDE or text editor and start coding.
Create Google Custom Search Engine With Python
Let's start with importing the required modules and credentials, API_KEY , and Search Engine ID .
#google api modules
from apiclient.discovery import build
#credentials
api_key = "AIzaSyCkaDGV3Ca_jndjdhbswMgPE3dMuciuYKw"
search_engine_id= "42nskjsh426jsn63d2" #here put your own credential
Now, we need to create a Resource object, which will send the request to the Custom Search Engine. To build a resource, we will use the
build()
model and call the
cse()
method on it.
#build resouce
resource = build("customsearch", "v1", developerKey=api_key).cse()
The
build
module accepts three mandatory parameters. Here, we are using the Custom Search Engine, and that's why we specify the
customsearch
service with version
v1
and its API key.
Now, we need to create a request and execute it with the
resource
object. The request will contain the query
q
and the Search Engine ID
cx
. After creating the query, we need to execute it using the
execute()
method.
#create a request
request =resource.list(q="Python", cx=search_engine_id)
#execute request
result = request.execute()
Normally, the web results are sent and received in JSON format, but the execute() method will return the result in a dictionary format, which is a similar data structure to JSON. The result would be a dictionary, where the
items
key holds all the values for all the search results. We can print len of
result["items"]
to see the total number of results found on the first page.
print("Total",len(result['items']), "results found on 1st page")
By default, the Custom Search Engine shows only ten results per page. Now, we can loop through the items and grab the link, title, and description for all the result items.
for item in result["items"]:
print("|-----------------------------------------|")
print("Title:", item["title"]) #access title
print("Link:", item["link"]) #access link
print("Description:",item["snippet"]) #access description
print("|-----------------------------------------|\n")
Finally, put all the code together and execute.
#A Python Program to Use Google Custom Search Engine API
#google api modules
from apiclient.discovery import build
#credentials
api_key = "AIzaSyCkaDGV3Ca_jndjdhbswMgPE3dMuciuYKw"
search_engine_id= "42nskjsh426jsn63d2" #here put your own credential
#build resouce
resource = build("customsearch", "v1", developerKey=api_key).cse()
#create a request
request =resource.list(q="Python", cx=search_engine_id)
#execute request
result = request.execute()
print("Total",len(result['items']), "results found on 1st page")
print("---------------------------")
for item in result["items"]:
print("|-----------------------------------------|")
print("Title:", item["title"]) #access title
print("Link:", item["link"]) #access link
print("Description:",item["snippet"]) #access description
print("|-----------------------------------------|\n")
Output
Total 10 results found on 1st page
---------------------------
|-----------------------------------------|
Title: Python Global, Local and Nonlocal Variables - TechGeekBuzz
Link: https://www.techgeekbuzz.com/python-global-local-nonlocal-variables/
Description: Nov 1, 2020 ... Python Global Variables. All those variables in python which are declared
outside the function in a global scope are known asa global variable. In ...
|-----------------------------------------|
|-----------------------------------------|
Title: Python Lambda Function - TechGeekBuzz
Link: https://www.techgeekbuzz.com/python-lambda-function/
Description: Nov 1, 2020 ... Python know as lambda function aka anonymous function. we will discuss how
and where to use it. so let's discuss Python Lambda Function.
|-----------------------------------------|
|-----------------------------------------|
Title: Python Nested Dictionary
Link: https://www.techgeekbuzz.com/python-nested-dictionary/
Description: Feb 23, 2020 ... In this tutorial we will discuss nested dictionaries in python, we will learn how can
we create and excess nested dictionaries and its elements.
|-----------------------------------------|
|-----------------------------------------|
Title: Python Variables, Constants & Literals - TechGeekBuzz
Link: https://www.techgeekbuzz.com/python-variables-constants-literals/
Description: Oct 31, 2020 ... In this tutorial, we are going to learn what are the Python variables, Constants &
Literals with their use. In Python, variables are those.
|-----------------------------------------|
|-----------------------------------------|
Title: Best Way to Learn Python | Tips for Learning Python Programming
Link: https://www.techgeekbuzz.com/best-way-to-learn-python/
Description: May 16, 2020 ... Readability counts. Python's syntax is relatively simple and unique than other
languages which can be learned easily. Python is used for server- ...
|-----------------------------------------|
|-----------------------------------------|
Title: Learn Python Programming Language - TechGekkBuzz
Link: https://www.techgeekbuzz.com/learn-python-programming/
Description: Nov 3, 2020 ... Python is slower than other programming languages such as C++ and Java, but
still, it is the number one choice for the Data Science and ...
|-----------------------------------------|
|-----------------------------------------|
Title: Python if, if...else, if...elif...else & Nested if Statement
Link: https://www.techgeekbuzz.com/python-if-elif-else-statements/
Description: Nov 1, 2020 ... we are learn the concept of Flow Control and how to use the if, elif and else
statement statements. so let's discuss Python If else elif.
|-----------------------------------------|
|-----------------------------------------|
Title: Best Python Compiler You Need to Know In 2020 [Ranked]
Link: https://www.techgeekbuzz.com/best-python-compiler/
Description: May 23, 2020 ... Though by default, python termed as an interpreter, there are many software on
the internet, which are known as Python compilers.
|-----------------------------------------|
|-----------------------------------------|
Title: 5 Best Python Interpreter You Need to Know in 2020 [Updated]
Link: https://www.techgeekbuzz.com/best-python-interpreter/
Description: Nov 20, 2020 ... Computer can translate and execute the python high-level code. When we install
python in our system, we apparently install the python ...
|-----------------------------------------|
|-----------------------------------------|
Title: Python Operators: Arithmetic, Comparison, Logical and more.
Link: https://www.techgeekbuzz.com/python-operators/
Description: Dec 12, 2020 ... From web development to artificial intelligence Python is everywhere. As like
other programming languages, Python also has many operators.
|-----------------------------------------|
From the above output, you can see that all the search results are from techgeekbuzz.com for the Python query.
Conclusion
In this Python tutorial, you learned how to set up Custom Search Engine API on the Google API console, how to create a Custom Search Engine, and how to write a Python program to interact with the Google Custom Search Engine API. To Interact with the Google CSE, you need the CSE API and the Custom Search Engine ID, and you all are good to go. While creating the API, do not share the API key or ID with anyone.
We have shared the key and ID to explain this tutorial. After writing this tutorial, we deleted the API key and ID for obvious security reasons.
People are also reading:
Leave a Comment on this Post