Tutorials

plot route on Google Maps with Python Google Maps API, optimize route with Google Maps API, Singapore

Plot Route on Google Maps with Python

Introduction

You probably use Google Maps a lot in your daily life, such as locate a popular restaurant, check the distance between one place to another, or find the nearest driving route and shortest travelling time etc. If you have been thrown with a big set of location data where you need check and validate them from a map, and then find the optimal routes, you probably would think how this can be done programmatically in Python. In this article, we will be exploring how all these can be achieved with Python Google Maps APIs.

Prerequisites

To be able to use Google Maps APIs, you will need to create a Google Cloud Service account and set up a billing method. If you are a new signed up user, you would get $300 credit for you to try out all the various Google APIs. You can follow the official docs to set up a new project, and from where you would get an API key for using its services.

You will also need to install the GoogleMaps package in your working environment, below is the pip command to install the package:

pip install -U googlemaps

Now let’s import the package at the beginning of our code, and initialize the service with your API Key:

import googlemaps

gmaps = googlemaps.Client(key='YOUR API KEY')

If you did not get any error up to this step, you are all set to go. Let’s start to explore the various function calls you can do with this package.

Get Geolocation of the Addresses

Often you need to check a location by address, postal code or even the name of the store/company, this can be done via the geocode function which is similar to search a place in Google Maps from your browser. The function returns a list of possible locations with the detailed address info such as the formatted address, country, region, street, lat/lng etc.

Below are a few possible address info you can pass to this API call:

#short form of address, such as country + postal code
geocode_result = gmaps.geocode('singapore 018956')

#full address
geocode_result = gmaps.geocode("10 Bayfront Ave, Singapore 018956")

#a place name
geocode_result = gmaps.geocode("zhongshan park")

#Chinese characters
geocode_result = gmaps.geocode('滨海湾花园')

#place name/restaurant name
geocode_result = gmaps.geocode('jumbo seafood east coast')

print(geocode_result[0]["formatted_address"]) 
print(geocode_result[0]["geometry"]["location"]["lat"]) 
print(geocode_result[0]["geometry"]["location"]["lng"])

Depends how complete the information you have supplied to the function call, you may get some sample output as per below:

1206 ECP, #01-07/08 East Coast Seafood Centre, Singapore 449883
1.3051669,
103.930673

Reverse Geocoding

You can also use the latitude and longitude to check the address information, this is called reverse geocoding. For instance, the below will give you the human readable addresses for the given lat/lng:

reverse_geocode_result = gmaps.reverse_geocode((1.3550021,103.7084641))

print(reverse_geocode_result[0]["formatted_address"])
#'87 Farrer Dr, Singapore 259287'

You may get multiple address info with the same lat/lng as Google just return the list of addresses closest to this lat/lng.

Check the Distance Between Locations

To check the distance between multiple locations, you can use the Google distance matrix API, where you can supply multiple locations, and Google will return the travelling distance between each location as well as the travelling time. E.g.:

from datetime import datetime, timedelta

gmaps.distance_matrix(origins=geocode_result[0]['formatted_address'], 
                      destinations=reverse_geocode_result[0]["formatted_address"], 
                      departure_time=datetime.now() + timedelta(minutes=10))

In above example, we have provided the departure time as 10 minutes later from current time for Google to calculate the travelling time. The departure time cannot be any time in the past.

The return JSON object would include the travelling distance and time based on the current traffic condition (default transport mode is driving):

{'destination_addresses': ['87 Farrer Dr, Singapore 259287'],
 'origin_addresses': ['1206 ECP, #01-07/08 East Coast Seafood Centre, Singapore 449883'],
 'rows': [{'elements': [{'distance': {'text': '22.2 km', 'value': 22219},
     'duration': {'text': '24 mins', 'value': 1442},
     'duration_in_traffic': {'text': '22 mins', 'value': 1328},
     'status': 'OK'}]}],
 'status': 'OK'}

Get Directions Between Locations

One of the most frequent usage of Google Maps is to check the direction from one place to another. You can use directions API to get the route info as per below:

directions_result = gmaps.directions(geocode_result[0]['formatted_address'],
                                     reverse_geocode_result[0]["formatted_address"],
                                     mode="transit",
                                     arrival_time=datetime.now() + timedelta(minutes=0.5))

For this directions API, it returns you the detailed routing information based on what type of travelling mode you’ve chosen.

In the above example, we have specified the mode as “transit”, so Google Maps will suggest a route that you can take the public transport whenever possible to reach your final destination. It may not meet your arrival time if it is really infeasible anyway.

Below is the sample return with detailed routing directions:

[{'bounds': {'northeast': {'lat': 1.3229677, 'lng': 103.9314612},
   'southwest': {'lat': 1.2925606, 'lng': 103.8056495}},
  'copyrights': 'Map data ©2021 Google',
  'legs': [{'arrival_time': {'text': '9:16pm',
     'time_zone': 'Asia/Singapore',
     'value': 1611321373},
    'departure_time': {'text': '7:59pm',
     'time_zone': 'Asia/Singapore',
     'value': 1611316750},
    'distance': {'text': '18.0 km', 'value': 17992},
    'duration': {'text': '1 hour 17 mins', 'value': 4623},
    'end_address': '87 Farrer Dr, Singapore 259287',
    'end_location': {'lat': 1.3132547, 'lng': 103.8070619},
    'start_address': '1206 ECP, #01-07/08 East Coast Seafood Centre, Singapore 449883',

...

{'distance': {'text': '0.1 km', 'value': 106},
        'duration': {'text': '1 min', 'value': 76},
        'end_location': {'lat': 1.305934, 'lng': 103.9306822},
        'html_instructions': 'Turn <b>left</b>',
        'maneuver': 'turn-left',
        'polyline': {'points': 'q{}Fk_jyRM?KAIAE@a@DQBGDEBGDIF_@LQD'},
        'start_location': {'lat': 1.3050507, 'lng': 103.9309369},
        'travel_mode': 'WALKING'},
...
}]

You can also supply the waypoints parameter in order to route multiple locations between your origin and destination. For instance, if you want to go for a one day tour in Singapore, you can provide a list of the attractions and let Google to optimize the route for you with the optimize_waypoints = True parameter. The sample code as per below:

waypoints = ["Chinatown Buddha Tooth Relic Temple", 
"Sentosa Island, Singapore", 
"National Gallery Singapore", 
"Botanic Garden, Singapore",
"Boat Quay @ Bonham Street, Singapore 049782"]

results = gmaps.directions(origin = "Fort Canning Park, Singapore",
                                         destination = "Raffles Hotel, Singapore",                                     
                                         waypoints = waypoints,
                                         optimize_waypoints = True,
                                         departure_time=datetime.now() + timedelta(hours=1)

for i, leg in enumerate(results[0]["legs"]):
    print("Stop:" + str(i),
        leg["start_address"], 
        "==> ",
        leg["end_address"], 
        "distance: ",  
        leg["distance"]["value"], 
        "traveling Time: ",
        leg["duration"]["value"]
    )

To get a good result, you will need to make sure all the locations you’ve provided can be geocoded by Google. Below is the output:

plot route on Google Maps with Python Google Maps API

Plot Route on Google Maps

To visualize your route or location on a map, you can make use of the Maps Static API. It allows you to plot your locations as markers on the map and draw the path between each location.

To get started, we shall define the markers for our locations. We can specify the color, size and label attributes for each location in a “|” separated string. As the label only allows a single character from {A-Z, 0-9}, we will just use A-Z to indicate the sequence of each location. E.g.:

locations = ["Fort Canning Park, Singapore",
          "Chinatown Buddha Tooth Relic Temple", 
          "Sentosa Island, Singapore", 
          "National Gallery Singapore", 
          "Boat Quay @ Bonham Street, Singapore 049782",
          "Botanic Garden, Singapore",
          "Raffles Hotel, Singapore"]

markers = ["color:blue|size:mid|label:" + chr(65+i) + "|" 
                   + r for i, r in enumerate(locations)]

In the static_map function, you can specify the scale, size and zoom to define how many pixels of your output and whether you want to show the details up to the city or individual building on your map.

The format and maptype parameters are used to specify the output image format and what type of maps you want to use.

Lastly, we can specify the path parameter to connect the different locations together. Similar to how we define the markers, we can specify the attributes in a “|” separated string. Below is the sample code:

result_map = gmaps.static_map(
                 center=routes[0],
                 scale=2, 
                 zoom=12,
                 size=[640, 640], 
                 format="jpg", 
                 maptype="roadmap",
                 markers=markers,
                 path="color:0x0000ff|weight:2|" + "|".join(locations))

And if you save the return result into a .jpg file and you shall see something similar to the below:

plot route on Google Maps with Python Google Maps API

The routing sequence is based on the list of locations you’ve supplied, so you can probably use directions function to return a optimal route and then plot them with static_map.

A few things to be noted are that the static map only support small size of image (up to 1280×1280), and you will have to plot in more points if you want to draw a nicer driver routes. For our above code, we only take 7 location points, so the lines are not making any sense for driving. If we use the location points suggested by Google from directions function, the result would look much better. E.g.:

marker_points = []
waypoints = []

#extract the location points from the previous directions function

for leg in results[0]["legs"]:
    leg_start_loc = leg["start_location"]
    marker_points.append(f'{leg_start_loc["lat"]},{leg_start_loc["lng"]}')
    for step in leg["steps"]:
        end_loc = step["end_location"]
        waypoints.append(f'{end_loc["lat"]},{end_loc["lng"]}')
last_stop = results[0]["legs"][-1]["end_location"]
marker_points.append(f'{last_stop["lat"]},{last_stop["lng"]}')
        
markers = [ "color:blue|size:mid|label:" + chr(65+i) + "|" 
           + r for i, r in enumerate(marker_points)]
result_map = gmaps.static_map(
                 center = waypoints[0],
                 scale=2, 
                 zoom=13,
                 size=[640, 640], 
                 format="jpg", 
                 maptype="roadmap",
                 markers=markers,
                 path="color:0x0000ff|weight:2|" + "|".join(waypoints))

We are extracting all the location points from the directions function and pass them to the path parameter to draw the connection lines. The output would be similar to below which makes more sense for driving:

plot route on Google Maps with Python Google Maps API, optimize route with Google Maps API, Singapore

Conclusion

In this article, we have reviewed through a few Google Maps APIs which hopefully will help you for any feasibility study or even in your real projects. For example, to cleanse the dirty address for a large set of data, or to calculate the distance/travelling time or get the optimal routes between multiple locations. If your objective is to generate a dynamic map, you probably have to go for the Maps JavaScript API where you can display a map on web page and make it interactive, but still you may find these Python APIs would be more efficient in processing your raw data rather than doing it in JavaScript code.

create animated charts and gif in python with pandas-alive

Create Animated Charts In Python

Introduction

If you are working as a data analyst or data scientist for some time, you may have already known how to use matplotlib to visualize and present data in various charts. The matplotlib library provides an animation module to generate dynamic charts to make your data more engaging, however it still takes you a few steps to format your data, and initialize and update the data into the charts. In this article, I will demonstrate you another Python library – pandas-alive which allows you to generate animated charts directly from pandas data without any format conversion.

Prerequisites

You can install this library via pip command as per below if you do not have it in your working environment yet:

pip install pandas-alive

It will also install its dependencies such as pandas, pillow and numpy etc.

For demonstration of our later examples, let’s grab some sample covid-19 data from internet, you can download it from here.

Before we start, we shall import all the necessary modules and do a preview of our sample data:

import pandas as pd
import pandas-alive

df_covid = pd.read_excel("covid-19 sample data.xlsx")

The data we will be working on would be something similar to the below:

create animated charts and gif in python with pandas-alive

Now with all above ready, let’s dive into the code examples.

Generate animated bar chart race

Bar chart is the most straightforward way to present data, it can be drawn in horizontal or vertical manner. Let’s do a minor formatting on our data so that we can use date as horizontal or vertical axis to present the data.

df_covid = df_covid.pivot(index="date", columns="location", values="total_cases").fillna(0)

To create an animated bar chart horizontally, you can simply call plot_animated as per below:

df_covid.plot_animated("covid-19-h-bar.gif", period_fmt="%Y-%m", title="Covid-19 Cases")

The plot_animated function has default parameters kind=”race” and orientation = “h”, hence the output gif would be generated as per below:

create animated charts and gif in python with pandas-alive

You can change the default values of these two parameters to generate a vertical bar chart race:

df_covid.plot_animated("covid-19-v-bar.gif", 
                     period_fmt="%Y-%m", 
                     title="Covid-19 Cases", 
                     orientation='v')

The output chart would be something similar to below:

create animated charts and gif in python with pandas-alive

 

Generate animated line chart

To create an animated line chart, you just need to change the parameter kind = “line” as per below:

df_covid.plot_animated("covid-19-line.gif",
                     title="Covid-19 Cases",
                     kind='line',
                     period_fmt="%Y-%m",
                     period_label={
                         'x':0.25,
                         'y':0.9,
                         'family': 'sans-serif',
                         'color':  'darkred'
                    })

There are some other parameters such as period_label to control the format of the label, or n_visible to constrain how many records to be shown on the chart. The output chart would be as per the below:

create animated charts and gif in python with pandas-alive

 

Generate animated pie chart

Similar to other charts, you can create a simple pie chart with below parameters:

df_covid.plot_animated(filename='covid-19-pie-chart.gif',
                     kind="pie",                     
                     rotatelabels=True,
                     tick_label_size=5,
                     dpi=300,
                     period_fmt="%Y-%m",
                     )

You can also use other Axes.Pie parameters to define the pie chart behavior. The output from above code would be:

 

 

create animated charts and gif in python with pandas-alive

 

 

Generate scatter chart

Generate scatter chart or bubble chart is slightly complicated than other charts, but for our sample data, it does not make much sense to visualize it in this type of charts. E.g.

df_covid.plot_animated(filename='covid-19-scatter-chart.gif',
                     kind="scatter",
                     period_label={'x':0.05,'y':0.9},
                     steps_per_period=5
                    )

You shall see the output is similar to the line chart:

create animated charts and gif in python with pandas-alive

 

Conclusion

Pandas-Alive provides very convenient ways to generate all sorts of animated charts from pandas data frame with the underlying support from the Matplotlib library. It accepts most of the parameters you used in matplotlib, so you don’t have to learn a lot of new things before applying it for your charts.

There are many more features beyond the basis I have covered in above, such as supplying custom figures, generating GeoSpatial charts or combining multiple animated charts in one view. You can check more examples from its project page.

 

generate password with python random and python secrets

You Might Have Misused Python Random

Introduction

Python random module provides a convenient way for generating pseudo-random numbers in case you need some unpredictable results for your application such as the computer games, a lucky draw system or even the shuffling logic for your music player. Since it provides various functions to generate results in “unpredictable” manner, developers attempted to use this feature to produce random password or authentication token for security purpose without understanding of it’s fundamental implementation. In this article, we will be discussing how the Python random module has been misunderstood and misused for the scenarios which it shall not be used.

Basic usage of Python random module

Let’s take a look at some basic usage of this module. You can use it to generate random integers, float numbers or bytes as per below:

#Generate a random integer between 1 to 10
random.randint(1,10) 
#2

#generate a random floating point number between 0 to 1
random.random()
#0.3103975786510934

#Generate random number between 1 to 2 in uniform distribution
random.uniform(1, 2) 
#1.9530600469459607 

#Generate random number between 1 to 100, with step as 2
random.randrange(1, 100, 2) 
#43 

#Generate random bytes, available in Python version 3.9
random.randbytes(8)
#b'v\xf7\xb2v`\xc8U]'
#Generate a integer within 8 bits
random.getrandbits(8)
#68

And shuffling or sampling the items in a sequence can be achieved easily with below:

slangs = ["Boomer", "Cap", "Basic", "Retweet", "Fit", "Fr", "Clout"]
random.shuffle(slangs)
#['Fit', 'Basic', 'Fr', 'Clout', 'Cap', 'Retweet', 'Boomer']

random.sample(slangs, k=5)
['Fit', 'Fr', 'Clout', 'Retweet', 'Basic']

You can also use the choice function to choose a random option from a given sequence, for instance:

random.choice(["Boomer", "Cap", "Basic", "Retweet", "Fit", "Fr", "Clout"])
#'Retweet'

With this function, It’s easy to implement a lucky draw logic where all the participants’ name are passed in and a winner is selected randomly which seems to be the perfect use case of it. The problem comes when developers try to use this function to generate  password or security related tokens which they believe it’s random enough and resistant to predication. But it is indeed wrong.

Why the random numbers generated are not random enough?

To answer this question, we will need to take a further look at the implementation of this Python random module. This module uses Mersenne Twister algorithm as the core generator which is designed for modelling and simulation purpose rather than security or cryptography. Some study show it’s not difficult to reconstruct the internal state of the MT to predict the outcome and it can be attacked through this MT randomness.

Actually random module does provide a SystemRandom class which uses the sources from operation system to generate random numbers without relying on the software state and the result is not reproducible. For instance, depends on the actual implementation, some OS uses the atmospheric noise or the exact time of the key presses as the source for generating unpredictable result which is more suitable for cryptography.

You can use it in the same way as the random class except the state is not available:

sys_random = random.SystemRandom()

sys_random.random()
#0.04883551877802528

sys_random.randint(1, 100)
#72

sys_random.choice(["Boomer", "Cap", "Basic", "Retweet", "Fit", "Fr", "Clout"])
#'Clout'

Unfortunately, this class has been overlooked for many years and some developers continue using the functions from random module for generating password or security tokens which exposed a lot of security vulnerability. To address these concerns, an enhancement proposal raised to add a new secrets module for some common security-related functions, so that people won’t mix up it with the random module.

Python secrets module

Let’s take a look at the functions in secrets module:

#Generate random integer between 0 to 50
secrets.randbelow(50)
#2

#Generate random integer with 8 bits
secrets.randbits(8)
#125

#Generate random bytes
secrets.token_bytes(20)
#b'\x8a\xb3\xa92)S:\xf2\xac\x90\xaf\xb1\xb3Q\xc5\xfe\x80\xdb\xc2`'

#Generate random string in hexadecimal with default 32 bytes
secrets.token_hex()
#'cf4f964edf810ca7f6ad6b533038fdcf538c73bd59e11d3340a838e7fd88fdf9'

#Generate URL-safe text string
secrets.token_urlsafe(10)
#z9zQQsxcq6SRug

import string
secrets.choice(string.ascii_letters + string.digits + string.punctuation)
#'k'

The functions are pretty similar to what we have seen in random module, just that results are generated from os.urandom function which meant to be unpredictable enough for cryptographic applications.

Implementing a strong password generator with Python secrets

With all above said, let’s implement a strong password generator with the secrets module. Assuming we have below requirements on our password:

  • Length between 8 to 16
  • At least 1 lowercase
  • At least 1 uppercase
  • At least 1 number
  • At least 1 special characters among !#$%&@_~

We can use the choice function to randomly choose 1 character each time for a random iteration between 8 to 16 times, then test if all the requirements are met for the generated password. Below the sample code:

def generate_strong_password():
    special_characters = '!#$%&@_~'
    password_choices = string.ascii_letters + string.digits + special_characters
    while True:
        password = ''.join(secrets.choice(password_choices) for _ in range(random.randint(8, 16)))
        if (any(c.islower() for c in password)
                and any(c.isupper() for c in password)
                and sum(special_characters.find(c) > -1 for c in password) == 1
                and any(c.isdigit() for c in password)):
            break
    return password

To check the randomness, you can try the below:

[generate_strong_password() for _ in range(10)]
# sample output
['C9A&PbswcLMpJ',
 'DHrbuzZU&io7Io',
 'B1zx4YqKUS@01m',
 'Q!F8tsZJsw',
 '9QVF8oASt_jceq2',
 '2~EKErrHAc9jvbyZ',
 '8ceU_XZbYdqv8b',
 'h8oS@tZ6',
 '3w@OX33tw12J6',
 'FX~mkO0aNatqp']

Conclusion

In this article, we have reviewed through the most commonly used functions in Python random module for generating pseudo-random results for modelling and simulation. Due to misuse of random module for security related applications from the earlier Python developers, Python has introduced a new secrets module to focus on password, authentication or security related tokens. We have also demonstrated an example on how to use secrets module to implement a strong password generator. Although you can generate a strong password with the combination of the alphanumeric and special characters, you shall never store your password in plaintext or simple encrypted format, a good practice is to always use libraries like hashlib or bcrypt to salt and hash it before storing.

web scraping and automate twitter post with selenium

Automate Your Tweets with Selenium

Introduction

In the previous post, we have discussed about how to start web scraping with requests and lxml libraries, and we also summarized two limitations with this approach:

  • Time & effort required to chain all the requests for some complicated operations such as user authentication
  • Triggering a button click or calling JavaScript code is not possible from the HTML response

To solve these two issues, I recommended to use selenium package. In fact you have checked this post, you may still remember that we can use selenium to simulate human actions such as open URL on browser or trigger a button click on the web page and so on.

In this post, I will demonstrate how to use selenium to automatically login to tweeter account, view and post tweets, where the same approach can be used for your web scraping project.

Prerequisites

In order to use selenium to launch browser, you will need to download a web driver for the browser you are using. You can check all the supported browsers as well as the download links from here.

For the below code example, I will use Chrome version 86 and download the driver with this version supported. For simplicity, I will save the chromedriver.exe into my current code directory.

Besides the driver file, you will also need to install selenium in your working environment. Below is the pip command for installation of the latest version:

pip install --upgrade selenium

Let’s also import all the modules at the beginning of our code. Explanation will be given later where these modules are used:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as ec

With the above ready, let’s dive into our code example.

Login to twitter account with Selenium

Similar to a human behavior on the browser, Selenium does not allow you to interact with the invisible elements, and you would encounter ElementNotVisibleException when trying to access the element if it is not fully loaded or not in the view. So the best practice is to always maximize your browser window, so that majority of the information you need are visible and interactable.

To maximize the browser upon launching, you can set –start-maximized in the chrome operations as per below:

chromeOptions = Options()
chromeOptions.add_argument("--start-maximized")

(You can also launch the browser first and later call the maximize_window function to maximize it)

This Chrome options shall be passed into the web driver constructor when it is initiated. We also need to specify the full path of driver exe file, for our case, it’s under the current directory.

driver = webdriver.Chrome(executable_path="chromedriver.exe", options=chromeOptions)

With the above code, a new Chrome browser will be launched. The web driver object has a get method which accepts a URL parameter and opens it from the browser. Below will open the twitter login page on your browser:

tweeter_url = "https://twitter.com/login"
driver.get(tweeter_url)

As there are many factors impact how fast the web page can be fully loaded, you may need to add in some delays at certain steps to make sure that the current action has been completed successfully before moving to the next step.

In Selenium, there are two types of waiting approaches: implicit wait and explicit wait. The implicit wait will just instruct web driver to wait for maximum of certain time when polling the DOM, while explicit wait will check the presence/visibility of the element periodically until the condition is met or the maximum waiting time reached. As implicit wait applies to the entire lifecycle of the web driver, the explicit wait is relatively more flexible. Let’s define our explicit wait for a max of 10 seconds in our example:

wait = WebDriverWait(driver, 10)

Now, we shall follow what we have discussed in the previous post to find a unique identifier of the login username and password fields. By inspecting the web page HTML, you can easily find out the name attribute of the username and password field. Below is the screenshot of the HTML structure for username field:

web scraping and automating twitter post with selenium

 

To locate the username element, we can use the XPath with its element name. And let’s also use the explicit wait to locate it until the element is fully loaded and visible on the page:

username_input = wait.until(ec.visibility_of_element_located((By.NAME, "session[username_or_email]")))

Once we located the username input field, we can send our login ID to this field with send_keys function as per below:

username_input.send_keys(username)

Note: you will need to replace this username/password variable with your twitter login credentials

Similarly, we can locate our password field by its name and send in our password:

password_input = wait.until(ec.visibility_of_element_located((By.NAME, "session[password]")))
password_input.send_keys(password)

Once we have successfully set the values into these two fields, we can simulate the button click on the login button:

  • Firstly we shall locate to the login button by its attribute data-testid=’LoginForm_Login_Button’
  • Then call the WebElement click function to simulate how user clicks on the button

With the below code, you shall be able to login into your tweeter account and view the tweets on your home screen:

login_button = wait.until(ec.visibility_of_element_located((By.XPATH, "//div[@data-testid='LoginForm_Login_Button']")))
login_button.click()

To showcase how to interact with your web page like a normal user, let’s move to the next example to search a tweeter posts with some keywords.

Search tweeter posts by keywords

Same as previously, we shall first locate our search input box by its data-testid attribute as per below:

search_input = wait.until(ec.visibility_of_element_located((By.XPATH, 
"//div/input[@data-testid='SearchBox_Search_Input']")))

As a normal user, I can key in some keywords in the search box and hit ENTER for a search. We can do the same from Selenium via the send_keys function. Let’s first clear the input box and then send a keyword “ethereum” together with a ENTER key:

search_input.clear()
search_input.send_keys("ethereum" + Keys.ENTER)

Upon receiving the ENTER key event, you shall see the search results are loading on the page. The next is to extract the tweeter posts from the searching results.

Below is the sample code that I extracted all the text from the tweets and printed as output:

tweet_divs = driver.find_elements_by_xpath("//div[@data-testid='tweet']")
for div in tweet_divs:
    spans = div.find_elements_by_xpath(".//div/span")
    tweets = ''.join([span.text for span in spans])
    print(tweets)

You shall see the output similar to below:

web scraping and automating twitter post with selenium

With this plain text results, you may use some text processing tools to further analyze what people are discussing around to this keyword.

Automatically post new tweets

Since we are able to search within tweeter, we shall also be able to post a new tweet with Selenium.

Let’s first locate the below text area by the data-testid attribute:

web scraping and automating twitter post with selenium

Below is the code to locate to the span of the text area by it’s ancestor div:

tweet_text_span = driver.find_element_by_xpath("//div[@data-testid='tweetTextarea_0']/div/div/div/span")

Then we can send whatever text we want to tweet:

tweet_text_span.send_keys("Do you know we can tweet with selenium?")

Once the text is written into the span, the tweet button will be enabled. You can locate the button and click to submit the post:

tweet_button = wait.until(ec.visibility_of_element_located((By.XPATH, 
                                                           "//div[@data-testid='tweetButtonInline']")))
tweet_button.click()

Upon submission, you shall see a new post added to your timeline as per below:

web scraping and automating twitter post with selenium

 

Move invisible element into visible view

There are always cases that you need to scroll up and down or left and right to view some information on the web page. You will also need to make sure your elements are in the view before you can do any operation such as getting its attributes or performing clicks.

To move the elements into the view, you can execute some JavaScript code to scroll to the element as per below:

who_to_follow = driver.find_element_by_xpath("//div/span[text() = 'Who to follow']")
driver.execute_script("arguments[0].scrollIntoView(true);", who_to_follow)

Hide your browser with headless mode

When you use Selenium for some automation or scraping job, you may not wish to see the web pages jumping around in front of you. To make it running peacefully in the background, you can set the headless parameter in the Chrome option before the initialization of the web driver:

chromeOptions.add_argument('--headless')

With this parameter, we would not see the browser launched and everything will be running quietly in the background. It’s good that you always test your code properly before you enable this headless mode.

Conclusion

In this article, we have demonstrated how to use Selenium to automatically login to tweeter account, and read or post tweets. And we have also reviewed through how to trigger the JavaScript code with Selenium web driver and run everything totally in the background. In your real project, you may not want to use the same approach to scrap website like tweeter since it has already provided developer account with all the API access. So this article is more to showcase the capability of the Selenium package.

With Selenium, dealing with complicated operations such as user authentication become much simpler as everything is performed like a normal browser user, and it also provides action chains to support all sorts of mouse movement actions such hover over or drag and drop etc. You shall consider to use it in your automation project or web scraping project if your target website relies heavily on the front-end JavaScript code.

The key for understanding python positional and keyword arguments

In one of the previous article, we have summarized the different ways of passing arguments to Python script. In this article, we will be reviewing through the various approaches for defining and passing arguments into a function in python.

First, let’s start from the basis.

Parameter vs argument

By definition, parameters are the identifiers you specified when you define your function, while arguments are the actual values you supplied to the function when you make the function call. Sometimes you may see people mix up the word parameter and argument, but ultimately they are all referring to the same thing.

Basically Python function supports two types of parameters: positional and keyword arguments. Positional argument is designed to accept argument by following its position during the definition time, while for keyword arguments, you will need to specify the identifier (keyword) followed by the values.

You are allowed to use a combination of positional and keyword arguments when you define your function parameters. Below are the 4 types of variations:

Positional or keyword parameter

By default when you define a Python function, you can either pass in the arguments as positional or keyword. For instance, the below function requires two parameters – file_name and separator:

def read_file(file_name, separator):    
    print(f"file_name={file_name}, separator={separator}")
    file_text  = "Assuming this is the first paragraph from the file."
    return file_text.split(separator)

You can make the function call by supplying the arguments by parameter position or providing the parameter keywords:

read_file("text.txt", " ")
#or
read_file(file_name="text.txt", separator=" ")
#or
read_file("text.txt", separator=" ")
#or
read_file(separator=" ", file_name="text.txt")

All above 4 calls would give you the same results as per below:

file_name=text.txt, separator= 
['Assuming', 'this', 'is', 'the', 'first', 'paragraph', 'from', 'the', 'file.']

Python accepts these arguments regardless of the arguments are provided in positional form or keyword form. When all arguments are by keywords, you can provide them in any order. But take note that positional arguments must be placed before the keyword arguments. For instance, the below throws syntax error,

read_file(file_name="text.txt", " ")

which shows “positional argument follows keyword argument”.

python positional argument and keyword argument

keyword only parameter

For clarity, you may want to implement functions that only accept keyword arguments, and your callers are restricted to only use keyword arguments to invoke the function. To achieve that, you can tweak a bit on your function definition with an additional “*” to indicate all parameters after it must be passed as keywords arguments. E.g.:

def write_file(file_name, *, separator, end, flush):
    print(f"file={file_name}, sep={separator}, end={end}, flush flag={flush}")

For the above function, the separator, end, flush parameters will only accept keyword arguments. You can call it as per below:

write_file("test.txt", separator=" ", end="\r\n", flush=True)

And you shall see output as per below:

file=test.txt, sep= , end=
, flush flag=True

But if you try to pass in all as positional arguments:

write_file("test.txt", " ", "\n", False)

You would see the below error message, which shows the last 3 positional arguments were not accepted.

python positional argument and keyword argument

To further restrict all parameters to be keyword only, you just need to shift the “*” to the beginning of all parameters:

def write_file(*, file_name, separator, end, flush):
    print(f"file={file_name}, sep={separator}, end={end}, flush flag={flush}")

This would make all parameters to only accept keyword arguments. And you can unpack an existing dictionary and pass it arguments into the function:

options = dict(file_name="test.txt", separator=",", end="\n", flush=True)
write(**options)

Positional only arguments

Many Python built-in functions only accept positional arguments, for instance the pow, divmod and abs etc. Prior to Python version 3.8, there is no easy way to define your own function with positional-only parameters. But from version 3.8 onward, you can restrict your function to only accept positional arguments by specifying the “/” (PEP 570) in the function definition. All the parameters come before “/” will be positional-only arguments. For example:

def read(file_name, separator, /):
    print(f"file_name={file_name}, separator={separator}")

For these parameters, if you try to pass in the keyword argument as per below:

read("test.txt", separator=",")

You would see error message indicating the particular parameter is positional-only argument.

    read("test.txt", separator=",")
TypeError: read() got some positional-only arguments passed as keyword arguments: 'separator'

Arbitrary number of arguments

There are cases that you have a number of positional or keyword arguments and you do not want to have a long list of parameters in the function definition. For such case, you can use *args and **kwargs to define arbitrary number of arguments to be accepted by the function. For instance:

def log(*args,**kwargs):
    print(f"args={args}, kwargs={kwargs}")

The above function accepts any number of positional arguments and keyword arguments. All the positional arguments will be packed into a tuple, and keyword arguments are packed into a dictionary.

log("start", "debugging", program_name="python", version=3.7)

When you make a function call as per above, you can see the below output:

args=('start', 'debugging'), kwargs={'program_name': 'python', 'version': 3.7}

This is especially useful when you are just trying to capture a snapshot of all the input arguments (such as logging) or implement a wrapper function for decorator where you do not need to know the exact arguments being passed in. The arbitrary arguments give the callers more flexibility on what they want to pass into your function.

The disadvantage are also obvious, it’s unclear to the new reader what are the parameters to be provided in order to get the correct result from the function call; and you shall not expect all the arguments to be present during the call, so you will have to write some logic to handle the various scenarios when the particular parameters are missing or present.

The best practice for function arguments

When you only have 1 or 2 parameters for the function, you won’t typically see any issue with the code readability/clarity. Problems usually emerge when you have more parameters and some are mandatory and some are optional. Consider the below send_email example:

def send_email(subject, to, cc, 
               bcc, message, attachment, 
               onbehalf, important, read_receipt_requested):
    print(f"{subject}, {to}, {cc}, \
          {bcc} {onbehalf}, {message}, \
          {attachment}, {important}, {read_receipt_requested}")

When you try to include as many parameters as possible to make the function generic for everybody, you’ll have to maintain a very long list of the parameters in the function definition. Calling this function by passing the positional arguments can be very confusing, as you will have to follow exactly the same sequence to provide the arguments as per in the function definition without omitting any single optional argument. E.g.:

send_email("hello", "abc@company.com", "bcd@company.com",
           None, None, 
           "hello", None, False, True)

Without referring back to the function definition, it would be very hard to know which argument represents for which parameter. The best way to make handle such scenario is to always use keyword arguments and set the optional arguments with a default value. For example, the below improved version:

def send_email(subject, to, message, 
               cc=None, bcc=None, attachment=None, 
               onbehalf=None, important=False, read_receipt_requested=False):
    print(f"{subject}, {to}, {cc}, \
          {bcc} {onbehalf}, {message}, \
          {attachment}, {important}, {read_receipt_requested}")

#specify all parameters with keyword arguments
send_email(subject="hello", 
  to="abc@company.com", 
  cc="bcd@company.com", 
  message="hello", 
  read_receipt_requested=True)

With keyword arguments, it improves readability of your code and also allows you to specify default values for the optional arguments. Further more, it gives you the flexibility to extend your parameter list in the future without refactoring your existing code, so that it provides the backwards compatibility at the very beginning.

Conclusion

In this article, we have reviewed through the different approaches for defining and passing arguments to Python function as well as their advantages and disadvantages. Based on your own scenario, you may need to evaluate whether to use positional-only, keyword-only, mix of positional and keyword, or even arbitrary arguments. For code clarity, the general recommendation is to use keyword argument as much as possible, so that all the arguments are understandable at the first glance, and reduces the chances of error.