Tuesday, 5 March 2019

Powershell Web Access

Sometimes we used to monitor some websites continually for getting a specific information, like breaking news, online product availability checks etc: Here I am showing how can we capture a website information suing powershell. From the below example I am capturing information from one of the online newspaper and displaying as a popup notification, I used windows form to create the popup notification. Please follow the below script to capture some information from a website and display as a popup notification on your desktop.
There are three faces in this script,                               
  1. Capture the information from a website.
  2. Create a windows form to display the output.
  3. Schedule a task to run the script in a specific time\date.

We are using Invoke-WebRequest PowerShell cmdlet to capture the information from the website, please go through the below Microsoft website to know more about the cmdlet

Before we start we need to have some basic understanding about HTML to identify what is an HTML “Class” and “tags” (Div, H2, UL Etc.), then decide what information we are going to capture from the website and access the DevTool (Developer Console) from the website. Below are some examples to capture the information from the website

Browse the website by typing the URL

Right click on the page (The area which we are going to capture) and click on Inspect (I am using Chrome, it’s Inspect element in IE) to access the DevTool, we can easily identify the required HTML tag and class from DevTool.

As per the above example, we are going to display the news headlines which is arranged using an HTML List tag (UL Tag) and its class name is list8 – Based on the website design change, the HTML class name and tag may get changed. 

Let’s start Part -1 Getting Information from the website

We are going to load the newspaper page on a PowerShell console, before loading the website I have declared some variable for getting the basic details about the website name, URL, html tag and class name

$WebsiteURL = "https://timesofindia.indiatimes.com/"
$WebsiteName = "Time of India"
$Class = "list8"

$TypeClass = "ul"  

I am saving the website page in a variable, please check the below screenshot to know more about the properties of the web request stored in the variable. The variable contains a huge amount of details about the website

$WebRequest = Invoke-WebRequest $WebsiteURL

$News = ($WebRequest.ParsedHtml.getElementsByTagName("$TypeClass"| Where$_.className -eq "$Class" } ) | Select-Object -ExpandProperty innertext -First 1

ParsedHtml – It is basically taking in HTML code and extracting relevant information like the title of the page, paragraphs in the page, headings in the page, links, bold text etc.

GetElementsByTagName – The getElementsByTagName() method returns a collection of all elements in the document with the specified tag name, as a NodeList object.

Part 2 - Creating a Windows form to display the news

Follow my below post about “How to build a form in PowerShell?” to understand more about the windows form.

We have captured the required information from the website and stored in a variable called $News, we need to now display the information on a window, please follow the below cmdlet to create a windows form to hold the information

Basic Form creation cmdlet, I did some customization in our form.

$Form = New-Object system.Windows.Forms.Form
$Form.Size = New-Object System.Drawing.Size(800,400)
$Form.AutoSize = $false
$Form.AutoSizeMode = "GrowAndShrink"
$Form.Text = "News Update"
$Form.StartPosition = "CenterScreen"
$Form.FormBorderStyle = 'Fixed3D'
$Form.KeyPreview = $True
$Form.MaximizeBox = $false

Below is the output display part of the form, you can customize based on your requirement.

$Font = New-Object System.Drawing.Font("Segoe UI",15,[System.Drawing.FontStyle]::Italic)
$outputBox = New-Object System.Windows.Forms.TextBox
$outputBox.Location = New-Object System.Drawing.Size(10,10)
$outputBox.Size = New-Object System.Drawing.Size(780,330)
$outputBox.Font = $Font
$outputBox.MultiLine = $True
$outputBox.ScrollBars = "Vertical"
$outputBox.BackColor = "skyblue"
$Date = get-date
$OutputBox.AppendText("Updated : $Date")

Below is the hyperlink creation in form in the bottom of the window to navigate to the website to read more information

$LinkLabel = New-Object System.Windows.Forms.LinkLabel
$LinkLabel.Location = New-Object System.Drawing.Size(10,345)
$LinkLabel.Size = New-Object System.Drawing.Size(100,50)
$LinkLabel.LinkColor = "BLUE"
$LinkLabel.ActiveLinkColor = "RED"
$LinkLabel.Text = "$Websitename"

Now add all the controls to form and display the output


Finally, below is the output

Part-3 Schedule a task to run the script in a specific time\date.

Now we can run the below PowerShell command for scheduling our news update. Please note that the task scheduler command works only on the latest Powershell version, if you are using old PowerShell version, please import the module frPowerShellell gallery.

$Trigger= New-ScheduledTaskTrigger -At 10:00am –Daily
$UserDetails = "administrator"
$Action= New-ScheduledTaskAction -Execute "PowerShell.exe" -Argument "D:\Powershell Script\NewsScript.ps1"
Register-ScheduledTask -TaskName "NewsUpdate" -Trigger $Trigger -User $User -Action $Action -RunLevel Highest –Force

Bellow is the another example of collecting HTML details from an online shopping website, we can create a script to get notifications for product availability - Please try to create the script.