Question

Write a python program that prompts the user for a web address (full url, html page) and then uses the BeautifulSoup and urllib to read its data.

You can use the any html page you'd like:

The program should output the following:

1. The page title (hint: read the content from the metatag "title")
2. The total number of words on the page
3. The total number of unique words on the page
4. The total number of each heading used on the page (hint: count the number of "H1"-"H6" header tags used on the page)
5. The total number of paragraphs on the page (hint: use the <p> tag to count)
6. The total number of links on the page
7. After outputting, the program should ask the users if they would like to parse another web address or quit.
8. Use at least 3 web addresses to check your program. List their urls as comments in the beginning of your program.

Remember:

• Only use the material covered in this module -- do not use more advanced functions not covered yet in the course
• Make sure to include comments that explain all your steps (starts with #). Also use a comment to sign your name at the beginning of the program!
• Work individually and only submit original work
• Run the program a few times to make sure it executes and meets all the requirements
• Submit a .py file!

Answer 1

GIVEN:

write a python program that prompts the user for the web address (full, url, html page ) and then uses the Beautifulsoap and urllib to read its data.

PROGRAM:

from urllib import urlopen

from bs4 import BeautifulSoup // header files

url = input("Enter the url: ")

r = request.get(url)

soup = BeautifulSoup(r.content) // getting input from user

text_p = (' '.join(s.findAll(text = True)) for s in soup.findAll('p'))

c_p = Counter((x.rstrip(punctuation).lower() for y in text_p for x in y.split()))

text_div = (' '.join(s.findAll(text = True)) for s in soup.findAll('div'))

c_div = Counter((x.rstrip(punctuation).lower() for y in text_div for x in y.split()))

total = c_div + c_p

print(total) // total number of words on page

c_H1 = c_H2 = c_H3 = c_H4 = c_H5 = c_H6 = 0

for tag in soup.findAll():

if(tag.name =="H1" or tag.name == "<H1>):

c_H1 = c_H1+1

if(tag.name =="H2" or tag.name == "<H2>):

c_H2 = c_H2+1

if(tag.name =="H3" or tag.name == "<H3>):

c_H3 = c_H3+1

if(tag.name =="H4" or tag.name == "<H4>):

c_H4 = c_H4+1

if(tag.name =="H5" or tag.name == "<H5>):

c_H5 = c_H5+1

if(tag.name =="H6" or tag.name == "<H6>):

c_H6 = c_H6+1 // header tags used on the page

print(c_H1)

print(c_H2)

print(c_H3)

print(c_H4)

print(c_H5)

print(c_H6)

count = 0

for tag in soup.findAll(): // total number of paragraphs on the page

if(tag.name == 'p' or tag.name == '<p>)

count = count+1

print(count)

print(len(soup.find_all('img')))

count = 0

for link in soup.find_all('a', href = True): // total number of links on the page

count = count+1

print(count)

title = soup.find("meta", property="og:title")

print(title)

des = soup.find("meta", property = "og:description")

print(des)