Python Crawl Chain Home Home Price Information

The house problem has been getting more and more attention in recent years, to understand the house price in recent years, first of all, we have to get the house price information on the Internet, we take the information of house price sold on the Internet by Chain Home as an example, and crawl down the data and store it.

This time, we still take the line of requests-Beautiful Soup to crawl the information of selling houses on the Chain Home website. You need to install anaconda, and ensure that the system already has the requests library, Beautiful Soup4 library and csv library has been installed.

Web analytics

The page we are going to crawl is as follows, the information we need is the name and price of the house

/ershoufang/ 

Below:

Let's analyze the location of the information we want to extract, open the developer mode to find the element, we find the name and price of the house; as shown below:

We can see that the information we need about the name of the house is in {div class="title"}, the price information is in {div class="totalPrice"}, and all the information is encapsulated in the li tag.

We analyzed the structure of the web pages inside a web page, to crawl other web pages we have to see more structure;

First web link: /ershoufang/pg1

Link to second web page: /ershoufang/pg2

Third web link: /ershoufang/pg3

The links on each page end in pg + a certain page number, and we use this to spot patterns.

program structure

Next we write the main structure of the program:

import requests



from bs4 import BeautifulSoup



import csv



 



def getHTMLText(url): #Get the web page information



    return ""



 



def get_data(list,html): #get_data_and_store_it



 



def main():



    start_url = "/ershoufang/pg"



    depth = 20



    info_list = []



    for i in range(depth):



        url = start_url + str(i)



        html = getHTMLText(url)



        get_data(info_list,html)



main()

According to eachfunction (math.)The purpose of the function to fill in the specifics of the function, the first function to get the web page information:

def getHTMLText(url):



    try:



        r = (url)



        r.raise_for_status()



         = r.apparent_encoding



        return 



    except:



        return "Exception thrown"

第二个函数是要获取数据并存储在文件中；

def get_data(list,html):



    soup = BeautifulSoup(html,'')



    infos = ('ul',{'class':'sellListContent'}).find_all('li') #find all li tags



    with open(r'/Users/11641/Desktop/','a',encoding='utf-8') as f: #Create a file and write to it.



        for info in infos:



            name = ('div',{'class':'title'}).find('a').get_text()



            price =('div',{'class':'priceInfo'}).find('div',{'class','totalPrice'}).find('span').get_text()



            ("{},{}\n".format(name,price))

Above we have finished writing the crawler program

However, it should be noted that: run the completed program to save the csv file, if you open directly with excel may have a garbled phenomenon, generally with the following steps to solve the problem:

1. Open the file with Notepad, Save As select the encoding as: ANSI and store the file;
2. The new file will be stored in excel open, save as excel file format, you can permanently store the file

Let's take a look at the effect of file storage as shown below:

program code

Above we have successfully written the code for the Chain Home used house crawler, and successfully output the results we roughly want, all the code is as follows;

#Crawl Chain Home Used Home Information



import requests



from bs4 import BeautifulSoup



import csv



 



def getHTMLText(url):



    try:



        r = (url,timeout=30)



        r.raise_for_status()



         = r.apparent_encoding



        return 



    except:



        return 'Exception thrown'



 



def get_data(list,html):



    soup = BeautifulSoup(html,'')



    infos = ('ul',{'class':'sellListContent'}).find_all('li')



    with open(r'/Users/11641/Desktop/','a',encoding='utf-8') as f:



        for info in infos:



            name = ('div',{'class':'title'}).find('a').get_text()



            price =('div',{'class':'priceInfo'}).find('div',{'class','totalPrice'}).find('span').get_text()



            ("{},{}\n".format(name,price))



      



def main():



    start_url = '/ershoufang/pg'



    depth = 20



    info_list =[]



    for i in range(depth):



        url = start_url + str(i)



        html = getHTMLText(url)



        get_data(info_list,html)



main()

Summary:

1. Crawler mainly with requests library and Beautiful Soup library can be succinctly crawl the web page information;

2.First set the main framework of the program, and then according to the purpose of the demand to fill the function content: get web page information > crawl web page data > store data;

3. For all the information stored on multiple pages, it is important to observe the web page information and construct a url link to each page to solve the problem;

4. The most important thing is to parse the structure of the web page, it is best to use the form of the tag tree to determine the label where the field is located, and traverse all the labels to store data.