BeautifulSoup & Requests for Dynamic Websites Python

[Naksh-Rathore]

Body

I was wondering if I could use BeautifulSoup and Requests in Python to scrape cargurus.com. I think it is a dynamic website, so can I get the car prices and images from those libraries?

Guidelines

• I have read and understood this category's guidelines before making this post.

[Aryan-Vora]

Yes, Cargurus is a dynamic site, but you can scrape it using requests since the API returns JSON data. No need for BeautifulSoup. Here's an example:

import requests
url = 'https://www.cargurus.com/Cars/searchPage.action?searchId=edbf91d6-ee03-4d89-b58d-20891db3a9a0&zip=95064&distance=50&sourceContext=carGurusHomePageModel&inventorySearchWidgetType=AUTO&sortDir=ASC&sortType=BEST_MATCH&srpVariation=DEFAULT_SEARCH&isDeliveryEnabled=true&nonShippableBaseline=0'
print(requests.get(url).json())

I got this url by

1. Going to cargurus.com
2. Making a search for all cars
3. Going into inspect element and reloading the page
4. Finding the request starting with seachPage.action?
5. Clicking on it and looking at the headers and then putting the url from there into the python file

You can create a second list called headers and write code to generate the url based on what you want to find. So for example if you want a car from 2000 to 2010 you can modify the headers to get that instead of parsing the whole body on your own. I would highly suggest watching a few videos on youtube about scraping ecommerce sites as they would be the closest example to the cargurus site.