Scraping Multiple Web Pages Has The Same Results As The First Page Using Python

My question is about that I tried to get the product names from CME group website. However, why the code be wouldn't be able to access the next page although I changed the URLs in

Solution 1:

You could try using the requests library rather than urllib. I just accessed page 5 successfully using code similar to yours with this difference.

Note that the literal 'D3' appears on page five but not on page one.

>>>import requests>>>i = 5>>>url='http://www.cmegroup.com/trading/products/#pageNumber='+str(i)+'&sortAsc=false'>>>page = requests.get(url).content>>>import bs4>>>soup = bs4.BeautifulSoup(page, 'lxml')>>>soup.find_all(string='D3')
['D3', 'D3']