Parse HTML Infinite Scroll

I'm trying to parse the HTML of a page with infinite scrolling. I want to load all of the content so that I can parse it all. I'm using Python. Any hints?

Solution 1:

Those pages update their html with AJAX. Usually you just need to find the new AJAX requests send by browser, guess the meaning of the AJAX url parameters and fetch the data from the API.

API servers may validate the user agent, referer, cookie, oauth_token ... of the AJAX request, keep an eye on them.

Solution 2:

the data is

1. either loaded in advance

2. or the page sends a request while you scroll

you can use httpfox to find the request and send it