Memory Error When Appending To List In Python

I have a list of 8000 website urls. I would like to scrape the text off of the websites and save everything as a csv file. To do this i wanted to save each text-page in a list. Thi

Solution 1:

If you can't hold all your data in memory, then don't. At a high level, your code has this structure

for k in links:
    temp = []
    temp2 = []
    browser.visit(k)

    # do stuff that fills in tempfor s in temp:
        ss = re.sub(r'[^\w]', ' ', s)
        temp2.append(ss)

    temp2 = ' '.join(temp2)
    print(temp2.strip())

    df.append(temp2.strip())

withopen('Hair_Salons text', 'w') as myfile:
    wr = csv.writer(myfile, quoting=csv.QUOTE_ALL)
    wr.writerow(df)

So, you put lots of stuff into a data frame, then write it - you don't use it in the loop. Instead of the df.append(temp2.strip()) write to the file there. Make you you either open the file once, outside the loop (perhaps more sensible) or open for appending (using 'a' instead of 'w').