Decoding Problems In Django And Lxml

I have a strange problem with lxml when using the deployed version of my Django application. I use lxml to parse another HTML page which I fetch from my server. This works perfec

Solution 1:

"\x85why hello there!" is not a utf-8 encoded string. You should try decoding the webpage before passing it to lxml. Check what encoding it uses by looking at the http headers when you fetch the page maybe you find the problem there.

Solution 2:

Doesn't syntax such as u"\x85why hello there!" help?

You may find the following resources from the official Python documentation helpful:

• Python introduction, Unicode Strings
• Sequence Types — str, unicode, list, tuple, buffer, xrange

Solution 3:

Since modifying site.py is not an ideal solution try this at the start of your program:

import sys
reload(sys)
sys.setdefaultencoding("utf-8")