I am attempting to work with a very large dataset that has some non-standard characters in it. I need to use unicode, as per the job specs, but I am baffled. (And quite possibly doing it all wrong.)
I open the CSV using:
15 ncesReader = csv.reader(open('geocoded_output.csv', 'rb'), delimiter='\t', quotechar='"')
Then, I attempt to encode it with:
name=school_name.encode('utf-8'), street=row.encode('utf-8'), city=row.encode('utf-8'), state=row.encode('utf-8'), zip5=row, zip4=row,county=row.encode('utf-8'), lat=row, lng=row)
I’m encoding everything except the lat and lng because those need to be sent out to an API. When I run the program to parse the dataset into what I can use, I get the following Traceback.
Traceback (most recent call last): File "push_into_db.py", line 80, in <module> main() File "push_into_db.py", line 74, in main district_map = buildDistrictSchoolMap() File "push_into_db.py", line 32, in buildDistrictSchoolMap county=row.encode('utf-8'), lat=row, lng=row) UnicodeDecodeError: 'ascii' codec can't decode byte 0xd1 in position 2: ordinal not in range(128)
I think I should tell you that I’m using python 2.7.2, and this is part of an app build on django 1.4. I’ve read several posts on this topic, but none of them seem to directly apply. Any help will be greatly appreciated.
You might also want to know that some of the non-standard characters causing the issue are Ñ and possibly É.
Unicode is not equal to UTF-8. The latter is just an encoding for the former.
You are doing it the wrong way around. You are reading UTF-8-encoded data, so you have to decode the UTF-8-encoded String into a unicode string.
So just replace
.decode, and it should work (if your .csv is UTF-8-encoded).
Nothing to be ashamed of, though. I bet 3 in 5 programmers had trouble at first understanding this, if not more 😉
If your input data is not UTF-8 encoded, then you have to
.decode() with the appropriate encoding, of course. If nothing is given, python assumes ASCII, which obviously fails on non-ASCII-characters.
Just add this lines to your codes :
import sys reload(sys) sys.setdefaultencoding('utf-8')
The main reason for the error is that the default encoding assumed by python is ASCII.
Hence, if the string data to be encoded by
encode('utf8') contains character that is outside of ASCII range e.g. for a string like ‘hgvcj터파크387’, python would throw error because the string is not in the expected encoding format.
If you are using python version earlier than version 3.5, a reliable fix would be to set the default encoding assumed by python to
import sys reload(sys) sys.setdefaultencoding('utf8') name = school_name.encode('utf8')
This way python would be able to anticipate characters within a string that fall outside of ASCII range.
However, if you are using python version 3.5 or above, reload() function is not available, so you would have to fix it using decode e.g.
name = school_name.decode('utf8').encode('utf8')
for Python 3 users. you can do
with open(csv_name_here, 'r', encoding="utf-8") as f: #some codes
it works with flask too 🙂