Home » Python » UnicodeDecodeError: 'utf-8' codec can't decode byte

UnicodeDecodeError: 'utf-8' codec can't decode byte

Posted by: admin November 29, 2017 Leave a comment

Questions:

Here is my code,

for line in open('u.item'):
#read each line

whenever I run this code it gives the following error:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 2892: invalid continuation byte

I tried to solve this and add an extra parameter in open(), the code looks like;

for line in open('u.item', encoding='utf-8'):
#read each line

But again it gives the same error.
what should I do then! Please help.

Answers:

As suggested by Mark Ransom, I found the right encoding for that problem. The encoding was “ISO-8859-1”, so replacing open("u.item", encoding="utf-8") with open('u.item', encoding = "ISO-8859-1") will solve the problem.

Questions:
Answers:

Your file doesn’t actually contain utf-8 encoded data, it contains some other encoding. Figure out what that encoding is and use it in the open call.

In Windows-1252 encoding for example the 0xe9 would be the character é.

Questions:
Answers:

Try this to read using pandas

pd.read_csv('u.item', sep='|', names=m_cols , encoding='latin-1')

Questions:
Answers:

If you are using Python 2 the following will the solution:

import io
for line in io.open("u.item", encoding="ISO-8859-1"):
    # do something

Because encoding parameter doesn’t work with open(), you will be getting the following error:

TypeError: 'encoding' is an invalid keyword argument for this function

Questions:
Answers:

If someone looking for these, this is an example for converting a CSV file in Python 3:

try:
    inputReader = csv.reader(open(argv[1], encoding='ISO-8859-1'), delimiter=',',quotechar='"')
except IOError:
    pass

Questions:
Answers:

Also worked for me, ISO 8859-1 is going to save a lot, hahaha, mainly if using Speech Recognition API’s

Example:

file = open('../Resources/' + filename, 'r', encoding="ISO-8859-1");