Home » excel » excel – Conversion from hexadecimal string to Double yields wrong results

# excel – Conversion from hexadecimal string to Double yields wrong results

Posted by: admin May 14, 2020 Leave a comment

Questions:

I am trying to convert 14 bit hex numbers to decimal.

I have this VBA code.

``````Option Explicit

Public Function HexadecimalToDecimal(HexValue As String) As Double

Dim ModifiedHexValue As String
ModifiedHexValue = Replace(HexValue, "0x", "&H")

HexadecimalToDecimal = CDec(ModifiedHexValue)
End Function
``````

With numbers like this to convert to decimal

• 0x047B1142591E80
• 0x044A81325A1E80
• 0x047B7542591E80

I keep getting random results across large amounts of data. Sometimes spot on other times the numbers are off by 6 or 2.

How to&Answers:

Try changing the return type of the function from `Double` to `Variant`. `Double` has only about 15 decimal digits of precision, so can’t, for example, capture the value 1261213964639872 (which has 16 digits) exactly. The closest it can get is 1261213964639870. By changing the return type to `Variant`, the full precision returned by `CDec` will be preserved. You can’t use a `Decimal` return type, because VBA for some reason does not support this.

### Answer：

The problem isn’t with VBA. Excel cells can only hold 15 digits in number format. So the “number” `1234567891234567` will always display `1234567891234560`. This can be avoided by converting items to text AND/OR changing the cell format to text.

But this doesn’t always work.

The only surefire way to make sure it will retain all digits is to append something onto the string that isn’t a number.

This code will append an apostrophe before the number, but return the entire string.

``````Public Function HexadecimalToDecimal(HexValue As String) As String

Dim ModifiedHexValue As String
ModifiedHexValue = Replace(HexValue, "0x", "&H")

HexadecimalToDecimal = "'" & CDec(ModifiedHexValue)
End Function
``````

Unfortunately, not a perfect solution.