Home » excel » excel – Conversion from hexadecimal string to Double yields wrong results

excel – Conversion from hexadecimal string to Double yields wrong results

Posted by: admin May 14, 2020 Leave a comment

Questions:

I am trying to convert 14 bit hex numbers to decimal.

I have this VBA code.

Option Explicit

Public Function HexadecimalToDecimal(HexValue As String) As Double

Dim ModifiedHexValue As String
ModifiedHexValue = Replace(HexValue, "0x", "&H")

HexadecimalToDecimal = CDec(ModifiedHexValue)
End Function

With numbers like this to convert to decimal

  • 0x047B1142591E80
  • 0x044A81325A1E80
  • 0x047B7542591E80

I keep getting random results across large amounts of data. Sometimes spot on other times the numbers are off by 6 or 2.

How to&Answers:

Try changing the return type of the function from Double to Variant. Double has only about 15 decimal digits of precision, so can’t, for example, capture the value 1261213964639872 (which has 16 digits) exactly. The closest it can get is 1261213964639870. By changing the return type to Variant, the full precision returned by CDec will be preserved. You can’t use a Decimal return type, because VBA for some reason does not support this.

Answer:

The problem isn’t with VBA. Excel cells can only hold 15 digits in number format. So the “number” 1234567891234567 will always display 1234567891234560. This can be avoided by converting items to text AND/OR changing the cell format to text.

But this doesn’t always work.

The only surefire way to make sure it will retain all digits is to append something onto the string that isn’t a number.

This code will append an apostrophe before the number, but return the entire string.

Public Function HexadecimalToDecimal(HexValue As String) As String

    Dim ModifiedHexValue As String
    ModifiedHexValue = Replace(HexValue, "0x", "&H")

    HexadecimalToDecimal = "'" & CDec(ModifiedHexValue)
End Function

Unfortunately, not a perfect solution.