Home » excel » tsql – Converting varchar to decimal when various special characters exists in data originating from excel

tsql – Converting varchar to decimal when various special characters exists in data originating from excel

Posted by: admin April 23, 2020 Leave a comment

Questions:

I have an imported csv file from excel. All values are imported as varchar due to the need to retain raw values. Now I need to convert monetary values to decimal(18,4). However many of the rows contain special characters (see a short list below).

I would like to create a function to clean special characters and return a decimal data type. This seems like many before me should have faced and addressed this problem and I’m asking for the best approach to accomplish this.

Here is a small list of what I am currently facing and would like to know if there are any other characters I should consider in addition to ($,-) leading space(s), trail space(s):

 $634,375.00 
 (104,055.00)
 -139,686.03 
 (72,631.45)
17774137.14
8374187.29
$-7041078.47

Below is a select statement that I am working on to aid in the development of the function:

SELECT i.[AsOfDate]
,COALESCE(case when len(ltrim(rtrim(i.[DealNum])))=  0 then null else i.[DealNum] end,
    case when len(ltrim(rtrim(i.[CUSIP])))=  0 then null else i.[CUSIP] end,
    case when len(ltrim(rtrim(i.[PoolNum])))=  0 then null else i.[PoolNum] end) AS [DealNum_CusIP_PoolNum]
,case when isnumeric(i.[Coupon]) = 1 then cast(i.[Coupon] as decimal(18,4)) else 0 end as [Coupon]
,case when isdate(i.[PurchaseDate]) = 1 then cast(i.[PurchaseDate] as date) else null end as [PurchaseDate]
,case when isdate (i.[SettleDate]) = 1 then cast(i.[SettleDate] as date) else null end as [SettleDate]
,case when isnumeric(Replace(Replace(ltrim(rtrim(i.[CurrentFace])),'(','-'),')','')) = 1 then cast(Replace(Replace(ltrim(rtrim(i.[CurrentFace])),'(','-'),')','') as decimal(18,4)) else 0 end as [CurrentFace]
,case when isnumeric(i.[PurchasePrice]) = 1 then cast(i.[PurchasePrice] as decimal(18,4)) else 0 end as [PurchasePrice]
,case when isnumeric(i.[CurrentPrice]) = 1 then cast(i.[CurrentPrice] as decimal(18,4)) else 0 end as [CurrentPrice]
,case when isnumeric(i.[RealizedGL]) = 1 then cast(i.[RealizedGL] as decimal(18,4)) else 0 end as [RealizedGL]
,case when isnumeric(Replace(i.[Premium],'$','')) = 1 then cast(Replace(i.[Premium],'$','') as decimal(18,4)) else 0 end as [Premium]
,case when isnumeric(i.[OLMTM]) = 1 then cast(i.[OLMTM] as decimal(18,4)) else 0 end as [OLMTM]
,case when isnumeric(i.[CurrentMTM]) = 1 then cast(i.[CurrentMTM] as decimal(18,4)) else 0 end as [CurrentMTM]
FROM [import].[Openlink_Position_Detail] I

Here is what I have so far but its still not working completely:

CREATE FUNCTION Stage.CleanForDecimal 
(
    @input varchar(100)
)
RETURNS decimal(18,4)
AS
BEGIN

    DECLARE @rv as decimal(18,2), @wv as varchar(100)

    SELECT @wv = ltrim(rtrim(@input));
    SELECT @wv = replace(@input,'$','');
    SELECT @wv = replace(@input,',','');
    SELECT @wv = replace(@input,'(','-');
    SELECT @wv = replace(@input,')','');

    SELECT @rv = case when isnumeric(@wv) = 1 then cast(@wv as decimal(18,4)) else 0 end


    -- Return the result of the function
    RETURN @rv

END
How to&Answers:

It used to be hell taking in formatted currency strings in Excel. Then a developer mentioned TRY_PARSE, introduced in SQL Server 2012:

CAST(TRY_PARSE(MyColumn AS money USING 'en-US') AS float)

This works with spaces, dollar signs, commas, and brackets. It works with all the samples data you supplied, but won’t work with the Euro sign, as an example.

You shouldn’t use money because all calculations are rounded to 2 decimal places, even during the intermediate steps. Here, it’s only used for convenience.

Answer:

Here is the corrected function:

ALTER FUNCTION [Stage].[CleanForDecimal] 
(
@input varchar(100)
)
RETURNS decimal(18,4)
AS
BEGIN

DECLARE @rv as decimal(18,4), @wv as varchar(100)

SELECT @wv = ltrim(rtrim(@input));
SELECT @wv = replace(@wv,'$','');
SELECT @wv = replace(@wv,',','');
SELECT @wv = replace(@wv,'(','-');
SELECT @wv = replace(@wv,')','');

SELECT @rv = case when isnumeric(@wv) = 1 then cast(@wv as decimal(18,4)) else 0 end

RETURN @rv

/* execution example:
select Stage.CleanForDecimal(' ($123,44.2345)')
*/

END

Answer:

FWIW here’s an Oracle solution for the same problem. Of course you can’t use it as is, but perhaps you can use the logic of how we approached the problem. Information is power!

/********************************************************************************************************
      Name:       STR_TO_NUMBER

      Desc:       Converts a string to a number, stripping dollar signs, commas and converting if a negative is
                    shown by parentheses. Used when converting data froma spreadsheet where
                    the format shows negatives in parens.

      Args:       string_in IN VARCHAR2

      Returns:    NUMBER 

      Usage:      SELECT thc_utl.str_to_number('(1,234.56)')
                  FROM dual;

     REVISIONS:
     Ver        Date        Author           Description
     ---------  ----------  ---------------  ------------------------------------
     1.0        12/12/2014   Gary_W          - Created function.
  ************************************************************************************************************************/
FUNCTION STR_TO_NUMBER(string_in IN VARCHAR2) RETURN NUMBER AS
  v_nbr NUMBER;
BEGIN

  -- TRIM the string, then
  --   strip commas with REPLACE()
  --   strip dollar signs with REPLACE()
  --   then take what's in the parens, add a negative sign and make it a number.
  v_nbr := to_number(regexp_replace(replace(replace(TRIM(string_in),','),'$'), '^\(([^)]+)\)$', '-'));

  RETURN v_nbr; -- Return the converted number.

END STR_TO_NUMBER;

Gary_W puts on fire-retardant suit and awaits replies.

Answer:

Here is another solution based on JAC’s input.

declare @somestring as varchar(100)

set @somestring = ' ($123,234.5567)'
Select isnull(CAST(TRY_PARSE(@somestring AS money USING 'en-US') AS decimal(18,4)),0) as rv
-- returns -123234.5567

set @somestring = ' ($123,2fubar34.5567)'
Select isnull(CAST(TRY_PARSE(@somestring AS money USING 'en-US') AS decimal(18,4)),0) as rv
-- returns 0.0000

I’m not sure if this is more efficient than the function I posted earlier but I hope it will help others with this issue.