I created a proc this week that was a simple SELECT ALL from a lookup and the output was going to be used in a drop down menu in a Reporting Services report. The data wasn’t the prettiest and I wanted to capitalize the first letter of every item within my SELECT without using a function. Here is how I did it:

CREATE TABLE #StringData(StringData VARCHAR(100));

INSERT INTO #StringData(StringData)
VALUES ('High life truffaut artisan salvia'),
('raw denim tofu terry richardson skateboard'),
('iphone PickLED'),
('BOOTH SQUID'),
(' ')

SELECT UPPER(LEFT(StringData,1)) + RIGHT(StringData,(LEN(StringData)-1)) AS 'With Capital',
StringData AS 'Original'
FROM #StringData
WHERE LEN(StringData) > 0

UNION

SELECT '' AS 'With Capital',
StringData AS 'Original'
FROM #StringData
WHERE LEN(StringData) = 0

The output is below: it works in every case except a string with a length of 0.


SELECT CAST(DATEPART(YEAR, GETDATE()) AS VARCHAR) AS 'Year'
SELECT CAST(DATEPART(MONTH, GETDATE()) AS VARCHAR) AS 'Month'
SELECT CAST(DATEPART(DAY, GETDATE()) AS VARCHAR) AS 'Day'
SELECT CAST(DATEPART(HOUR, GETDATE()) AS VARCHAR) AS 'Hour'

SELECT CAST(DATEPART(YEAR,GETDATE()) AS VARCHAR) +
'/' +
CAST(DATEPART(MONTH,GETDATE()) AS VARCHAR) +
'/' +
CAST(DATEPART(DAY,GETDATE()) AS VARCHAR) +
SPACE(1) +
CAST(DATEPART(HOUR,GETDATE()) - 1 AS VARCHAR) +
':00:00.000' AS 'Beginning Of Prvious Hour'

The script is quite in-depth and should be run with caution on production servers.

DECLARE @objectID INT;
SET @objectID = OBJECT_ID(QUOTENAME(DB_NAME(DB_ID())));

DECLARE @Objects TABLE (HardCodedIndexes VARCHAR(1000));
INSERT INTO @Objects
SELECT name
FROM [sys].[objects]
WHERE [object_id] IN (SELECT [object_id] FROM [sys].[sql_modules] WHERE [definition] LIKE '%INDEX = %');

IF NOT EXISTS (SELECT TOP 1 0 FROM @Objects)
 BEGIN;
  INSERT INTO @Objects SELECT 'None';
 END;

WITH indexCTE AS
	(SELECT sps.name AS 'partition_scheme_name',
		spf.name AS 'partition_function_name',
		sps.data_space_id AS 'data_space_id'
    FROM [sys].[partition_schemes] AS sps
	INNER JOIN [sys].[partition_functions] AS spf
		ON sps.function_id = spf.function_id),

DMV_OperationalStats AS
	(SELECT index_id,
		[object_id],
		range_scan_count,
		singleton_lookup_count,		-- number of times the index has been used to fetch a single row
		forwarded_fetch_count,		-- for a heap, shows the number of forward pointers have been used to resolve a query
		lob_fetch_in_pages,		-- Quantifies the #of large object (varchar(max),varbinary(max),text etc) retrieved using this index
		lob_fetch_in_bytes,
		row_lock_count,			-- number of row locks that have been requested against this index
		row_lock_wait_count,		-- number of times a process has waited on a row lock against this index
		row_lock_wait_in_ms,		-- amount of time spent waiting on a row lock against this index
		page_latch_wait_count,		-- number of waits and time waited on the physical page of the object to have the latch removed
		page_io_latch_wait_count
	FROM [sys].[dm_db_index_operational_stats](DB_ID(),NULL, NULL, NULL)),

DMV_UsageStats AS
	(SELECT QUOTENAME(DB_NAME(u.database_id))	AS 'database_name',
		i.name					AS 'index_name',
		COALESCE(u.user_seeks,NULL)		AS 'user_seeks',
		COALESCE(u.last_user_seek, NULL)	AS 'last_user_seek',
		COALESCE(u.last_system_seek, NULL)	AS 'last_system_seek',
		COALESCE(u.user_scans, NULL)		AS 'user_scans',
		COALESCE(u.last_user_scan, NULL)	AS 'last_user_scan',
		COALESCE(u.last_system_scan, NULL)	AS 'last_system_scan',
		COALESCE(u.user_lookups, NULL)		AS 'user_lookups',
		COALESCE(u.last_user_lookup, NULL)	AS 'last_user_lookup',
		COALESCE(u.last_system_lookup, NULL) AS 'last_system_lookup',
		COALESCE(u.user_updates, NULL)		AS 'user_updates'
	FROM [sys].[indexes] AS i
		INNER JOIN [sys].[objects] AS o
			ON i.object_id = o.object_id
		LEFT OUTER JOIN [sys].[dm_db_index_usage_stats] AS u
			ON i.object_id = u.object_id
				AND i.index_id = u.index_id
	WHERE u.database_id = DB_ID()
		AND o.[type] <> 'S'),

Fragmentation AS
	(SELECT QUOTENAME(DB_NAME(DB_ID()))		AS 'databaseName',
			ps.[object_id]			AS 'objectID',
			ps.index_id			AS 'indexID',
			ps.partition_number		AS 'partitionNumber',
			SUM(ps.avg_fragmentation_in_percent) AS 'fragmentation',
			SUM(ps.page_count)		AS 'page_count',
			os.range_scan_count,
			GETDATE()			AS 'scanDate'
	FROM [sys].[dm_db_index_physical_stats](DB_ID(), NULL, NULL , NULL, 'LIMITED') AS ps
		INNER JOIN [sys].[dm_db_index_operational_stats](DB_ID(), NULL, NULL , NULL) AS os
			ON ps.database_id = os.database_id
				AND ps.[OBJECT_ID] = os.[OBJECT_ID]
				AND ps.index_id = os.index_id
				AND ps.partition_number = os.partition_number
	GROUP BY QUOTENAME(DB_NAME(ps.database_id)), ps.[OBJECT_ID], ps.index_id, ps.partition_number, os.range_scan_count),

SpaceUsed AS
	(SELECT i.[object_id],
		i.index_id,
		a.total_pages * 8	AS 'Reserved(KB)',
		a.used_pages * 8	AS 'Used(KB)',
		o.create_date		AS 'TableCreateDate',
		o.modify_date		AS 'TableLastModified'
FROM [sys].[indexes] AS i
    INNER JOIN [sys].[partitions] AS p
		ON i.[object_id] = p.[object_id]
			AND i.index_id = p.index_id
    INNER JOIN [sys].[allocation_units] AS a
		ON p.partition_id = a.container_id
    LEFT OUTER JOIN [sys].[objects] AS o
		ON o.[object_id] = i.[object_id]
WHERE o.type = 'U'),

HardCodedIndexes
AS (SELECT COALESCE(HardCodedIndexes + ',' + HardCodedIndexes, HardCodedIndexes) AS 'HardCodedIndexes' FROM @Objects)

SELECT dus.database_name,
	st.name				AS 'Table',
	ISNULL(ix.name, '')		AS 'Index',
	dus.user_seeks			AS 'User Seeks',
	dus.user_scans			AS 'User Scans',
	dus.user_lookups		AS 'User Lookups',
	dus.user_updates		AS 'User Updates',
	CAST(CASE WHEN ix.index_id = 1 THEN 'clustered' WHEN ix.index_id = 0 THEN 'heap' ELSE 'nonclustered' END +
		 CASE WHEN ix.ignore_dup_key <> 0 THEN ', ignore duplicate keys' ELSE '' END +
		 CASE WHEN ix.is_unique <> 0 THEN ', unique' ELSE '' END +
		 CASE WHEN ix.is_primary_key <> 0 THEN ', primary key' ELSE '' END AS VARCHAR(210)) AS 'Index Type',
	STATS_DATE(ix.[object_id], ix.index_id) AS 'Last Stats Update',
	fr.page_count,
	fr.fragmentation,
	dos.range_scan_count,
		SUM(ROWS)		AS 'row_count',
		su.[Reserved(KB)],
		su.[Used(KB)],
		h.HardCodedIndexes	AS 'Indexes Hard-Coded',
		ISNULL(REPLACE( REPLACE( REPLACE(
        (   SELECT c.name	AS 'columnName'
            FROM [sys].[index_columns] AS sic
				INNER JOIN [sys].[columns] AS c
					ON c.column_id = sic.column_id
						AND c.[object_id] = sic.[object_id]
            WHERE sic.[object_id] = ix.[object_id]
                AND sic.index_id = ix.index_id
                AND is_included_column = 0
            ORDER BY sic.index_column_id
            FOR XML Raw)
            , '"/><row columnName="', ', ')
            , '<row columnName="', '')
            , '"/>', ''), '') AS 'indexed_columns',
		ISNULL(REPLACE(REPLACE(REPLACE(
        (   SELECT c.name AS 'columnName'
            FROM [sys].[index_columns] AS sic
				INNER JOIN [sys].[columns] AS c
					ON c.column_id = sic.column_id
						AND c.[object_id] = sic.[object_id]
            WHERE sic.[object_id] = ix.[object_id]
                AND sic.index_id = ix.index_id
                AND is_included_column = 1
            ORDER BY sic.index_column_id
            FOR XML Raw)
            , '"/><row columnName="', ', ')
            , '<row columnName="', '')
            , '"/>', ''), '')        AS 'included_columns',
        fr.partitionNumber,
	dus.last_user_seek,
	dus.last_system_seek,
	dus.last_user_scan,
	dus.last_system_scan,
	dus.last_user_lookup,
	dus.last_system_lookup,
	ix.[object_id],
	ix.index_id,
	COUNT(partition_number)		AS 'partition_count',
	ISNULL(cte.partition_scheme_name,'') AS 'partition_scheme_name',
	su.TableCreateDate,
	su.TableLastModified,
	dos.singleton_lookup_count,
	dos.forwarded_fetch_count,
	dos.lob_fetch_in_pages,
	dos.lob_fetch_in_bytes,
	dos.row_lock_count,
	dos.row_lock_wait_count,
	dos.row_lock_wait_in_ms,
	dos.page_latch_wait_count,
	dos.page_io_latch_wait_count
FROM [sys].[indexes] AS ix
	INNER JOIN [sys].[partitions] AS sp
		ON ix.OBJECT_ID = sp.OBJECT_ID
			AND ix.index_id = sp.index_id
	INNER JOIN [sys].[tables] AS st
		ON ix.OBJECT_ID = st.OBJECT_ID
	LEFT OUTER JOIN indexCTE AS cte
		ON ix.data_space_id = cte.data_space_id
	LEFT OUTER JOIN DMV_OperationalStats AS dos
		ON dos.index_id = ix.index_id
			AND dos.object_id = ix.object_id
	INNER JOIN DMV_UsageStats AS dus
		ON ix.name = dus.index_name
	LEFT OUTER JOIN Fragmentation AS fr
		ON fr.objectID = IX.OBJECT_ID
			AND fr.databaseName = dus.database_name
			AND fr.indexID = ix.index_id
	LEFT OUTER JOIN SpaceUsed AS su
		ON fr.objectid = su.object_id
			AND su.index_id = fr.indexID
	CROSS JOIN HardCodedIndexes AS h
WHERE ix.[object_id] = ISNULL(@objectID, ix.[object_id])
GROUP BY st.name,
ISNULL(ix.name, ''),
ix.[object_id],
ix.index_id,
CAST(CASE WHEN ix.index_id = 1 THEN 'clustered' WHEN ix.index_id = 0 THEN 'heap' ELSE 'nonclustered' END
	+ CASE WHEN ix.ignore_dup_key <> 0 THEN ', ignore duplicate keys' ELSE '' END
	+ CASE WHEN ix.is_unique <> 0 THEN ', unique' ELSE '' END
	+ CASE WHEN ix.is_primary_key <> 0 THEN ', primary key' ELSE '' END AS VARCHAR(210)),
ISNULL(cte.partition_scheme_name, ''),
ISNULL(cte.partition_function_name, ''),
dus.database_name,
dus.user_seeks,
dus.last_user_seek,
dus.last_system_seek,
dus.user_scans,
dus.last_user_scan,
dus.last_system_scan,
dus.user_lookups,
dus.last_user_lookup,
dus.last_system_lookup,
dus.user_updates,
dos.range_scan_count,
dos.singleton_lookup_count,
dos.forwarded_fetch_count,
dos.lob_fetch_in_pages,
dos.lob_fetch_in_bytes,
dos.row_lock_count,
dos.row_lock_wait_count,
dos.row_lock_wait_in_ms,
dos.page_latch_wait_count,
dos.page_io_latch_wait_count,
fr.fragmentation,
fr.partitionNumber,
fr.page_count,
su.[Reserved(KB)],
su.[Used(KB)],
su.TableCreateDate,
su.TableLastModified,
h.HardCodedIndexes
ORDER BY dus.user_seeks ASC,
dus.user_scans ASC,
dus.user_lookups ASC,
CAST(CASE WHEN ix.index_id = 1 THEN 'clustered' WHEN ix.index_id = 0 THEN 'heap' ELSE 'nonclustered' END +
     CASE WHEN ix.ignore_dup_key <> 0 THEN ', ignore duplicate keys' ELSE '' END +
    CASE WHEN ix.is_unique <> 0 THEN ', unique' ELSE '' END +
     CASE WHEN ix.is_primary_key <> 0 THEN ', primary key' ELSE '' END AS VARCHAR(210)) DESC,
fr.page_count DESC
OPTION (MAXDOP 1);

The first step is to verify you have the correct MySQL drivers on your development machine. I used 5.0.5. You can download the necessary drivers for free from mysql.com.

Now create a solution and your SSIS package. Add a MSSQL connection, then a MySQL connection. To create a MySQL connection, right click in the Connection Managers pane and choose New ADO.NET Connection and then click New. In the Provider dropdown you should have an option for MySQL Data Provider in the .Net Providers tree. If not, you’ll need to add a MySQL Connector to Visual Studio. Go to the mysql.com download page, find the section for Connector/Net and download the database driver. After installation you’ll need to close and re-open Visual Studio for the connector to show.

I named my connection MySQL. In the image below you can see the MySQL Data Provider connection and the fields I filled in: Database, Server, Password, and User ID.

Now we will add two variables: one for the last LogID in our destination log table and another to assist with dynamically building the SQL command that we will send to the source MySQL database. The scope is the entire package and not one of the steps. I also added an Execute SQL task and a Data Flow task in my Control Flow that you can see to the right; they’re named so that their purpose should be fairly obvious See the image below:

Double click the execute SQL task “Get last LogID” in the Control Flow window so we can begin filling out the task. In the Execute SQL Task Editor window that pops up you’ll need to go to General and match the settings in the image below. As you can see my SQLStatement is a simple select that gets the most recent id from my MSSQL log table.

Once you’ve set General skip down to Result Set and assign the output from your SQL statement to the MaxLogID variable you created earlier. You’ll need to click Add and choose your MaxLogID in the Variable Name dropdown. The image below shows how my Result Set window looks.

Next we’ll set up our data flow task “Copy MySQL data to MSSQL database”. Double click it in the Control Flow window and add ADO NET Source from Data Flow Sources and OLD DB Destination from Data Flow Destinations. We’ll assume instructions for associating the MSSQL connection to the OLE DB Destination step aren’t needed. Setting up the ADO NET Source is trickier.

First we set the ADO.NET connection manager to point to our MySQL connection. Data access mode should be “SQL command”. Last, you’ll need to put your MySQL query into the SQL command text. I aliased the table as l and every column must be surrounded by grave accents (to the left of the 1 key on a QWERTY keyboard).  I ran into a DATETIME conversion issue where the date in MySQL was stored as UNIXTIME and I wanted to use DATETIME so I’ve shown that conversion in my query as well. Now, for your WHERE clause, you’ll need to choose a valid entry. I chose 0, which you’ll see is surrounded by standard single quotes. Note: this is not what will execute at runtime.  See below for my settings.

Finally the the most difficult task: building the query with the MaxLogID parameter. Return to the Control Flow tab and open your SQLCommand variable. Your value should be your MySQL query all the way up to but not including your variable. It must be contained in a single line with no carriage returns.

Next, single click the “Copy MySQL data to MSSQL database” task and look in your Properties window. Scroll to and click Expressions.

Click the ellipsis with three periods in it and a Property Expressions Editor should pop up. Choose [ADO NET Source].[SqlCommand] for Property. Now we will build our expression. Click the far right ellipsis to bring up the Expression Builder. Expand the Variables tree and drag-and-drop User::SQLCommand into the Expression window.

This is how we will append the rest of the query to what we set the SQLCommand variable to earlier. Next, add a single quote to the right of your expression (addition symbol, space, double quote, space, single quote,double quote, space, addition symbol).  Since we are building a string to execute we’ll need to cast our MaxLogID variable as a string. Expand Type Casts and choose (DT_STR, <<length>>, <<code_page>>). Drag and drop it to the Expression window then fill in an appropriate length you expect your variable to be. 1252 is standard for <<code_page>>.  Now add your MaxLogID variable by dragging and dropping it from the Variables tree. Last, append another single quote to finish out the query. Below is how my finished expression appears.

That’s it! One other issue I ran into was that my package was timing out when validating into the MySQL server, so I set DelayValidation to True.

(I set statistics on so I could easily compare this to other solutions)

SET STATISTICS IO ON
SET NOCOUNT ON
GO

DECLARE @VarcharList NVARCHAR(256) = '100,101,102,103,104,105,106,107'
DECLARE @DestinationTable TABLE(ID TINYINT)

-- Caveat: IDList string must be comma delimited and end with ',' e.g. '111,222,333,'

IF RIGHT(@VarcharList,1) <> ','
BEGIN
SET @VarcharList = @VarcharList + ','
END

-- Break out IDs from @IDList into @DestinationTable
WHILE 1=1
BEGIN
DECLARE @CommaLocation INT = CHARINDEX(',',@VarcharList)

INSERT INTO @DestinationTable(ID) SELECT LEFT(@VarcharList,@CommaLocation-1)

SET @VarcharList = RIGHT(@VarcharList,(LEN(@VarcharList)-@CommaLocation))

IF LEN(@VarcharList) = 0

BREAK
END

SELECT * FROM @DestinationTable

SET STATISTICS IO OFF
GO

Your results should be as follows:

And here is my statistics output:

I’ve not tried this on a set more then 100 records or so and cannot attest to this working well for large data sets.

IF OBJECT_ID('tempdb.dbo.#TestData','U') IS NOT NULL DROP TABLE #TestData;
CREATE TABLE #TestData(
ID INT IDENTITY(1,1) PRIMARY KEY NOT NULL,
Glaven VARCHAR(100))

INSERT INTO #TestData(Glaven)
SELECT NEWID() UNION
SELECT NEWID() UNION
SELECT NEWID() UNION
SELECT NEWID() UNION
SELECT NEWID()

SELECT ID, Glaven
FROM #TestData
WHERE ABS(ID) % 2 = 1 -- Select only Odds!

SELECT ID, Glaven
FROM #TestData
WHERE ABS(ID) % 2 = 0 -- Select only Evens!

First, let’s insert 10,000 rows to a temporary table. I’m fond of using the NEWID() function.

IF OBJECT_ID('tempdb.dbo.#TestData','U') IS NOT NULL DROP TABLE #TestData

CREATE TABLE #TestData(
ID INT IDENTITY(1,1) PRIMARY KEY NOT NULL,
data1 VARCHAR(100),
data2 VARCHAR(12))

SET NOCOUNT ON
GO

DECLARE @LoopIteration INT = 1

WHILE @LoopIteration < 10001
BEGIN

INSERT INTO #TestData(data1)
SELECT NEWID()

SET @LoopIteration = @LoopIteration + 1

END

Now that we have test data, let’s run an update in a while loop and monitor it’s progress.

DECLARE @LoopIteration INT = 1

DECLARE @RowsToProcess CHAR(5) = CONVERT(VARCHAR,(SELECT COUNT(0) FROM #TestData))

DECLARE @Data1 VARCHAR(100) = (SELECT data1 FROM #TestData WHERE ID = @LoopIteration)

DECLARE @MaxIteration INT = (SELECT COUNT(0) FROM #TestData) + 1

DECLARE @Msg VARCHAR(256) = '[' + CAST(@LoopIteration AS VARCHAR) + '/' + @RowsToProcess + ']' + ' Processing ' + @Data1 + ' at ' + CONVERT(CHAR(8),GETDATE(),14)

WHILE @LoopIteration < @MaxIteration
BEGIN

RAISERROR(@Msg, 0, 1) WITH NOWAIT

UPDATE #TestData
SET data2 = REVERSE(LEFT(data1,12)) -- let's have some fun!
WHERE ID = @LoopIteration

SET @LoopIteration = @LoopIteration + 1;
SET @Data1 = (SELECT data1 FROM #TestData WHERE ID = @LoopIteration)
SET @Msg = '[' + CAST(@LoopIteration AS VARCHAR) + '/' + @RowsToProcess + ']' + ' Processing ' + (SELECT data1 FROM #TestData WHERE ID = @LoopIteration) + ' at ' + CONVERT(CHAR(8),GETDATE(),14)

END

Voila! You should see detailed progress in your Messages window. This script is flexible, and I’ve added @@ROWCOUNT and other relevant tidbits for different use cases. Below is the output from the script:

First, let’s create a table and insert some fake numeric data using a fictional database named KSQL01.

USE [KSQL01]
GO

CREATE TABLE dbo.scratch_bcp_testing(ID INT)

INSERT INTO dbo.scratch_bcp_testing(ID)
SELECT CONVERT(INT,RAND()*1000) UNION
SELECT CONVERT(INT,RAND()*1000) UNION
SELECT CONVERT(INT,RAND()*1000) UNION
SELECT CONVERT(INT,RAND()*1000) UNION
SELECT CONVERT(INT,RAND()*1000)

Our data should look something like this:

Next, let’s export the data to a .data file on a destination server. For your destination you can choose the SQL server you’re developing with, your local PC, or another SQL server entirely.

-- Export via SSMS or a stored procedure
EXEC master.dbo.xp_cmdshell 'bcp "KSQL01.dbo.scratch_bcp_testing" out "c:\\SourceData.data" -E -k -c -t"\t" -T -SSOURCESQLSERVER' >> "c:\logfile.log"'

I’ve bolded the strings you’ll need to replace with your own data. You could end the statement after the SOURCESERVER as exporting a log of actions taken is optional. If this is ran in SSMS, the results window displays the same data that is exported in the .log file. The .log file is useful to have for troubleshooting if this code is called multiple times in a stored procedure.

When we run this in SSMS, it should export two files, the .data and the .log file to the path specified. Both can be opened in a text editor.

Now we can reverse the process and import the data back to the table.

-- Import .data file back into the table

EXEC master.dbo.xp_cmdshell 'bcp "KSQL01.dbo.scratch_bcp_testing" in "\\DESTINATIONSERVER\c$\SourceData.data" -k -c -t"\t" -T -SSOURCESERVER'

And there you have it! And old school alternative to SSIS on the off chance you need it.