Dynamically selected columns with column switch option

Hi,
my requirement is to dynamically select certain columns from a table
and depending on a flag some columns must be swapped. This is easier
explained with the following statements one can paste into Query
Analyzer:
declare @.sql nvarchar(4000), @.cols nvarchar (500)
set @.cols = '
v1 = case when sel=1 then
v1a else v1b end,
v2 = case when sel=1 then v2a
else v2b end
'
create table ##temp_test (
v1a int,
v1b int,
v2a varchar(15),
v2b varchar(15),
sel bit
)
insert into ##temp_test (v1a, v1b, v2a, v2b, sel)
values (1, 2, 'a1', 'b1', 0)
insert into ##temp_test (v1a, v1b, v2a, v2b, sel)
values (3, 4, 'a2', 'b2', 1)
set @.sql = 'select ' + @.cols + 'from
##temp_test'
exec sp_executesql @.sql
drop table ##temp_test
Result:
v1 v2
-- --
2 b1
3 a2
This fits bots requirements: @.cols (which is stored in a table in the
real world application) holds the column names to be selected, and it
holds it in a way which also enables to switch between the a and b
version of the columns via the CASE statements. The performance is OK.
My problem: the real world query is a lot more complex, ##temp_test is
actually a table with about 80 columns and there are a lot of other
tables joined in.
Since I am doomed to not use more than 4000 characters for the dynamic
sql part (not using sp_executesql results in a huge performance
penalty in my scenario) this approach in the end works find without
all the CASE-ELSE-END stuff, but when I insert the CASE stuff, the
@.cols variable length will grow by factor 2-3, typically from around
1000 - 2000 chars to about 2000 - 6000 chars.
That's the end of the sp_executesql approach.
Maybe our design is wrong in the first place. We have tried to
alternate between the columns by using UNION to a View that contains
the alternate column, but ended up in quiet a performance hit with
more complex queries plus sometimes the SORT function would not work
anymore and the like.
Maybe I can shorten the
v1 = case when sel=1 then
v1a else v1b end
part somehow? Or in the best case someone knows a similar approach...
just better.
TIA for any comments!
Regards
DChttp://www.sommarskog.se/dynamic_sql.html
HTH,
Vinod Kumar
MCSE, DBA, MCAD, MCSD
http://www.extremeexperts.com
Books Online for SQL Server SP3 at
http://www.microsoft.com/sql/techin.../2000/books.asp
"DC" <dc@.upsize.de> wrote in message
news:5b7bac12.0502250228.4d94de6c@.posting.google.com...
> Hi,
> my requirement is to dynamically select certain columns from a table
> and depending on a flag some columns must be swapped. This is easier
> explained with the following statements one can paste into Query
> Analyzer:
>
> declare @.sql nvarchar(4000), @.cols nvarchar (500)
> set @.cols = '
> v1 = case when sel=1 then
> v1a else v1b end,
> v2 = case when sel=1 then v2a
> else v2b end
> '
> create table ##temp_test (
> v1a int,
> v1b int,
> v2a varchar(15),
> v2b varchar(15),
> sel bit
> )
> insert into ##temp_test (v1a, v1b, v2a, v2b, sel)
> values (1, 2, 'a1', 'b1', 0)
> insert into ##temp_test (v1a, v1b, v2a, v2b, sel)
> values (3, 4, 'a2', 'b2', 1)
> set @.sql = 'select ' + @.cols + 'from
> ##temp_test'
> exec sp_executesql @.sql
> drop table ##temp_test
>
> Result:
> v1 v2
> -- --
> 2 b1
> 3 a2
> This fits bots requirements: @.cols (which is stored in a table in the
> real world application) holds the column names to be selected, and it
> holds it in a way which also enables to switch between the a and b
> version of the columns via the CASE statements. The performance is OK.
> My problem: the real world query is a lot more complex, ##temp_test is
> actually a table with about 80 columns and there are a lot of other
> tables joined in.
> Since I am doomed to not use more than 4000 characters for the dynamic
> sql part (not using sp_executesql results in a huge performance
> penalty in my scenario) this approach in the end works find without
> all the CASE-ELSE-END stuff, but when I insert the CASE stuff, the
> @.cols variable length will grow by factor 2-3, typically from around
> 1000 - 2000 chars to about 2000 - 6000 chars.
> That's the end of the sp_executesql approach.
> Maybe our design is wrong in the first place. We have tried to
> alternate between the columns by using UNION to a View that contains
> the alternate column, but ended up in quiet a performance hit with
> more complex queries plus sometimes the SORT function would not work
> anymore and the like.
> Maybe I can shorten the
> v1 = case when sel=1 then
> v1a else v1b end
> part somehow? Or in the best case someone knows a similar approach...
> just better.
> TIA for any comments!
> Regards
> DC|||I don't understand why you want to use dynamic SQL for this. Where is
the string in @.cols generated from? You say it comes from a table but
why put it in a table at all? Why can't you define views for these
different views of the data?
David Portas
SQL Server MVP
--|||>> my requirement is to dynamically select certain columns from a table
and depending on a flag some columns must be swapped. <<
You might want to get any book on **basic software engineering** and
look up the concept of cohesion in a code module.|||Thank you! Can you recommend a good book on this topic? I probably
focussed too much on the loose coupling aspects? I support your claim
for the art of programming, but then again: it's only T-SQL. Hacky
stuff! Wouldn't a real programmer avoid messing around with SQL anyway?|||Thanks David, views or pre-generated stored procs are an option. We are
in the process of upgrading a life application though, and wanted to
integrate this additional requirement (the CASE req.) with the least
effort possible.|||Thank you Vinod, great article with a wealth of dynamic sql info. Does
not provide an exact solution, but pointed out that using EXEC shoud
not make as much of a difference as I experienced.|||> Wouldn't a real programmer avoid messing around with SQL anyway?
Sure. Why go looking for a new peg when you can just keep bashing that
square one into a round hole? After all, tables are only arrays aren't
they?
David Portas
SQL Server MVP
--|||On 25 Feb 2005 05:52:23 -0800, David Portas wrote:

>Sure. Why go looking for a new peg when you can just keep bashing that
>square one into a round hole? After all, tables are only arrays aren't
>they?
Hi David,
A "real" programmer - isn't that a guy who needs nothing but 8 switches
and 8 LED's to program the 'puter? Who needs all that modern assembler
language mnemonics anyway, huh?
(Hmmm - am I showing my age now <g> )
Best, Hugo
--
(Remove _NO_ and _SPAM_ to get my e-mail address)|||>> Can you recommend a good book on this topic? <<
Any of the classic by Yourdon, Constantine, DeMarco or Gane & Sarson
are a good place to start.
They would avoid dynamic SQL and do the order of presentation of the
columns in the front end. I have become a fan of the idea that one
team in the shop handles the database and writes all the SQL for
everyone, and the application developers make requests to that team.

Dynamically create text file as destination from sql script in SSIS

I have a select Script as follows:

SELECT c.ABC AS 'ABC'

, a.Qty AS 'Quantity_Recived'

, b.PC AS 'PC'

, b.PC AS 'PC'

, 'I' AS 'Flag'

FROM TNRInventory.dbo.tInventoryAlloc AS a

LEFT OUTER JOIN vwInventoryAllocMapping AS vwMap ON a.TNRAllocTypeID = vwMap.TNRInventoryAllocID

LEFT OUTER JOIN ABC.dbo.ZREFRESHTAB AS b ON a.DispenserID = b.Asset

LEFT OUTER JOIN ABC.dbo.TableJoinKey AS c ON a.TitleID = c.TITLE_ID

WHERE (vwMap.DataSourceID = 3) and vwMap.[DataSourceAllocName] = 'I'

group by c.SKU_NO , vwMap.[DataSourceAllocName],a.Qty , b.Profit_Center

order by c.SKU_NO,vwMap.[DataSourceAllocName]

GO

i have to send the result of aforesaid script in batch of 300 records per file (tab delimited text file)
now the file name must be dynamically created as each file will contain 300 records.

I have found some document related to same issue on this url

http://forums.microsoft.com/TechNet/ShowPost.aspx?PostID=1238184&SiteID=17

but still there is a catch.

Can any one guide/suggest me better way to do the aforesaid.

Thanks

Thinking while typing, this could be done with a few steps.

1 - Data Flow - Load a staging table with the results of the SQL. Add to it a row number counter so that every row is numbered uniquely.

2 - Control Flow - Run an Execute SQL Task to select max(rownumber) from that staging table

3 - Control Flow - Use a script task to take the output of the Execute SQL task and populate another variable with the number of iterations needed to populate files with up to 300 rows. (max(rownumber) / 300 - if no remainder, use that value, if remainder add one to the integer, etc...)

4 - Control Flow - Use a for loop to iterate the output variable from step 3 above.

5 - Control Flow - Build a variable set to an expression to use a base filename and the variable from step 3 above.

6 - Control Flow For Loop - Loop through the variable from step 3 above and add a data flow. Inside this data flow, use an OLE DB (or whatever) to connect to the staging table from step 1, filtering by (variable_step3 * 300) you can select just the records for this iteration. Hook that up to a flat file destination, which uses expressions to set the file name equal to the variable from step 5 above.

Something like that.|||

You might also check Jamie's blog post on this topic.

http://blogs.conchango.com/jamiethomson/archive/2005/12/04/SSIS-Nugget_3A00_-Splitting-a-file-into-multiple-files.aspx

Essentially the same technique that Phil recommended, but he's got some sample code already

|||

jwelch wrote:

You might also check Jamie's blog post on this topic.

http://blogs.conchango.com/jamiethomson/archive/2005/12/04/SSIS-Nugget_3A00_-Splitting-a-file-into-multiple-files.aspx

Essentially the same technique that Phil recommended, but he's got some sample code already

D'oh! Should've just looked there first!|||

What i did is i create a view for aforesaid script and then used that script in the following VB.net script

' Microsoft SQL Server Integration Services Script Task

' Write scripts using Microsoft Visual Basic

' The ScriptMain class is the entry point of the Script Task.

Imports System

Imports System.Data

Imports System.Math

Imports Microsoft.SqlServer.Dts.Runtime

Imports System.Data.SqlClient

Imports System.Text

Imports System.IO

Public Class ScriptMain

Public Sub Main()

Dim ConnectionString As String = "Server=ServerName;Database=DatabaseName;uid=Login;pwd=Password;"

Dim querystring As String = "SELECT * FROM vwInventory_I"

Dim con As New SqlConnection(ConnectionString)

Dim adapter As New SqlDataAdapter()

Dim ds As New DataSet

Dim dt As New DataTable

adapter.SelectCommand = New SqlCommand(querystring, con)

adapter.Fill(ds)

dt = ds.Tables(0)

CreateFile(dt)

End Sub

Private Sub CreateFile( ByVal table As DataTable)

Dim intRowCount As Integer = table.Rows.Count

Dim Column1Value As String

Dim Column2Value As String

Dim Column3Value As String

Dim Column4Value As String

Dim Column5Value As String

Dim sb As New StringBuilder

Dim seperator As String = vbTab

Dim nFileNameCount As Integer

nFileNameCount = 0

'For i As Integer = 0 To table.Columns.Count - 1

Dim i As Integer = 0

Dim j As Integer = 0

For j = 0 To intRowCount - 1

If nFileNameCount = 100 Then

PrintFile(sb, j)

'clears the stringbuilder

sb.Remove(0, sb.ToString.Length - 1)

nFileNameCount = 0

End If

Column1Value = table.Rows(j)(i).ToString

Column2Value = table.Rows(j)(i + 1).ToString

Column3Value = table.Rows(j)(i + 2).ToString

Column4Value = table.Rows(j)(i + 3).ToString

Column5Value = table.Rows(j)(i + 4).ToString

sb.Append(Column1Value & seperator & Column2Value & seperator & Column3Value & seperator & _

Column4Value & seperator & Column5Value & seperator & vbCrLf)

nFileNameCount = nFileNameCount + 1

Next

PrintFile(sb, j)

'Next

End Sub

Private Sub PrintFile( ByVal sb As StringBuilder, ByVal recordcount As Integer)

' Create an instance of StreamWriter to write text to a file.

Dim strDate As String

strDate = String.Format("{0:yyyy}" , DateTime.Now)

strDate = strDate & String.Format( "{0:MM}", DateTime.Now)

strDate = strDate & String.Format( "{0d}", DateTime.Now)

Using sw As StreamWriter = New StreamWriter("C:\SSIS\I_" & strDate & "_" & recordcount & ".txt")

sw.WriteLine(sb.ToString)

sw.Close()

End Using

'this will create a text file in bin directory

End Sub

End Class

This is how you generate tab delimited files to test check Use Script Tas from SSIS

|||Uh, okay. Good. So why even use SSIS then? Write your own program as you have done, compile it, and execute the resulting binary file. Leave the bloat of SSIS out of it.|||

I am going to put this piece in middle of my design.

It was the one of the key to finish my jik-so-puzzle.

Other one is to delete the files from Unix Aix Server with FTP connection.

And here too I have to go all the way round, as there FTP connection works fine on Microsoft server but not on UNIX server.

It does not allow you to delete the files on UNIX server.

That is the reason I am using the code in SSIS.