Showing posts with label point. Show all posts
Showing posts with label point. Show all posts

Monday, March 26, 2012

Hilary Cotters cleanup script

Hi,
Can someone point me to where I can find a copy of Hilary's replication celanup script?
Thanks,
Andrew
go to ava.co.uk and look in the technical resources section.
There is a version of it there. I'll try to post another one soon that has a
correction for non dbo objects.
"Andrew" <Andrew@.discussions.microsoft.com> wrote in message
news:21679998-9A1D-4645-BD94-93EFC264C7BF@.microsoft.com...
> Hi,
> Can someone point me to where I can find a copy of Hilary's replication
celanup script?
> Thanks,
> Andrew

Friday, March 23, 2012

High reliability == single point of failure?

I am trying to learn about high-reliability clusters (for IIS and Sql in my
case) but there is a fundamental thing I don't understand about the approach
that keeps me from swallowing the high-reliability kool-aid:
Currently, I have a "non-high-reliability" (low-reliability?) system --
i.e., one server that handles IIS, Sql, and data store all at once. This
server uses RAID 1 and has redundant power supplies. Let's assume that the
probability of catastrophic failure of this server is P.
Now I move to a high-reliability solution. Microsoft suggests at least two
machines handling IIS in an active/active configuration, two machines
handling Sql in an active/passive configuration, and one SAN-style data
store running RAID 1 (at minimum) with redundant power supplies. For the
sake of argument, let's assume that all five of these new boxes have similar
complexity both to each other and to the server in my original
low-reliability system, and therefore that the probability of individual
failure of any of these servers is also P.
The weakness of my original low-reliability system is that there is a single
point of failure (the server) which may fail catastrophically with
probability P. But my new high-reliability system also has a single point of
failure (the storage disk array) which also may fail catastrophically with
probability P. What exactly have I gained?
Michael Carr
Most SAN storage arrays have multiple controllers, multiple SAN switches
and multiple disks configured in a redundant RAID array.
Therefore, for your storage array to completely fail, you would need
multiple components of it to fail in order for you to completely lose it.
So, in your example, although the storage array may seem a single point of
failure, due to its redundant nature, it is far less likely to fail
completely than any of your other components.
HTH
Lee
"Michael Carr" <mcarr@.umich.edu> wrote in message
news:uX2oCtewFHA.2792@.tk2msftngp13.phx.gbl...
>I am trying to learn about high-reliability clusters (for IIS and Sql in my
>case) but there is a fundamental thing I don't understand about the
>approach that keeps me from swallowing the high-reliability kool-aid:
> Currently, I have a "non-high-reliability" (low-reliability?) system --
> i.e., one server that handles IIS, Sql, and data store all at once. This
> server uses RAID 1 and has redundant power supplies. Let's assume that the
> probability of catastrophic failure of this server is P.
> Now I move to a high-reliability solution. Microsoft suggests at least two
> machines handling IIS in an active/active configuration, two machines
> handling Sql in an active/passive configuration, and one SAN-style data
> store running RAID 1 (at minimum) with redundant power supplies. For the
> sake of argument, let's assume that all five of these new boxes have
> similar complexity both to each other and to the server in my original
> low-reliability system, and therefore that the probability of individual
> failure of any of these servers is also P.
> The weakness of my original low-reliability system is that there is a
> single point of failure (the server) which may fail catastrophically with
> probability P. But my new high-reliability system also has a single point
> of failure (the storage disk array) which also may fail catastrophically
> with probability P. What exactly have I gained?
> Michael Carr
>
|||Lets focus on the storage array first. A standard SCSI enclosure runs on a
single backplane, single controller system. You can run a multi-channel
controller card to two enclosures set up with RAID 1+0 across the enclosures
but you are still on a single controller. A mid-range SAN typically has two
internal controllers designed to check and supplement each other. You also
have multiple attachments (HBAs) from the host computer to the SAN, multiple
power supplies, and multiple enclosure racks. Basically, a well-designed
SAN implementation seems like a single array but in reality the only
non-redundant part of the SAN is the sheet metal enclosure. Those have a
very low failure rate once they are installed.
You can still drop your main production database through operator error, but
that is where processes and procedure either help or hurt your availability.
BTW, clustering IIS servers is a waste of time and money. Those are
typically set up as a pool of stateless servers where computers can be
dropped offline or brought online and the worst a user sees is a slow
response or maybe a retry. The idea of dedicating a server to each task is
for scalability, stability, and managability.
Geoff N. Hiten
Senior Database Administrator
Microsoft SQL Server MVP
"Michael Carr" <mcarr@.umich.edu> wrote in message
news:uX2oCtewFHA.2792@.tk2msftngp13.phx.gbl...
>I am trying to learn about high-reliability clusters (for IIS and Sql in my
>case) but there is a fundamental thing I don't understand about the
>approach that keeps me from swallowing the high-reliability kool-aid:
> Currently, I have a "non-high-reliability" (low-reliability?) system --
> i.e., one server that handles IIS, Sql, and data store all at once. This
> server uses RAID 1 and has redundant power supplies. Let's assume that the
> probability of catastrophic failure of this server is P.
> Now I move to a high-reliability solution. Microsoft suggests at least two
> machines handling IIS in an active/active configuration, two machines
> handling Sql in an active/passive configuration, and one SAN-style data
> store running RAID 1 (at minimum) with redundant power supplies. For the
> sake of argument, let's assume that all five of these new boxes have
> similar complexity both to each other and to the server in my original
> low-reliability system, and therefore that the probability of individual
> failure of any of these servers is also P.
> The weakness of my original low-reliability system is that there is a
> single point of failure (the server) which may fail catastrophically with
> probability P. But my new high-reliability system also has a single point
> of failure (the storage disk array) which also may fail catastrophically
> with probability P. What exactly have I gained?
> Michael Carr
>
|||"Michael Carr" <mcarr@.umich.edu> wrote in message
news:uX2oCtewFHA.2792@.tk2msftngp13.phx.gbl...
>I am trying to learn about high-reliability clusters (for IIS and Sql in my
>case) but there is a fundamental thing I don't understand about the
>approach that keeps me from swallowing the high-reliability kool-aid:
> Currently, I have a "non-high-reliability" (low-reliability?) system --
> i.e., one server that handles IIS, Sql, and data store all at once. This
> server uses RAID 1 and has redundant power supplies. Let's assume that the
> probability of catastrophic failure of this server is P.
> Now I move to a high-reliability solution. Microsoft suggests at least two
> machines handling IIS in an active/active configuration, two machines
> handling Sql in an active/passive configuration, and one SAN-style data
> store running RAID 1 (at minimum) with redundant power supplies. For the
> sake of argument, let's assume that all five of these new boxes have
> similar complexity both to each other and to the server in my original
> low-reliability system, and therefore that the probability of individual
> failure of any of these servers is also P.
> The weakness of my original low-reliability system is that there is a
> single point of failure (the server) which may fail catastrophically with
> probability P. But my new high-reliability system also has a single point
> of failure (the storage disk array) which also may fail catastrophically
> with probability P. What exactly have I gained?
P, in your case is the Probability of your non-HA environment failing. P',
in this case is the probability of the SAN array failing.
P' is extremely low in that all internal channels of a SAN have multiple
paths, all drives configured with in the SAN are RAID protected at some
level or another, all power supplies and fans are highly redundant.
Basically, in a high-end SAN, it would take several failed components to
cause a failure of the SAN itself. Since the SAN is so vital, it will
normally have monitoring solutions keeping an eye on it, and in most cases,
a modem that dials the vendor to report any failure or prefailure
conditions.
So, the probability of P is much higher than the probability of P'.
Russ Kaufmann
MVP - Windows Server - Clustering
http://www.clusterhelp.com - Cluster Website
http://msmvps.com/clusterhelp - New Blog
http://spaces.msn.com/members/russkaufmann - Old Blog
|||"Geoff N. Hiten" <SRDBA@.Careerbuilder.com> wrote in message
news:OZ4pC$fwFHA.3756@.tk2msftngp13.phx.gbl...

> BTW, clustering IIS servers is a waste of time and money. Those are
> typically set up as a pool of stateless servers where computers can be
> dropped offline or brought online and the worst a user sees is a slow
> response or maybe a retry.
I may not agree with Geoff on this subject depending on the definition of
the cluster referred to in the original post. If Geoff interpretted
Michael's post to mean IIS would be configured in a server cluster using
MSCS, then I agree with him. It is a waste of time and money.
However, if Michael meant NLB clustering, then I do not agree with Geoff on
this subjet. IIS with NLB is a very good solution in that it does
1. Provide horizontal scaling of the application front end.
2. Provides high availability in that users can fail over to a surviving
node in the NLB cluster in the event of a node failure.
Applications that do require state information often maintain that state
using the SQL database or cookies on the client side.
Clustering IIS server through NLB is not a waste of time or money if your
business depends on the availability of your web based applications.
Russ Kaufmann
MVP - Windows Server - Clustering
http://www.clusterhelp.com - Cluster Website
http://msmvps.com/clusterhelp - New Blog
http://spaces.msn.com/members/russkaufmann - Old Blog
|||I was referring to Clustering IIS using MSCS as a waste of time and money.
IIS scale-out shoud be done with NLB clustering or any other load-balancing
system. I am indifferent as to whether NLB or a third-party solution is
used. As always, which method is best for you will depend on your exact
situation and requirements. I have used NLB in the past and have found it
quite useful.
Geoff N. Hiten
Senior Database Administrator
Microsoft SQL Server MVP
"Russ Kaufmann [MVP]" <russ@.exchangemct.com> wrote in message
news:eXxH1RswFHA.2656@.TK2MSFTNGP09.phx.gbl...
> "Geoff N. Hiten" <SRDBA@.Careerbuilder.com> wrote in message
> news:OZ4pC$fwFHA.3756@.tk2msftngp13.phx.gbl...
>
> I may not agree with Geoff on this subject depending on the definition of
> the cluster referred to in the original post. If Geoff interpretted
> Michael's post to mean IIS would be configured in a server cluster using
> MSCS, then I agree with him. It is a waste of time and money.
> However, if Michael meant NLB clustering, then I do not agree with Geoff
> on this subjet. IIS with NLB is a very good solution in that it does
> 1. Provide horizontal scaling of the application front end.
> 2. Provides high availability in that users can fail over to a surviving
> node in the NLB cluster in the event of a node failure.
> Applications that do require state information often maintain that state
> using the SQL database or cookies on the client side.
> Clustering IIS server through NLB is not a waste of time or money if your
> business depends on the availability of your web based applications.
>
> --
> Russ Kaufmann
> MVP - Windows Server - Clustering
> http://www.clusterhelp.com - Cluster Website
> http://msmvps.com/clusterhelp - New Blog
> http://spaces.msn.com/members/russkaufmann - Old Blog
>
sql

Friday, March 9, 2012

High Availability

Does anyone have a good starting point for creating a "high availabilty"
solution. I'm working on designing a solution that will allow for us to have
a primary site in Orlando and a secondary site in another state. The system
needs to have data moving in almost real time between the master and
secondary site. I've looked into the sql "active-passive" setup as well as
CA's HA solution, but would like input from someone whose accomplished
something similar.
Thanks,
GTM.
Have you looked at this white paper yet?
http://www.microsoft.com/windowsserv...lustergeo.mspx
Cheers,
Rod
MVP - Windows Server - Clustering
http://www.nw-america.com - Clustering
"Greg" <yaya@.yaya.com> wrote in message
news:eLulS41VEHA.2168@.TK2MSFTNGP10.phx.gbl...
> Does anyone have a good starting point for creating a "high availabilty"
> solution. I'm working on designing a solution that will allow for us to
have
> a primary site in Orlando and a secondary site in another state. The
system
> needs to have data moving in almost real time between the master and
> secondary site. I've looked into the sql "active-passive" setup as well as
> CA's HA solution, but would like input from someone whose accomplished
> something similar.
> Thanks,
> GTM.
>
|||Note that one of the considerations for the active/passive GEO Cluster is that the network connections must be able to provide a maximum guaranteed round trip latency between nodes of no more than 500
milliseconds. The cluster uses heartbeat to detect whether a node is alive or not responding. These heartbeats are sent out on a periodic basis (every 1.2 seconds). If a node takes too long to respond to
heartbeat packets, MSCS starts a heavy-weight protocol to figure out which nodes are really still alive and which ones are dead; this is known as a cluster re-group. The heartbeat interval is not a configurable
parameter for the cluster service (there are many reasons for this, but the bottom line is that changing this parameter can have a significant impact on the stability of the cluster and the failover time). . 500 ms round-
trip is significantly below any threshold to ensure that artificial re-group operations are not triggered
You may want to research SQL Server Replication/ Log Shipping if you can have some latency (few secs).
Addtional Information:
=====================
http://www.microsoft.com/technet/pro...n/default.mspx
SQL Server 2000 High Availability Series
Plan and deploy a highly available data center that uses SQL Server 2000. This guide covers procedures to help you increase availability of your servers, recover from catastrophic disasters, and prevent and
minimize downtime.
SQL Server 2000 Failover Clustering
Understand how failover clustering works and how to implement, administer, and troubleshoot a failover cluster. Also included: worksheets and checklists to assist you during installation.
Log Shipping in SQL Server 2000 - Part 1
Increase your SQL Server availability by setting up, reconfiguring, and monitoring log shipping, which is supported in both SQL Server 2000 Enterprise Edition and SQL Server 2000 Developer Edition.
Log Shipping in SQL Server 2000 - Part 2
Find out how to change the roles of the primary and secondary servers, how to fully reverse their roles, and where to place the monitor server for the most effective monitoring.
Replication Overview
http://msdn.microsoft.com/library/de...lover_694n.asp
Typical Uses of Replication
http://msdn.microsoft.com/library/de...intro_6trn.asp
How Replication Works
http://msdn.microsoft.com/library/de...intro_6trn.asp
Planning for Each Type of Replication
http://msdn.microsoft.com/library/de...lplan_82zy.asp
Transactional Replication and Log Shipping
http://msdn.microsoft.com/library/de...ackup_3js7.asp
Microsoft SQL Server 2000: The Types of Replication and When to Implement Each
http://support.microsoft.com/default...b;EN-US;818714
Introducing New Features in Microsoft SQL Server 2000 Transactional Replication
http://support.microsoft.com/default...b;en-us;324966
Best Regards,
Uttam Parui
Microsoft Corporation
This posting is provided "AS IS" with no warranties, and confers no rights.
Are you secure? For information about the Strategic Technology Protection Program and to order your FREE Security Tool Kit, please visit http://www.microsoft.com/security.
Microsoft highly recommends that users with Internet access update their Microsoft software to better protect against viruses and security vulnerabilities. The easiest way to do this is to visit the following websites:
http://www.microsoft.com/protect
http://www.microsoft.com/security/guidance/default.mspx
|||another option would be to use an asynchronous replication software that
also does failover.
Async replication is real-time, as opposed to the scheduled Log Shipping
function, which reduces the latency of the data.
And for availability, you can failover similar to a cluster (assuming a
server name, IP and starting services) without the distance limitation.
Check out Double-Take by NSI Software - here is a link to their SQL
whitepaper.
http://www.nsisoftware.com/pro/white/
jason
"Uttam Parui[MS]" <uttamkp@.online.microsoft.com> wrote in message
news:qhGiyIlYEHA.2688@.cpmsftngxa06.phx.gbl...
> Note that one of the considerations for the active/passive GEO Cluster is
that the network connections must be able to provide a maximum guaranteed
round trip latency between nodes of no more than 500
> milliseconds. The cluster uses heartbeat to detect whether a node is alive
or not responding. These heartbeats are sent out on a periodic basis (every
1.2 seconds). If a node takes too long to respond to
> heartbeat packets, MSCS starts a heavy-weight protocol to figure out which
nodes are really still alive and which ones are dead; this is known as a
cluster re-group. The heartbeat interval is not a configurable
> parameter for the cluster service (there are many reasons for this, but
the bottom line is that changing this parameter can have a significant
impact on the stability of the cluster and the failover time). . 500 ms
round-
> trip is significantly below any threshold to ensure that artificial
re-group operations are not triggered
> You may want to research SQL Server Replication/ Log Shipping if you can
have some latency (few secs).
> Addtional Information:
> =====================
>
http://www.microsoft.com/technet/pro...n/default.mspx
> SQL Server 2000 High Availability Series
> Plan and deploy a highly available data center that uses SQL Server 2000.
This guide covers procedures to help you increase availability of your
servers, recover from catastrophic disasters, and prevent and
> minimize downtime.
> SQL Server 2000 Failover Clustering
> Understand how failover clustering works and how to implement, administer,
and troubleshoot a failover cluster. Also included: worksheets and
checklists to assist you during installation.
> Log Shipping in SQL Server 2000 - Part 1
> Increase your SQL Server availability by setting up, reconfiguring, and
monitoring log shipping, which is supported in both SQL Server 2000
Enterprise Edition and SQL Server 2000 Developer Edition.
> Log Shipping in SQL Server 2000 - Part 2
> Find out how to change the roles of the primary and secondary servers, how
to fully reverse their roles, and where to place the monitor server for the
most effective monitoring.
> Replication Overview
>
http://msdn.microsoft.com/library/de...lover_694n.asp
> Typical Uses of Replication
>
http://msdn.microsoft.com/library/de...intro_6trn.asp
> How Replication Works
>
http://msdn.microsoft.com/library/de...intro_6trn.asp
> Planning for Each Type of Replication
>
http://msdn.microsoft.com/library/de...lplan_82zy.asp
> Transactional Replication and Log Shipping
>
http://msdn.microsoft.com/library/de...ackup_3js7.asp
> Microsoft SQL Server 2000: The Types of Replication and When to Implement
Each
> http://support.microsoft.com/default...b;EN-US;818714
> Introducing New Features in Microsoft SQL Server 2000 Transactional
Replication
> http://support.microsoft.com/default...b;en-us;324966
>
> Best Regards,
> Uttam Parui
> Microsoft Corporation
> This posting is provided "AS IS" with no warranties, and confers no
rights.
> Are you secure? For information about the Strategic Technology Protection
Program and to order your FREE Security Tool Kit, please visit
http://www.microsoft.com/security.
> Microsoft highly recommends that users with Internet access update their
Microsoft software to better protect against viruses and security
vulnerabilities. The easiest way to do this is to visit the following
websites:
> http://www.microsoft.com/protect
> http://www.microsoft.com/security/guidance/default.mspx
>

Wednesday, March 7, 2012

Hierarchical XML import?

Hello,

Can anyone point me at a tutorial or sample that shows how to use IS for importing an xml file containing hierarchically arranged records ?

I have a file which contains multiple orders , the orders contain multiple line items.. the file also contains an element with details of the file source etc...

So, I want to make an insert in the FileLog table an then make inserts into the orders table .. then make inserts into the OrderItems table which will have the foreign key from the orders table in the records...

if you get what I mean...

But I have searched hign and low and can't see any info on how to load anything but a very flat xml file structure...

Thanks

Vida.

The XML Source will take a hierarchical XML feed and crack it, giving multiple outputs, one for each level. Have a play.

To handle the FKs you need to enforce order of inserts which cannot be done in a single Data Flow task, so stage the data and use several tasks, one for each successive table with the FK. Raw Files would be the ideal staging area for this, fast and efficient.

|||

Darren,

Am I right in thinkin that SQLIS is just not designed to load anything but a flat file containing a simple set of same-format rows?

I just can't seem to find very much info on loading master-detail records , whether in XML or in any ordinary flat file ....

Now I've been given a task of loading a simple text file... first row is a bit of header info.. then the next ten rows are data rows in a simple csv format... last row is a checksum.

20060518,SUPPLIER1,,
EUR,AUD,1.6869,
GBP,AUD,2.4896,
USD,AUD,0.7568,
DKK,AUD,4.4581,
SEK,AUD,5.6246,
CHF,AUD,0.9246,
JPY,AUD,84.5870,
NZD,AUD,1.2293,
NOK,AUD,4.6809,
SGD,AUD,1.2060,
HKD,AUD,5.9217,
AUD,AUD,1.0000,
CAD,AUD,0.8515,
END,2454534,,

And I've got to insert a row in a Header table.. containing the header info.. the data rows have to be individual record inserts into a Data table... and they need to be foreign keyed with the ID of the record in the Header table...

I am looking at the example in the SQLIS website http://www.sqlis.com/default.aspx?54 which shows how to handle different row types... problem there is that my file does not have a row-type field. Its just: first row is header.. the rest are detail.. and the last is checksum....

So I'm just trying to figure out a strategy to do this... what approach should I take ?

BTW: I'm a newbie at SSIS... am reading the Wrox book... but its slow going...

Thanks

|||

Multi-format rows can be handled in the same data-flow.When you import them you'll have to interpret the file as simply a file with one long text column in it and then, after you've split the data (see below) parse the proper columns out.


Regarding splitting the data into header and detail records...

If there is a characteristic of each row that determines what type it is then the Conditional Split transform will split it out into the different sections. From the snippet you have posted above it looks like that shouldn't be too difficult (i.e. Are the first 8 characters all digits? If so, its a header record.)

-Jamie

|||

Thanks for your advice Jamie..

I suppose I could use the fact that the first eight chars all digits... I was hoping to just be able to say.. this is the First Row in the file.. so do this...

I dunno.. think I'm having some trouble getting my head around the SSIS way of thinking... being a bit too procedural in my approach...

Like.. I have in my head a procedural way that i think this should be done.. but can't quite map it to an SSIS package...

Maybe you could look at that with me?

For instance I feel that the steps I'd need to carry out to load this file are:

1. Read first line and create a FileLoaded rec in the FileLoaded table. Get an ID for this FileLoaded rec. If there is an error inserting the FileLoaded rec then report error (via email and insert of Error rec to LoadErrors table) and exit.

2. for next bunch of lines: insert FileLine rec in File Line table. Use FileLoaded ID as foreign key. If there's any error then report error.. rollback all inserts.. also rollback insert of FileLoaded rec... report error (detailing line of file that problem was encountered at)

3. When END line is reached, commit transaction. Report success (via email and via insert of record in some Logging table).

4. Copy file to archive.

I'm kinda stumped as to how to do this, especially how to FIRST insert a parent (FileLoaded) record BEFORE starting to insert child (FileLine) records. The sample on SQLIS just splits into master-rows and detail-rows.. seems to happen simultaneously.. so would we not have a race condition where child recs with the foreign key could get inserted before the parent recs ... and so cause a ref integrity error?

Also can't see how to start and roll back transactions and exit straight out of a package when some error occurs...

Please bear in mind that I am still learning this SSIS stuff, I have the Wrox book and am working through it.. but, as is always the case, am under pressure to get something done... something which the book just doesn't seem to cover... ie parent-child related data input...

I do really appreciate any time you (or anyone) spends reading my posts and helping...

PJ

|||

PJFINTRAX wrote:

I suppose I could use the fact that the first eight chars all digits... I was hoping to just be able to say.. this is the First Row in the file.. so do this...

You can do this in a script component if you really want to, I'm not sure its necassary though. This should help: http://blogs.conchango.com/jamiethomson/archive/2005/07/27/1877.aspx

PJFINTRAX wrote:

I dunno.. think I'm having some trouble getting my head around the SSIS way of thinking... being a bit too procedural in my approach...

Like.. I have in my head a procedural way that i think this should be done.. but can't quite map it to an SSIS package...

Maybe you could look at that with me?

For instance I feel that the steps I'd need to carry out to load this file are:

1. Read first line and create a FileLoaded rec in the FileLoaded table. Get an ID for this FileLoaded rec.

Fairly easy to do as I think we've already covered. Once you insert the record you can get the ID that you just created using an Execute SQL Task.

PJFINTRAX wrote:

If there is an error inserting the FileLoaded rec then report error (via email and insert of Error rec to LoadErrors table)

SendMail Task will do the emailing.

OnError precedence constraint or the OnError eventhandler can do the error reporting. Personally I would use the OnError eventhandler because you get more information about the error.

PJFINTRAX wrote:

and exit.

OnSuccess/OnError precedence constraints handles this

PJFINTRAX wrote:

2. for next bunch of lines: insert FileLine rec in File Line table. Use FileLoaded ID as foreign key.

FileLoadedID is, I presume, the record you created in the previous step? As stated, you can get hold of this value using an Execute SQL Task.

PJFINTRAX wrote:

If there's any error then report error

Already covered I think...

PJFINTRAX wrote:

.. rollback all inserts.. also rollback insert of FileLoaded rec...

SSIS supports MSDTC transactions: http://blogs.conchango.com/jamiethomson/archive/2004/12/14/456.aspx

PJFINTRAX wrote:

report error (detailing line of file that problem was encountered at)

That will be a bit harder to given that SSIS doesn't natively support row numbering within the pipeline (it would be quite nice if it did). You can create your own numbering as explained here: http://www.sqlis.com/default.aspx?37

PJFINTRAX wrote:

3. When END line is reached, commit transaction.

Transactions again

PJFINTRAX wrote:

Report success (via email

SendMail Task

PJFINTRAX wrote:

and via insert of record in some Logging table).

Lots of ways of doing this. I would use the OnPostExecute eventhandler!

PJFINTRAX wrote:

4. Copy file to archive.

FileSystem task will do this for you.

PJFINTRAX wrote:

I'm kinda stumped as to how to do this, especially how to FIRST insert a parent (FileLoaded) record BEFORE starting to insert child (FileLine) records.

I'm currently working on a blog post about this very thing so look out for that in teh next few days (http://blogs.conchango.com/jamiethomson). Its fairly easy, basically you need 2 data-flows. Insert the FileLoaded record in the first data-flow. Pass the child records to the data-flow using a raw file....and then insert them. Easy-peasy!

PJFINTRAX wrote:

The sample on SQLIS just splits into master-rows and detail-rows.. seems to happen simultaneously.. so would we not have a race condition where child recs with the foreign key could get inserted before the parent recs ... and so cause a ref integrity error?

Correct. See my previous comment. Raw files are a big help here. I use raw files all over the place in my implementations - they're incredibly useful.

PJFINTRAX wrote:

Also can't see how to start and roll back transactions and exit straight out of a package when some error occurs...

Please bear in mind that I am still learning this SSIS stuff, I have the Wrox book and am working through it.. but, as is always the case, am under pressure to get something done... something which the book just doesn't seem to cover... ie parent-child related data input...

I do really appreciate any time you (or anyone) spends reading my posts and helping...

PJ

Hope this has helped!!!

-Jamie

|||

Jamie,

thank you very very much for taking the time to go through my post.

I'll go and read all of those references you have given me and hopefully will be back to report some progress!

Thanks again,

PJ

|||

Jamie,
I'm kinda stuck for time on this so have been trying to do a few bits of it... not sure how to use a raw file to pause one dataflow until the other has completed....if you could give a very short description to elighten me a bit further?

But anyway.. I think I have the bit about picking out the first record cracked....

I just pumped the file into a script task that added line numbers to the lines of the file... and then did a conditional split based on the line number... if the line num is 1 then I have one output (for parent rec) and if its not = 1 then I have another ouput (for child records)...

Wish I could paste a picture of the dataflow in here...sigh...

Anyway, the script in the script block transform that adds line numbers is really simple:

Imports System
Imports System.Data
Imports System.Math
Imports Microsoft.SqlServer.Dts.Pipeline.Wrapper
Imports Microsoft.SqlServer.Dts.Runtime.Wrapper

Public Class ScriptMain
Inherits UserComponent

Public Overrides Sub Input0_ProcessInputRow(ByVal Row As Input0Buffer)
Static LineCounter As Integer = 0
LineCounter += 1
Me.NumberedRowsBuffer.AddRow()
Me.NumberedRowsBuffer.LineNum = LineCounter
Me.NumberedRowsBuffer.LineData = Row.Column0
End Sub

End Class

So the input file gets pumped through this scirpt.. two fields come out, one is the whole line as input from the file, and the other is a line number...

This then goes into the conditional split...

when the conditional split sends out a parent rec I wanted to insert a rec in the parent table and get back a parent ID....

But I've come up stumped against using an OLE DB Command to insert the parent record in the parent table .. I can't figure out how to get return values out of a stored proc called via an OLE DB Command....

Is this something I'm supposed to be able to do?

Thnks in advance,

PJ

Hierarchical XML import?

Hello,

Can anyone point me at a tutorial or sample that shows how to use IS for importing an xml file containing hierarchically arranged records ?

I have a file which contains multiple orders , the orders contain multiple line items.. the file also contains an element with details of the file source etc...

So, I want to make an insert in the FileLog table an then make inserts into the orders table .. then make inserts into the OrderItems table which will have the foreign key from the orders table in the records...

if you get what I mean...

But I have searched hign and low and can't see any info on how to load anything but a very flat xml file structure...

Thanks

Vida.

The XML Source will take a hierarchical XML feed and crack it, giving multiple outputs, one for each level. Have a play.

To handle the FKs you need to enforce order of inserts which cannot be done in a single Data Flow task, so stage the data and use several tasks, one for each successive table with the FK. Raw Files would be the ideal staging area for this, fast and efficient.

|||

Darren,

Am I right in thinkin that SQLIS is just not designed to load anything but a flat file containing a simple set of same-format rows?

I just can't seem to find very much info on loading master-detail records , whether in XML or in any ordinary flat file ....

Now I've been given a task of loading a simple text file... first row is a bit of header info.. then the next ten rows are data rows in a simple csv format... last row is a checksum.

20060518,SUPPLIER1,,
EUR,AUD,1.6869,
GBP,AUD,2.4896,
USD,AUD,0.7568,
DKK,AUD,4.4581,
SEK,AUD,5.6246,
CHF,AUD,0.9246,
JPY,AUD,84.5870,
NZD,AUD,1.2293,
NOK,AUD,4.6809,
SGD,AUD,1.2060,
HKD,AUD,5.9217,
AUD,AUD,1.0000,
CAD,AUD,0.8515,
END,2454534,,

And I've got to insert a row in a Header table.. containing the header info.. the data rows have to be individual record inserts into a Data table... and they need to be foreign keyed with the ID of the record in the Header table...

I am looking at the example in the SQLIS website http://www.sqlis.com/default.aspx?54 which shows how to handle different row types... problem there is that my file does not have a row-type field. Its just: first row is header.. the rest are detail.. and the last is checksum....

So I'm just trying to figure out a strategy to do this... what approach should I take ?

BTW: I'm a newbie at SSIS... am reading the Wrox book... but its slow going...

Thanks

|||

Multi-format rows can be handled in the same data-flow.When you import them you'll have to interpret the file as simply a file with one long text column in it and then, after you've split the data (see below) parse the proper columns out.


Regarding splitting the data into header and detail records...

If there is a characteristic of each row that determines what type it is then the Conditional Split transform will split it out into the different sections. From the snippet you have posted above it looks like that shouldn't be too difficult (i.e. Are the first 8 characters all digits? If so, its a header record.)

-Jamie

|||

Thanks for your advice Jamie..

I suppose I could use the fact that the first eight chars all digits... I was hoping to just be able to say.. this is the First Row in the file.. so do this...

I dunno.. think I'm having some trouble getting my head around the SSIS way of thinking... being a bit too procedural in my approach...

Like.. I have in my head a procedural way that i think this should be done.. but can't quite map it to an SSIS package...

Maybe you could look at that with me?

For instance I feel that the steps I'd need to carry out to load this file are:

1. Read first line and create a FileLoaded rec in the FileLoaded table. Get an ID for this FileLoaded rec. If there is an error inserting the FileLoaded rec then report error (via email and insert of Error rec to LoadErrors table) and exit.

2. for next bunch of lines: insert FileLine rec in File Line table. Use FileLoaded ID as foreign key. If there's any error then report error.. rollback all inserts.. also rollback insert of FileLoaded rec... report error (detailing line of file that problem was encountered at)

3. When END line is reached, commit transaction. Report success (via email and via insert of record in some Logging table).

4. Copy file to archive.

I'm kinda stumped as to how to do this, especially how to FIRST insert a parent (FileLoaded) record BEFORE starting to insert child (FileLine) records. The sample on SQLIS just splits into master-rows and detail-rows.. seems to happen simultaneously.. so would we not have a race condition where child recs with the foreign key could get inserted before the parent recs ... and so cause a ref integrity error?

Also can't see how to start and roll back transactions and exit straight out of a package when some error occurs...

Please bear in mind that I am still learning this SSIS stuff, I have the Wrox book and am working through it.. but, as is always the case, am under pressure to get something done... something which the book just doesn't seem to cover... ie parent-child related data input...

I do really appreciate any time you (or anyone) spends reading my posts and helping...

PJ

|||

PJFINTRAX wrote:

I suppose I could use the fact that the first eight chars all digits... I was hoping to just be able to say.. this is the First Row in the file.. so do this...

You can do this in a script component if you really want to, I'm not sure its necassary though. This should help: http://blogs.conchango.com/jamiethomson/archive/2005/07/27/1877.aspx

PJFINTRAX wrote:

I dunno.. think I'm having some trouble getting my head around the SSIS way of thinking... being a bit too procedural in my approach...

Like.. I have in my head a procedural way that i think this should be done.. but can't quite map it to an SSIS package...

Maybe you could look at that with me?

For instance I feel that the steps I'd need to carry out to load this file are:

1. Read first line and create a FileLoaded rec in the FileLoaded table. Get an ID for this FileLoaded rec.

Fairly easy to do as I think we've already covered. Once you insert the record you can get the ID that you just created using an Execute SQL Task.

PJFINTRAX wrote:

If there is an error inserting the FileLoaded rec then report error (via email and insert of Error rec to LoadErrors table)

SendMail Task will do the emailing.

OnError precedence constraint or the OnError eventhandler can do the error reporting. Personally I would use the OnError eventhandler because you get more information about the error.

PJFINTRAX wrote:

and exit.

OnSuccess/OnError precedence constraints handles this

PJFINTRAX wrote:

2. for next bunch of lines: insert FileLine rec in File Line table. Use FileLoaded ID as foreign key.

FileLoadedID is, I presume, the record you created in the previous step? As stated, you can get hold of this value using an Execute SQL Task.

PJFINTRAX wrote:

If there's any error then report error

Already covered I think...

PJFINTRAX wrote:

.. rollback all inserts.. also rollback insert of FileLoaded rec...

SSIS supports MSDTC transactions: http://blogs.conchango.com/jamiethomson/archive/2004/12/14/456.aspx

PJFINTRAX wrote:

report error (detailing line of file that problem was encountered at)

That will be a bit harder to given that SSIS doesn't natively support row numbering within the pipeline (it would be quite nice if it did). You can create your own numbering as explained here: http://www.sqlis.com/default.aspx?37

PJFINTRAX wrote:

3. When END line is reached, commit transaction.

Transactions again

PJFINTRAX wrote:

Report success (via email

SendMail Task

PJFINTRAX wrote:

and via insert of record in some Logging table).

Lots of ways of doing this. I would use the OnPostExecute eventhandler!

PJFINTRAX wrote:

4. Copy file to archive.

FileSystem task will do this for you.

PJFINTRAX wrote:

I'm kinda stumped as to how to do this, especially how to FIRST insert a parent (FileLoaded) record BEFORE starting to insert child (FileLine) records.

I'm currently working on a blog post about this very thing so look out for that in teh next few days (http://blogs.conchango.com/jamiethomson). Its fairly easy, basically you need 2 data-flows. Insert the FileLoaded record in the first data-flow. Pass the child records to the data-flow using a raw file....and then insert them. Easy-peasy!

PJFINTRAX wrote:

The sample on SQLIS just splits into master-rows and detail-rows.. seems to happen simultaneously.. so would we not have a race condition where child recs with the foreign key could get inserted before the parent recs ... and so cause a ref integrity error?

Correct. See my previous comment. Raw files are a big help here. I use raw files all over the place in my implementations - they're incredibly useful.

PJFINTRAX wrote:

Also can't see how to start and roll back transactions and exit straight out of a package when some error occurs...

Please bear in mind that I am still learning this SSIS stuff, I have the Wrox book and am working through it.. but, as is always the case, am under pressure to get something done... something which the book just doesn't seem to cover... ie parent-child related data input...

I do really appreciate any time you (or anyone) spends reading my posts and helping...

PJ

Hope this has helped!!!

-Jamie

|||

Jamie,

thank you very very much for taking the time to go through my post.

I'll go and read all of those references you have given me and hopefully will be back to report some progress!

Thanks again,

PJ

|||

Jamie,
I'm kinda stuck for time on this so have been trying to do a few bits of it... not sure how to use a raw file to pause one dataflow until the other has completed....if you could give a very short description to elighten me a bit further?

But anyway.. I think I have the bit about picking out the first record cracked....

I just pumped the file into a script task that added line numbers to the lines of the file... and then did a conditional split based on the line number... if the line num is 1 then I have one output (for parent rec) and if its not = 1 then I have another ouput (for child records)...

Wish I could paste a picture of the dataflow in here...sigh...

Anyway, the script in the script block transform that adds line numbers is really simple:

Imports System
Imports System.Data
Imports System.Math
Imports Microsoft.SqlServer.Dts.Pipeline.Wrapper
Imports Microsoft.SqlServer.Dts.Runtime.Wrapper

Public Class ScriptMain
Inherits UserComponent

Public Overrides Sub Input0_ProcessInputRow(ByVal Row As Input0Buffer)
Static LineCounter As Integer = 0
LineCounter += 1
Me.NumberedRowsBuffer.AddRow()
Me.NumberedRowsBuffer.LineNum = LineCounter
Me.NumberedRowsBuffer.LineData = Row.Column0
End Sub

End Class

So the input file gets pumped through this scirpt.. two fields come out, one is the whole line as input from the file, and the other is a line number...

This then goes into the conditional split...

when the conditional split sends out a parent rec I wanted to insert a rec in the parent table and get back a parent ID....

But I've come up stumped against using an OLE DB Command to insert the parent record in the parent table .. I can't figure out how to get return values out of a stored proc called via an OLE DB Command....

Is this something I'm supposed to be able to do?

Thnks in advance,

PJ

Friday, February 24, 2012

Hiding Point Labels

Hi all,
I have been working with a pie chart and have had some issues with data
overlapping due to the small size of the pie slice. Robert Bruckner has
graciously helped me with the expression below to hide the data points that
are smaller than X% but now the lines are still showing.
=iif( Sum(Fields!YTD.Value)/Sum(Fields!YTD.Value,"chart3") < 0.005 , " ",
Sum(Fields!YTD.Value)/Sum(Fields!YTD.Value,"chart2"))
I would assume that I can hide the line by going into the Label Style and
adding an expression that changes the Color to the same as the backgroud
color if the pie slice is smaller than X%. However, I can't figure out the
syntax for the expression...see below what I have and please let me know how
I should change.
=iif( Sum(Fields!YTD.Value)/Sum(Fields!YTD.Value,"chart2") < 0.005 ,
"#FFFFFF", "#000000"))
This is the error I get:
The color expression for the chart â'chart2â' contains an error: [BC30205] End
of statement expected.
Build complete -- 1 errors, 0 warningsYou only need one closing parenthesis at the end of the expression. Try
this:
=iif( Sum(Fields!YTD.Value)/Sum(Fields!YTD.Value,"chart2") < 0.005 ,
"#FFFFFF", "#000000")
-- Robert
This posting is provided "AS IS" with no warranties, and confers no rights.
"anthonysjo" <anthonysjo@.discussions.microsoft.com> wrote in message
news:173EAE83-5542-4FC5-A168-89E09F0CD5C0@.microsoft.com...
> Hi all,
> I have been working with a pie chart and have had some issues with data
> overlapping due to the small size of the pie slice. Robert Bruckner has
> graciously helped me with the expression below to hide the data points
> that
> are smaller than X% but now the lines are still showing.
> =iif( Sum(Fields!YTD.Value)/Sum(Fields!YTD.Value,"chart3") < 0.005 , " ",
> Sum(Fields!YTD.Value)/Sum(Fields!YTD.Value,"chart2"))
> I would assume that I can hide the line by going into the Label Style and
> adding an expression that changes the Color to the same as the backgroud
> color if the pie slice is smaller than X%. However, I can't figure out
> the
> syntax for the expression...see below what I have and please let me know
> how
> I should change.
> =iif( Sum(Fields!YTD.Value)/Sum(Fields!YTD.Value,"chart2") < 0.005 ,
> "#FFFFFF", "#000000"))
> This is the error I get:
> The color expression for the chart 'chart2' contains an error: [BC30205]
> End
> of statement expected.
> Build complete -- 1 errors, 0 warnings
>|||Well that was easy!! The only problem is...it changed the text of the label.
I need to make the line and the label disappear. Can I do this? If so
where does the expression need to go?
Thanks again for all the help!!
"Robert Bruckner [MSFT]" wrote:
> You only need one closing parenthesis at the end of the expression. Try
> this:
> =iif( Sum(Fields!YTD.Value)/Sum(Fields!YTD.Value,"chart2") < 0.005 ,
> "#FFFFFF", "#000000")
>
> -- Robert
> This posting is provided "AS IS" with no warranties, and confers no rights.
>
> "anthonysjo" <anthonysjo@.discussions.microsoft.com> wrote in message
> news:173EAE83-5542-4FC5-A168-89E09F0CD5C0@.microsoft.com...
> > Hi all,
> >
> > I have been working with a pie chart and have had some issues with data
> > overlapping due to the small size of the pie slice. Robert Bruckner has
> > graciously helped me with the expression below to hide the data points
> > that
> > are smaller than X% but now the lines are still showing.
> >
> > =iif( Sum(Fields!YTD.Value)/Sum(Fields!YTD.Value,"chart3") < 0.005 , " ",
> > Sum(Fields!YTD.Value)/Sum(Fields!YTD.Value,"chart2"))
> >
> > I would assume that I can hide the line by going into the Label Style and
> > adding an expression that changes the Color to the same as the backgroud
> > color if the pie slice is smaller than X%. However, I can't figure out
> > the
> > syntax for the expression...see below what I have and please let me know
> > how
> > I should change.
> >
> > =iif( Sum(Fields!YTD.Value)/Sum(Fields!YTD.Value,"chart2") < 0.005 ,
> > "#FFFFFF", "#000000"))
> >
> > This is the error I get:
> >
> > The color expression for the chart 'chart2' contains an error: [BC30205]
> > End
> > of statement expected.
> > Build complete -- 1 errors, 0 warnings
> >
> >
>
>|||If this is RS 2000 SP1/SP2:
Go to the Appearance tab of the data value. Click on the "series style"
button and modify the border line width property: =iif(
Sum(Fields!YTD.Value)/Sum(Fields!YTD.Value,"chart2") < 0.005, "0.5 pt", "1
pt")
If this is RS 2005, you should modify the border style expression:
=iif( Sum(Fields!YTD.Value)/Sum(Fields!YTD.Value,"chart2") < 0.005, "None",
"Solid")
-- Robert
This posting is provided "AS IS" with no warranties, and confers no rights.
"anthonysjo" <anthonysjo@.discussions.microsoft.com> wrote in message
news:1193D40C-9712-41E4-96EF-D018C7C017D2@.microsoft.com...
> Well that was easy!! The only problem is...it changed the text of the
> label.
> I need to make the line and the label disappear. Can I do this? If so
> where does the expression need to go?
> Thanks again for all the help!!
> "Robert Bruckner [MSFT]" wrote:
>> You only need one closing parenthesis at the end of the expression. Try
>> this:
>> =iif( Sum(Fields!YTD.Value)/Sum(Fields!YTD.Value,"chart2") < 0.005 ,
>> "#FFFFFF", "#000000")
>>
>> -- Robert
>> This posting is provided "AS IS" with no warranties, and confers no
>> rights.
>>
>> "anthonysjo" <anthonysjo@.discussions.microsoft.com> wrote in message
>> news:173EAE83-5542-4FC5-A168-89E09F0CD5C0@.microsoft.com...
>> > Hi all,
>> >
>> > I have been working with a pie chart and have had some issues with data
>> > overlapping due to the small size of the pie slice. Robert Bruckner
>> > has
>> > graciously helped me with the expression below to hide the data points
>> > that
>> > are smaller than X% but now the lines are still showing.
>> >
>> > =iif( Sum(Fields!YTD.Value)/Sum(Fields!YTD.Value,"chart3") < 0.005 , "
>> > ",
>> > Sum(Fields!YTD.Value)/Sum(Fields!YTD.Value,"chart2"))
>> >
>> > I would assume that I can hide the line by going into the Label Style
>> > and
>> > adding an expression that changes the Color to the same as the
>> > backgroud
>> > color if the pie slice is smaller than X%. However, I can't figure out
>> > the
>> > syntax for the expression...see below what I have and please let me
>> > know
>> > how
>> > I should change.
>> >
>> > =iif( Sum(Fields!YTD.Value)/Sum(Fields!YTD.Value,"chart2") < 0.005 ,
>> > "#FFFFFF", "#000000"))
>> >
>> > This is the error I get:
>> >
>> > The color expression for the chart 'chart2' contains an error:
>> > [BC30205]
>> > End
>> > of statement expected.
>> > Build complete -- 1 errors, 0 warnings
>> >
>> >
>>|||That did the trick!!
Thanks alot!!
Anthony
"Robert Bruckner [MSFT]" wrote:
> If this is RS 2000 SP1/SP2:
> Go to the Appearance tab of the data value. Click on the "series style"
> button and modify the border line width property: =iif(
> Sum(Fields!YTD.Value)/Sum(Fields!YTD.Value,"chart2") < 0.005, "0.5 pt", "1
> pt")
> If this is RS 2005, you should modify the border style expression:
> =iif( Sum(Fields!YTD.Value)/Sum(Fields!YTD.Value,"chart2") < 0.005, "None",
> "Solid")
> -- Robert
> This posting is provided "AS IS" with no warranties, and confers no rights.
>
> "anthonysjo" <anthonysjo@.discussions.microsoft.com> wrote in message
> news:1193D40C-9712-41E4-96EF-D018C7C017D2@.microsoft.com...
> > Well that was easy!! The only problem is...it changed the text of the
> > label.
> > I need to make the line and the label disappear. Can I do this? If so
> > where does the expression need to go?
> >
> > Thanks again for all the help!!
> >
> > "Robert Bruckner [MSFT]" wrote:
> >
> >> You only need one closing parenthesis at the end of the expression. Try
> >> this:
> >> =iif( Sum(Fields!YTD.Value)/Sum(Fields!YTD.Value,"chart2") < 0.005 ,
> >> "#FFFFFF", "#000000")
> >>
> >>
> >> -- Robert
> >> This posting is provided "AS IS" with no warranties, and confers no
> >> rights.
> >>
> >>
> >> "anthonysjo" <anthonysjo@.discussions.microsoft.com> wrote in message
> >> news:173EAE83-5542-4FC5-A168-89E09F0CD5C0@.microsoft.com...
> >> > Hi all,
> >> >
> >> > I have been working with a pie chart and have had some issues with data
> >> > overlapping due to the small size of the pie slice. Robert Bruckner
> >> > has
> >> > graciously helped me with the expression below to hide the data points
> >> > that
> >> > are smaller than X% but now the lines are still showing.
> >> >
> >> > =iif( Sum(Fields!YTD.Value)/Sum(Fields!YTD.Value,"chart3") < 0.005 , "
> >> > ",
> >> > Sum(Fields!YTD.Value)/Sum(Fields!YTD.Value,"chart2"))
> >> >
> >> > I would assume that I can hide the line by going into the Label Style
> >> > and
> >> > adding an expression that changes the Color to the same as the
> >> > backgroud
> >> > color if the pie slice is smaller than X%. However, I can't figure out
> >> > the
> >> > syntax for the expression...see below what I have and please let me
> >> > know
> >> > how
> >> > I should change.
> >> >
> >> > =iif( Sum(Fields!YTD.Value)/Sum(Fields!YTD.Value,"chart2") < 0.005 ,
> >> > "#FFFFFF", "#000000"))
> >> >
> >> > This is the error I get:
> >> >
> >> > The color expression for the chart 'chart2' contains an error:
> >> > [BC30205]
> >> > End
> >> > of statement expected.
> >> > Build complete -- 1 errors, 0 warnings
> >> >
> >> >
> >>
> >>
> >>
>
>