Showing posts with label store. Show all posts
Showing posts with label store. Show all posts

Monday, March 26, 2012

Historical tables, partitioning or what?

I have about 45000 records in a CSV file, which I am using as HTTP request parameters to query a website and store some results in a database. This is the kind of application which runs 24/7, so database grows really quickly. Every insert fires up a trigger, which has to look for some old records based on some criteria and modify the last inserted record. My client is crazy about performance on this one and suggested to move the old records into another table, which has exactly the same structure, but would serve as a historical table only (used to generate reports, statistics, etc.), whilst the original table would store only the latest rows (so no more than 45k at a given time, whereas the historical table may grow to millions of records). Is this a good idea? Having the performance in mind and the fact that there's that trigger - it has to run as quickly as possible - I might second that idea. Is it good or bad? What do you think?

I read a similar post here, which mentioned SQL Server 2005 partitioning, I might as well try this, although I never used it before.

I think you should use the archive table and partition it with 2005s new capabilities. This will allow you to easily administer the table(s). Heres a good link on 2005 partitioning. http://msdn2.microsoft.com/en-us/library/ms345146.aspx If you use a seperate archive table, you can create good indexes that will really speed queries. The only bad thing is that if you insert new records into it often, it could be slow. So, you might want to only insert new archive records during off-peak time if possible.
Tim|||I would caution to have the absolute minimal indexes on the archive table. In some situations, it may be efficient to create an index just for a report, and then remove the index as soon as the report is complete.|||Thanks for your comments. I read about SQL Server 2005's partitioning feature and find it very interesting (and useful, I think), but the application in question runs on the Express Edition currently with possible switch to a higher version, but doubtfuly the Enterprise one, so partitioning is out of the question, unfortunately. Sad I will hopefuly use it when I will work on some enterprise level application in the future.

Wednesday, March 7, 2012

Hierarchies: Any drawbacks doing it this way?

Hi all,
I have a requirement to store hierarchical data in a SQL database. I've
seen several different solutions posted here in the past on how to retrieve
hierarchical data.
In an effort to simplify the storage and retrieval of this data, I was
thinking of initally inserting the data into the table with some type of
"Level" or "Position" column which indicates the position in the hierarchy
of each item. I will need to insert and remove items in the hierarchy
somewhat frequently.
For an insertion into the hierarchy, if I know where the new item's position
should be, I could just insert the new item into the table with that
position number, then increment the item which previously had that
particular position and all the items below it by one. Removing an item
from the hierarchy would follow similar logic.
Are there any drawbacks to storing the hierarchical data this way? Is there
any reason for me to avoid this method of maintaining a "Position" column
for each item?
Any commenets are appreciated,
BenA "hierarchy level" column seems unlikely to be useful for subtree
maintenance in an Adjacency List model (I am assuming adjacency list
because you didn't specify what other model you might be using). In
fact it almost certainly hinders inserts because it is redundant data
that needs to be updated each time - probably on many rows for each row
inserted. Don't store the hierarchy level unless you regularly need to
query that information and have a good reason to denormalize.
For subtree maintenance, Materialized Path is typically the lowest cost
model. Nested Sets is cheap for pruning the tree but unpredictably
costly for grafting on subtrees.
David Portas
SQL Server MVP
--|||Check out this article from Joe Celko:
http://www.dbazine.com/ofinterest/oi-articles/celko24
--
HTH,
SriSamp
Email: srisamp@.gmail.com
Blog: http://blogs.sqlxml.org/srinivassampath
URL: http://www32.brinkster.com/srisamp
"Ben Amada" <ben@.REpoMOweVErpick.com> wrote in message
news:OL%23BJww0FHA.1032@.TK2MSFTNGP12.phx.gbl...
> Hi all,
> I have a requirement to store hierarchical data in a SQL database. I've
> seen several different solutions posted here in the past on how to
> retrieve hierarchical data.
> In an effort to simplify the storage and retrieval of this data, I was
> thinking of initally inserting the data into the table with some type of
> "Level" or "Position" column which indicates the position in the hierarchy
> of each item. I will need to insert and remove items in the hierarchy
> somewhat frequently.
> For an insertion into the hierarchy, if I know where the new item's
> position should be, I could just insert the new item into the table with
> that position number, then increment the item which previously had that
> particular position and all the items below it by one. Removing an item
> from the hierarchy would follow similar logic.
> Are there any drawbacks to storing the hierarchical data this way? Is
> there any reason for me to avoid this method of maintaining a "Position"
> column for each item?
> Any commenets are appreciated,
> Ben
>|||Hi David,
This is my first time dealing with hierarchies, so I'm not sure if the model
I'm working with is an Adjacency List model or not.
Below, I've included some DDL and insert statements on the type of data I'm
going to be storing -- the data below is for a single hierarchy tree I'll be
storing -- I'll actually be storing dozens of these trees, so I'll obviously
need another column in the table to identify the Hierarchy ID. There aren't
that many rows per hierarchy though, which is why I was thinking of storing
a "Position" or "Level" column.
When I retrieve the data from the table, the desired order of the data is
the same order of the INSERT statements below -- so "catOuter", "catOne",
"spnOne", etc.
I see myself having two or three options when retrieving the data:
(1) Create and maintain a "Position" column.
(2) Create a stored procedure or UDF which calls itself recursively to
retrieve the hierarchy in the desired order.
I suppose a 3rd option would be some set-based solution which is what I
usually try to do, but for hierarchies, I'm not sure how to implement a
set-based solution.
After seeing the sample data below, I'm interested in which option (1, 2 or
3) you think might work best?
Thanks again,
Ben
--
create table PageElements
( ElementID varchar(25) primary key,
ParentID varchar(25) )
insert into PageElements
(ElementID, ParentID)
select 'catOuter', NULL union all
select 'catOne', 'catOuter' union all
select 'spnOne', 'catOne' union all
select 'imgOne', 'catOne' union all
select 'catTwo', 'catOuter' union all
select 'spnTwo', 'catTwo' union all
select 'catThree', 'catOuter' union all
select 'spnThree', 'catThree' union all
select 'spnFour', 'catThree' union all
select 'spnFive', 'catThree' union all
select 'imgTwo', 'catThree' union all
select 'imgThree', 'catThree' union all
select 'spnSix', 'catThree' union all
select 'spnSeven', 'spnSix' union all
select 'spnEight', 'spnSix' union all
select 'spnNine', 'spnSix' union all
select 'spnTen', 'spnSix' union all
select 'spnEleven', 'spnSix' union all
select 'imgFour', 'spnSix'|||SriSamp wrote:

> Check out this article from Joe Celko:
> http://www.dbazine.com/ofinterest/oi-articles/celko24
Hi SriSamp,
Thank you for the helpful link! I'll take a look at it.
Ben|||Also buy the book TREES & HIERARCHIES IN SQL; it has more details and
other methods.|||If you want to store the items in order, you might want to look at a linked
list solution.
This way you only need to edit the item after the record you're inserting.
id parent
1 <null>
2 1
3 2
4 3
insert 5 between 2 & 3.
begin trans (serializable)
insert table values ( 5 , 2 )
update table set parent = 5 where parent = 2 and id != 5
commit trans
Then you end up with
id parent
1 <null>
2 1
5 2
3 5
4 3
HTH
"Ben Amada" <ben@.REpoMOweVErpick.com> wrote in message
news:OL%23BJww0FHA.1032@.TK2MSFTNGP12.phx.gbl...
> Hi all,
> I have a requirement to store hierarchical data in a SQL database. I've
> seen several different solutions posted here in the past on how to
retrieve
> hierarchical data.
> In an effort to simplify the storage and retrieval of this data, I was
> thinking of initally inserting the data into the table with some type of
> "Level" or "Position" column which indicates the position in the hierarchy
> of each item. I will need to insert and remove items in the hierarchy
> somewhat frequently.
> For an insertion into the hierarchy, if I know where the new item's
position
> should be, I could just insert the new item into the table with that
> position number, then increment the item which previously had that
> particular position and all the items below it by one. Removing an item
> from the hierarchy would follow similar logic.
> Are there any drawbacks to storing the hierarchical data this way? Is
there
> any reason for me to avoid this method of maintaining a "Position" column
> for each item?
> Any commenets are appreciated,
> Ben
>

hierarchical xml by DataSet()->WriteXml()

I'm playing with web-based front-end for hierarchical db in mssql2K.
So far my tables have 3 levels. I'm wonder how to store such hierarchical
trees outside (xml format).
DataSet()->WriteXml() method uses "2D" or flat dataset. How to create
tree-like xml?
gok
You should look at either the FOR XML clause or the SQLXML 3.0 annotated
schemas.
Best regards
Michael
"gok" <gok@.discussions.microsoft.com> wrote in message
news:43FC6F1B-AE18-4C77-80EB-549712A95E2B@.microsoft.com...
> I'm playing with web-based front-end for hierarchical db in mssql2K.
> So far my tables have 3 levels. I'm wonder how to store such hierarchical
> trees outside (xml format).
> DataSet()->WriteXml() method uses "2D" or flat dataset. How to create
> tree-like xml?
> --
> gok

hierarchical xml by DataSet()->WriteXml()

I'm playing with web-based front-end for hierarchical db in mssql2K.
So far my tables have 3 levels. I'm wonder how to store such hierarchical
trees outside (xml format).
DataSet()->WriteXml() method uses "2D" or flat dataset. How to create
tree-like xml?
--
gokYou should look at either the FOR XML clause or the SQLXML 3.0 annotated
schemas.
Best regards
Michael
"gok" <gok@.discussions.microsoft.com> wrote in message
news:43FC6F1B-AE18-4C77-80EB-549712A95E2B@.microsoft.com...
> I'm playing with web-based front-end for hierarchical db in mssql2K.
> So far my tables have 3 levels. I'm wonder how to store such hierarchical
> trees outside (xml format).
> DataSet()->WriteXml() method uses "2D" or flat dataset. How to create
> tree-like xml?
> --
> gok