|
|
Inserting 25 millions records SQL server takes 3hrs.
Last post 05-30-2008, 6:40 AM by Phil Factor. 6 replies.
-
05-14-2008, 5:27 AM |
-
Rajesh_r_Varma
-
-
-
Joined on 04-04-2007
-
-
-
-
|
Inserting 25 millions records SQL server takes 3hrs.
Hi Teachies,
I am using SQL Server Standard Edition .
in one of my table , there i am inserting around 25 millions records and that takes time around more than 3 hrs.
same thing is happening while fetching records from that table.
this database contains only single file group i.e primary
and that table contains .. Clustered as well as non clustered index.
it doesnot have any Triggers.
How do i increase this performance.
Paritioning of table cannot be use in SQL Server Standard Edition.
Or Dropping all non clustered index before insert operation will improve my performance.
Please suggest me.
Thanks
Rajesh Varma
|
|
-
05-16-2008, 3:27 PM |
-
DataGeek
-
-
-
Joined on 05-16-2008
-
-
-
-
|
Re: Inserting 25 millions records SQL server takes 3hrs.
Can you post the DDL and DML. The performance bottleneck could be IO, could be CPU, could be code. There are alot of unknowns. What are the server specs? How large is a row? Is the source of the load a flat file, excel, another database? What are you using to perform the load? DTS, SSIS, T-SQL, Other?
|
|
-
05-19-2008, 1:45 AM |
-
Rajesh_r_Varma
-
-
-
Joined on 04-04-2007
-
-
-
-
|
Re: Inserting 25 millions records SQL server takes 3hrs.
Thanks buddys ..for your instant reply.
SERVER CONFIGURATION:
Intel pentium (R) 4 CPU 2.88 GHZ, 2.79 GHZ ,2 GB RAM
Operating System: WINDOWS 2003 R2 STANDARD SERVICE PACK 2
Microsoft SQL Server 2005 - 9.00.1399.06 (Intel X86)
Oct 14 2005 00:33:37
Copyright (c) 1988-2005 Microsoft Corporation
Standard Edition on Windows NT 5.2 (Build 3790: Service Pack 2)
DAtABASE DETAILS:
MDF and LDF located on C:\ Drive
Available Space on C:\ DRIVE 2.94 GB
TABLES DETAILS
CREATE TABLE [dbo].[TIX_PAYMENT_SCHEDULE](
[PaymentScheduleId] [bigint] IDENTITY(1,1) NOT NULL,
[OwedAmountId] [int] NULL, --NonClusteredIndex
[ProposalId] [int] NOT NULL, --NonClusteredIndex
[BrandId] [int] NULL, -- NonClusteredIndex
[DueDate] [datetime] NULL, --NonClusteredIndex
[OverdueDate] [datetime] NULL,--NonClusteredIndex
[ExpectedAmount] [decimal] (18, 2) NULL,
[TransactionStatusId] [tinyint] NULL,--NonClusteredIndex
[IsLate] [char] (1) NULL,
[IsPaymentReceived] [char] (1) NULL,
[ScheduleBatchJournalId] [bigint] NULL,--NonClusteredIndex
[IsValidSchedule] [char] (1) NULL,
[RuleId] [int] NULL,
[ActionId] [int] NOT NULL,
[ReasonId] [tinyint] NULL,
[Comments] [nvarchar] (2000) NULL,
[NoofDays] [int] NULL,
[ActualAmountReceived] [decimal] (18, 2) NULL,
[CreatedBy] [uniqueidentifier] NULL,
[CreatedDateTime] [datetime] NOT NULL,
[LastUpdatedBy] [uniqueidentifier] NULL,
[LastUpdatedDateTime] [datetime] NOT NULL,
[CaseScheduleId] [bigint] NULL,--NonClusteredIndex
[ActionDate] [datetime] NULL,
[HasExactMatch] [char] (1) NULL,
[IsCatchupBalanced] [char] (1) NULL,
[HasModified] [char] (1) NULL,--NonClusteredIndex
[PendDate] [datetime] NULL,
[IsAutoAccept] [char] (1) NULL,
[CatchupBalanceIdentifier] [uniqueidentifier] NULL,--NonClusteredIndex
CONSTRAINT [PK_TIX_PAYMENT_SCHEDULE] PRIMARY KEY CLUSTERED
(
[PaymentScheduleId] ASC
) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
TABLE 2
CREATE TABLE [dbo].[TIX_PAYMENT_CASE_SCHEDULE](
[CaseScheduleId] [bigint] IDENTITY(1,1) NOT NULL,
[ProposalId] [int] NOT NULL,--NonClusteredIndex
[DueDate] [datetime] NOT NULL,
[OverDueDate] [datetime] NOT NULL,
[TotalExpectedAmount] [decimal] (18, 2) NOT NULL,
[TotalActualPaymentReceived] [decimal] (18, 2) NOT NULL,
[TransactionStatusId] [int] NOT NULL,--NonClusteredIndex
[ActionId] [int] NULL,
[CreatedBy] [uniqueidentifier] NULL,
[CreatedDateTime] [datetime] NULL,
[LastUpdatedBy] [uniqueidentifier] NULL,
[LastUpdatedDateTime] [datetime] NULL,
[IsValidSchedule] [char] (1) NULL,
[ScheduleBatchJournalId] [bigint] NULL,
[IsCatchupBalanced] [char] (1) NULL,
[HasModified] [char] (1) NULL,--NonClusteredIndex
[CatchupBalanceIdentifier] [uniqueidentifier] NULL,
CONSTRAINT [PK_TIX_PAYMENT_CASE_SCHEDULE] PRIMARY KEY CLUSTERED
(
[CaseScheduleId] ASC
) WITH (PAD_INDEX = ON, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
GO
STORED PROCEDURE:
CREATE PROC [dbo].[TIX_PRC_GENERATE_PAYMENTSCHEDULE_DATA] ( @XMLParams XML, @ToDate datetime, @HasModified char(1) ) AS BEGIN SET NOCOUNT ON --Exception Handling Variable Declaration DECLARE @ErrorMessage NVARCHAR(200), @ErrorNumber INT, @ErrorSeverity INT, @ErrorState INT, @ErrorProcedure NVARCHAR(50), @ErrorLine INT, @ErrorDesc NVARCHAR(100) DECLARE @XMLPayment INT BEGIN TRY IF @XMLParams IS NOT NULL BEGIN --BEGIN IF SET @ErrorDesc='Error Occured While Inserting into TIX_PAYMENT_SCHEDULE FROM XML' INSERT INTO TIX_PAYMENT_SCHEDULE ( OwedAmountId, ProposalId, BrandId, DueDate, OverdueDate , CreatedDateTime, LastUpdatedDateTime, ExpectedAmount, ActualAmountReceived, ScheduleBatchJournalId, RuleId, TransactionStatusId, ActionId, IsLate, IsPaymentReceived , IsValidSchedule, --Added by DC : 119 IsCatchupBalanced, CatchupBalanceIdentifier, HasModified --------------------------------------------------- ) SELECT Main.ELEMENT.value('(OwedAmountId)[1]','int') AS OwedAmountId, Main.ELEMENT.value('(ProposalId)[1]','int') AS ProposalId, Main.ELEMENT.value('(BrandId)[1]','int') AS BrandId, convert(datetime,Main.ELEMENT.value('(DueDate)[1]','varchar(100)')) AS DueDate, convert(datetime,Main.ELEMENT.value('(OverdueDate)[1]','varchar(100)')) AS OverdueDate, @ToDate AS CreatedDateTime, @ToDate AS LastUpdatedDateTime, convert(decimal(18,2),Main.ELEMENT.value('(ExpectedAmount)[1]','varchar(100)')) AS ExpectedAmount, convert(decimal(18,2),Main.ELEMENT.value('(ActualAmountReceived)[1]','varchar(100)')) AS ActualAmountReceived, Main.ELEMENT.value('(ScheduleBatchJournalId)[1]','bigint') AS ScheduleBatchJournalId, Main.ELEMENT.value('(RuleId)[1]','int') AS RuleId, Main.ELEMENT.value('(TransactionStatusId)[1]','int') AS TransactionStatusId, Main.ELEMENT.value('(ActionId)[1]','int') AS ActionId, Main.ELEMENT.value('(IsLate)[1]','char(1)') AS IsLate, Main.ELEMENT.value('(IsPaymentReceived)[1]','char(1)') AS IsPaymentReceived, Main.ELEMENT.value('(IsValidSchedule)[1]','char(1)') AS IsValidSchedule --Added by DC for 119 ,Main.ELEMENT.value('(IsCatchupBalanced)[1]','char(1)') AS IsCatchupBalanced ,Main.ELEMENT.value('(CatchupBalanceIdentifier)[1]','nvarchar(1000)') AS CatchupBalanceIdentifier ,@HasModified --------------------------------------------------------------------- FROM @XMLParams.nodes ('(/ROOT/DATA)') AS Main(ELEMENT) END--END IF END TRY--Main END TRY BEGIN CATCH --Main BEGIN CATCH SELECT @ErrorMessage = @ErrorDesc+Char(13)+Error_Message(), @ErrorSeverity = Error_Severity(), @ErrorState = Error_State(), @ErrorNumber = Error_Number(), @ErrorProcedure = Error_Procedure(), @ErrorLine = Error_Line() RAISERROR( @ErrorMessage, @ErrorSeverity, @ErrorState, @ErrorNumber, @ErrorProcedure, @ErrorLine ) END CATCH --Main END CATCH END --Main END
STOREDPROCEDURE 2
CREATE PROCEDURE [dbo].[TIX_PRC_GET_PAYMENTSCHEDULE_SCHEDULE_FOR_DATE_RANGE] ( @ToDate datetime, @IsValidSchedule char(1) ) AS BEGIN SET NOCOUNT ON --Execption Handling Variable Declaration DECLARE @ErrorMessage NVARCHAR(200), @ErrorNumber INT, @ErrorSeverity INT, @ErrorState INT, @ErrorProcedure NVARCHAR(50), @ErrorLine INT, @ErrorDesc NVARCHAR(100) BEGIN TRY --Exception Handling SET @ErrorDesc='Error Occured while fetching records from TIX_PAYMENT_SCHEDULE' SELECT PaymentScheduleId, OwedAmountId, ProposalId, DueDate, OverdueDate, ExpectedAmount, TransactionStatusId, IsPaymentReceived, IsLate, ActionId, ActualAmountReceived, IsValidSchedule, BrandId, CaseScheduleId, ReasonId, Comments, NoOfDays, ActionDate, IsCatchupBalanced, CatchupBalanceIdentifier, HasModified from TIX_PAYMENT_SCHEDULE with (nolock) WHERE DUEDATE <=@ToDate AND IsValidSchedule=@IsValidSchedule SELECT DISTINCT OwedAmountId,proposalId,brandId from TIX_PAYMENT_SCHEDULE with (nolock) WHERE DUEDATE <=@ToDate AND IsValidSchedule=@IsValidSchedule Order By OwedAmountId,ProposalId,BrandId asc SELECT DISTINCT ProposalId from TIX_PAYMENT_SCHEDULE with (nolock) WHERE DUEDATE <=@ToDate AND IsValidSchedule=@IsValidSchedule Order By ProposalId asc END TRY BEGIN CATCH SELECT @ErrorMessage=@ErrorDesc+CHAR(13)+ Error_Message(), @ErrorNumber=Error_Number(), @ErrorState=Error_State(), @ErrorProcedure=Error_Procedure(), @ErrorLine=Error_Line(), @ErrorSeverity=Error_Severity() RAISERROR( @ErrorMessage, @ErrorSeverity, @ErrorState, @ErrorNumber, @ErrorProcedure, @ErrorLine ) END CATCH END
-- Dropping index before insert is it feasible. during productional hour.
Thanks & Regards
Rajesh Varma
|
|
-
05-23-2008, 10:03 AM |
-
Rajesh_r_Varma
-
-
-
Joined on 04-04-2007
-
-
-
-
|
Re: Inserting 25 millions records SQL server takes 3hrs.
I have changed my sql server setting of memory
on server properties tab --> maximum servery memory to 1536
and minimum memory per query -->5120
and also implemented suggestion of Database Tunning Advisior
I got good amount of performance enhancements. for 15Lakh's Records.
But my problem here is , i am Excepted to get around 3 crore records.
and i am using Standard edition . i cant use table partioning
So in that case . if i increase my RAM memory to 4GB ,
and allocate 3GB of RAM MEMORY to SQL SERVER .....is it going to slove my problem..
wating for valuable reply
Thanks
Rajesh Varma.
|
|
-
05-27-2008, 3:54 PM |
-
tosscrosby
-
-
-
Joined on 05-27-2008
-
-
-
-
|
Re: Inserting 25 millions records SQL server takes 3hrs.
Increasing the memory may help some. I'd drop the indexes, insert the data and then recreate the indexes. That most assuredly will help. Also, if recovery is set to full, can you change it to simple before the insert and then back to full after the insert is complete?
|
|
-
05-30-2008, 5:38 AM |
-
Rajesh_r_Varma
-
-
-
Joined on 04-04-2007
-
-
-
-
|
Re: Inserting 25 millions records SQL server takes 3hrs.
To overcome this performance issue, we have distributed our database in to different File groups and spread it over different drives.
- PRIMARY GROUP: located on C:\ consists of all the tables and clustered indexes.
- INDEX GROUP: located on D:\ Consists of all the non-clustered indexes.
- LOG GROUP: located on C:\ consists of log file.
Doing so, will boost multiprocessing. And CPU can read the data and indexes Simultaneously
Should I separate tables and clustered index on different drive ??
And what else I can implement to improve enhancement.
Thanks
Rajesh
|
|
-
|
|