Ísis Santos Costa’s Reviews > The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modeling > Status Update
Ísis Santos Costa
is on page 551 of 608
p. 518
20 • ETL System Design + Dev Process & Tasks (cont.)
↳ ETL Process Overview (cont.)
... ↳ Develop Incremental ETL Processing
... ... ↳ Step 8: Fact Table Incremental Processing
... ... ... ↳ Late Arriving Facts & the Surrogate Key Pipeline
If SCD Type 2 attributes are involved, a lookup must be performed using valid from / to information. Even fresh, this not necessarily is the currently valid information.
— Mar 09, 2026 07:53AM
20 • ETL System Design + Dev Process & Tasks (cont.)
↳ ETL Process Overview (cont.)
... ↳ Develop Incremental ETL Processing
... ... ↳ Step 8: Fact Table Incremental Processing
... ... ... ↳ Late Arriving Facts & the Surrogate Key Pipeline
If SCD Type 2 attributes are involved, a lookup must be performed using valid from / to information. Even fresh, this not necessarily is the currently valid information.
Like flag
Ísis’s Previous Updates
Ísis Santos Costa
is on page 560 of 608
p. 526
20 • ETL System Design + Dev Process & Tasks (cont.)
↳ Real-Time Implications
... ↳ Real-Time Triage
... ↳ Real-Time Architecture Trade-Offs
➤ Replace Batch Files
➤ Limit Data Quality Screenings
➤ Post Facts with Available Dimensions
➤ Eliminate Data Staging
... ↳ Real-Time Partitions in the Presentation Server
... ... ↳ Transaction Real-Time Partition
... ... ↳ Periodic Snapshot Real-Time Partition
— Apr 30, 2026 07:56PM
20 • ETL System Design + Dev Process & Tasks (cont.)
↳ Real-Time Implications
... ↳ Real-Time Triage
... ↳ Real-Time Architecture Trade-Offs
➤ Replace Batch Files
➤ Limit Data Quality Screenings
➤ Post Facts with Available Dimensions
➤ Eliminate Data Staging
... ↳ Real-Time Partitions in the Presentation Server
... ... ↳ Transaction Real-Time Partition
... ... ↳ Periodic Snapshot Real-Time Partition
Ísis Santos Costa
is on page 554 of 608
p. 520
20 • ETL System Design + Dev Process & Tasks (cont.)
↳ ETL Process Overview (cont.)
... ↳ Develop Incremental ETL Processing
... ... ↳ Step 9: Aggregate Table and OLAP Loads
... ... ↳ Step 10: ETL System Operation and Automation
➤ Schedule Jobs
➤ Automatically Handle Predictable Exceptions and Errors
➤ Gracefully Handle Unpredictable Errors
— Apr 09, 2026 04:02PM
20 • ETL System Design + Dev Process & Tasks (cont.)
↳ ETL Process Overview (cont.)
... ↳ Develop Incremental ETL Processing
... ... ↳ Step 9: Aggregate Table and OLAP Loads
... ... ↳ Step 10: ETL System Operation and Automation
➤ Schedule Jobs
➤ Automatically Handle Predictable Exceptions and Errors
➤ Gracefully Handle Unpredictable Errors
Ísis Santos Costa
is on page 552 of 608
p. 518
20 • ETL System Design + Dev Process & Tasks (cont.)
↳ ETL Process Overview (cont.)
... ↳ Develop Incremental ETL Processing
... ... ↳ Step 8: Fact Table Incremental Processing
... ... ... ↳ Incremental Fact Table Load
... ... ... ↳ Load Snapshot Fact Tables
... ... ... ↳ Speed Up the Load Cycle
✓ More Frequent Loading
✓ Parallel Processing
✓ Parallel Structure
— Apr 09, 2026 03:56PM
20 • ETL System Design + Dev Process & Tasks (cont.)
↳ ETL Process Overview (cont.)
... ↳ Develop Incremental ETL Processing
... ... ↳ Step 8: Fact Table Incremental Processing
... ... ... ↳ Incremental Fact Table Load
... ... ... ↳ Load Snapshot Fact Tables
... ... ... ↳ Speed Up the Load Cycle
✓ More Frequent Loading
✓ Parallel Processing
✓ Parallel Structure
Ísis Santos Costa
is on page 550 of 608
p. 516
20 • ETL System Design + Dev Process & Tasks (cont.)
↳ ETL Process Overview (cont.)
... ↳ Develop Incremental ETL Processing
... ... ↳ Step 8: Fact Table Incremental Processing
... ... ... ↳ FT Transformations & Surrogate Key Pipeline
Handling referential integrity violations:
✗ Halt the load
✓ Throw away (when not relevant)
✗ Write errors file
✓ Create dummy dim + surrogate key
✗ Map to single unknown dim
— Feb 28, 2026 06:05AM
20 • ETL System Design + Dev Process & Tasks (cont.)
↳ ETL Process Overview (cont.)
... ↳ Develop Incremental ETL Processing
... ... ↳ Step 8: Fact Table Incremental Processing
... ... ... ↳ FT Transformations & Surrogate Key Pipeline
Handling referential integrity violations:
✗ Halt the load
✓ Throw away (when not relevant)
✗ Write errors file
✓ Create dummy dim + surrogate key
✗ Map to single unknown dim
Ísis Santos Costa
is on page 550 of 608
p. 516
20 • ETL System Design + Dev Process & Tasks (cont.)
↳ ETL Process Overview (cont.)
... ↳ Develop Incremental ETL Processing (cont.)
... ... ↳ Step 8: Fact Table Incremental Processing (cont.)
... ... ... ↳ Fact Table Extract & Data Quality Checkpoint
« Data is written to a staging area.
Data Quality metrics of the raw data are computed. »
— Feb 27, 2026 08:14AM
20 • ETL System Design + Dev Process & Tasks (cont.)
↳ ETL Process Overview (cont.)
... ↳ Develop Incremental ETL Processing (cont.)
... ... ↳ Step 8: Fact Table Incremental Processing (cont.)
... ... ... ↳ Fact Table Extract & Data Quality Checkpoint
« Data is written to a staging area.
Data Quality metrics of the raw data are computed. »
Ísis Santos Costa
is on page 549 of 608
p. 515
20 • ETL System Design + Dev Process & Tasks (cont.)
↳ ETL Process Overview (cont.)
... ↳ Develop Incremental ETL Processing (cont.)
... ... ↳ Step 8: Fact Table Incremental Processing
You can stop a historic load process;
« the incremental processing, by contrast, must be fully automated. »
— Feb 22, 2026 09:49AM
20 • ETL System Design + Dev Process & Tasks (cont.)
↳ ETL Process Overview (cont.)
... ↳ Develop Incremental ETL Processing (cont.)
... ... ↳ Step 8: Fact Table Incremental Processing
You can stop a historic load process;
« the incremental processing, by contrast, must be fully automated. »
Ísis Santos Costa
is on page 548 of 608
p. 514
20 • ETL System Design + Dev Process & Tasks (cont.)
↳ ETL Process Overview (cont.)
... ↳ Develop Incremental ETL Processing
... ... ↳ Step 7: Dimension Table Incremental Processing
... ... ... ↳ Dimension Table Extracts
... ... ... ↳ Identify New and Changed Dimension Rows
... ... ... ↳ Process Changes to Dimension Attributes
— Feb 21, 2026 05:31AM
20 • ETL System Design + Dev Process & Tasks (cont.)
↳ ETL Process Overview (cont.)
... ↳ Develop Incremental ETL Processing
... ... ↳ Step 7: Dimension Table Incremental Processing
... ... ... ↳ Dimension Table Extracts
... ... ... ↳ Identify New and Changed Dimension Rows
... ... ... ↳ Process Changes to Dimension Attributes
Ísis Santos Costa
is on page 546 of 608
p. 512
20 • ETL System Design + Dev Process & Tasks (cont.)
↳ ETL Process Overview (cont.)
... ↳ Develop One-Time Historic Load Processing
... ... ↳ Step 6: Perform the Fact Table Historic Load (cont.)
... ... ... ↳ Fact Table Loading
« The main concern when loading the fact table is load performance. Some database technilogies support fast loading with a specified batch size. »
— Feb 20, 2026 10:09AM
20 • ETL System Design + Dev Process & Tasks (cont.)
↳ ETL Process Overview (cont.)
... ↳ Develop One-Time Historic Load Processing
... ... ↳ Step 6: Perform the Fact Table Historic Load (cont.)
... ... ... ↳ Fact Table Loading
« The main concern when loading the fact table is load performance. Some database technilogies support fast loading with a specified batch size. »
Ísis Santos Costa
is on page 545 of 608
p. 511
20 • ETL System Design + Dev Process & Tasks (cont.)
↳ ETL Process Overview (cont.)
... ↳ Develop One-Time Historic Load Processing
... ... ↳ Step 6: Perform the Fact Table Historic Load (cont.)
... ... ... ↳ Fact Table Transformations
➤ Null Fact Values
➤ Improve Fact Table Content
➤ Pipeline the Dimension Surrogate Key Lookup
➤ Assign Audit Dimension Key
— Feb 19, 2026 01:53PM
20 • ETL System Design + Dev Process & Tasks (cont.)
↳ ETL Process Overview (cont.)
... ↳ Develop One-Time Historic Load Processing
... ... ↳ Step 6: Perform the Fact Table Historic Load (cont.)
... ... ... ↳ Fact Table Transformations
➤ Null Fact Values
➤ Improve Fact Table Content
➤ Pipeline the Dimension Surrogate Key Lookup
➤ Assign Audit Dimension Key
Ísis Santos Costa
is on page 543 of 608
p. 509
20 • ETL System Design + Dev Process & Tasks (cont.)
↳ ETL Process Overview (cont.)
... ↳ Develop One-Time Historic Load Processing
... ... ↳ Step 6: Perform the Fact Table Historic Load (cont.)
... ... ... ↳ Fact Table Transformations
« A lot of time (is spent) improving the dimension table, facts usually require modest transformation. »
System values like -1 shall be replaced by NULL. All FKs be NOT NULL.
— Feb 17, 2026 05:34AM
20 • ETL System Design + Dev Process & Tasks (cont.)
↳ ETL Process Overview (cont.)
... ↳ Develop One-Time Historic Load Processing
... ... ↳ Step 6: Perform the Fact Table Historic Load (cont.)
... ... ... ↳ Fact Table Transformations
« A lot of time (is spent) improving the dimension table, facts usually require modest transformation. »
System values like -1 shall be replaced by NULL. All FKs be NOT NULL.
