Incremental Data Loaders are a great solution to the problem of long-running data loads - they allow you to import only the data which has changed since the previous import*. You can even, optionally, remove items from the warehouse which have been deleted in the source. In order to support Incremental Loads, your source table must have:
- A Primary Key.
- A "Last Updated Date" field.
If these items are in-place, then you are ready to proceed.
In the Data Loader Details screen, set the Source Option to "Incremental Loads" or "Incremental With Deletes".
In the Table Details screen, you need to consider the following.
- Ensure that a Primary Key is identified.
- Set the Source Date Modified Field. (If you need to calculate this item, then enter the calculation wrapped in brackets, as shown below.)
- Select the Process Deletes check-box if you want to remove items from the warehouse which no longer exist in your source.
Example Calculation - Note the wrapping brackets:
(COALESCE(UpdateDt,InsertDt))
And that's it! Simply Generate your Data Loader to enjoy incremental loads without having had to write a single line of code.
* Actually, it is a little bit more complicated than that, but only a little! Data Academy imports all the data which has changed since midnight on the date the previous load started. This means you will sometimes import a data item twice, but it also means that there is no scope for data "falling through the cracks" because of slightly differing system clocks.