Admitting we have a problem is the first step toward fixing it…
A decade old essay on Doing It Wrong to depress you, then one on Agile Data Testing (that might be even older!) to lift the spirits.
Virtualize the warehouse wherever possible to reduce the amount of “data ROT”.
Data ROT: Data that is redundant, trivial or obsolete.
On-premise data platforms can hide the true cost of data ROT, but usage-based pricing in the cloud will expose your ROT as you pay for extra compute and storage each month.
This means using views as access layer and “delta calculators” in your data warehouse. Here is an example of a virtual type 2 SCD, to illustrate how it can be done.
How does your architecture enable scaled Agile software development for your organization?
Your data architecture rests upon the business strategy communicated by leadership, and should be about building bridges and preserving flexibility for future change, not slowing delivery or guard rails.
Pipeline LLC 