The unstructured data we are dealing with now has no relation to the original idea of the unstructured data we should be dealing with. What concepts am I talking about ? Don’t we have everything today, structured and unstructured data, working together? It’s not only a matter of names, it’s a matter of concepts lost in time, not being implemented, and not even having a name anymore, since their name, data warehouse, was stolen by something way smaller. I saw great data professionals speaking in huge technical events and using the name Data Warehouse meaning only the structured part of the data. Everyone only hear and accept Data Warehouse as being only the structured part of the data. It’s a marriage between Data Warehouse and Data Lake, dealing with structured and unstructured data, as if the original Data Warehosue concept didn’t already include unstrucutred data.Īt this time, the original concept of Data Warehouse is forgotten. The new tools brought the possibility of handling together structured and unstructured data resuting in the creation of a new concept: Data Lakehouse. Microsoft is going a step forward with Synapse Analytics, which integrates the capacity of all these tools in a single one, making it easier to manage and stay up-to-date, or at least we hope so. ![]() It’s hard to find the professionals for each task. The Data Lake concept arrived easily: We don’t move the data, we provision the tools and servers we need for processing, always pointing to the Data Lake.īack to our days, we have different tools to deal with each part of this: SQL Server for the data warehouse, Event Hub and Stream Analytics for the Big Data, Hadoop, Spark and Machine Learning for the data lake, besides Power BI, a powerful visualization tool.Įach tool having it’s own techniques to be learned, this makes the life of HR specialists very difficult. It’s simple: The amount of data is so huge we can’t afford moving it everytime we need some processsing. We got, with Big Data, the concept of Data Lake. At this point we were dealing with both, structured and unstructured data. Data ingestion techniques and architectures had to be created. A huge amount of data arriving in many different forms into our servers in a speed that regular relational databases were not ready to support. Big Dataįast forward many years, the internet brought to us the Big Data concept. Add into this the difficult to convince the company about the project and the unstructured data end up forgotten. It’s difficult to get support inside the company for an expensive project without a clear final result.Īt that time, we had plenty of tools to deal with structured data, but nothing integrated to deal with the unstructured data. In the beginning of a Data Warehouse project no one have any idea of all the possibilities it will create. ![]() It’s a too long project and may take a lot of time before it shows any result. ![]() The Data Warehouse was always a project very difficult to implement. The structured data is what we usually know, but what happened to the unstructured data on this concept? The definition of this concept may surprise you nowadays: the Data Warehouse is a collection of structured and unstructured data. A concept as great as difficult to implement. Once upon a time, we got the Data Warehouse concept.
0 Comments
Leave a Reply. |