SQL, the Un-sung Warrior: Why it Still Reigns in Big Data

Is SQL our old friend loosing the shine in the glamorous world of data science and big data? No not really. It is still the No. One choice.

SQL (structure query language) came onto the scene in the 1970s and since then it has undergone many changes to accommodate ever changing database structure.

However since the last decade of 20th century the data has been growing rapidly & exponentially. FB, Twitter, Gmails, Netflix have been contributing to ever growing database size and the companies have been finding ways to tap in the valuable data which motivated the growth of Dynamo, Map reduce, Cassandra, MongoDB. These companies were building their own sets of standard in retreiving the data like NoSQL.

The SQL at this point was loosing its shine and was only limited to retrieving small structured data (as compared to big data). The SQL could not be really used for unstructured data.

NoSQL began to be noticed and the developers around the world embraced it with open arms for obvious reason.

But in the fast paced impatient world, the end users and consumers were demanding more applications on top of the big data. Developers were trying to fix and join the broken pieces of code together to create the application layer with NoSQL technologies. Soon they realised, NoSQL was still uncooked for some applications and had limitation.

SQL on the other hands with years of in-flight experience was roped in and companies across the world was again embracing it. This led to the developement of SPARK.

SQL became the common standard for querying the data as the developers can easily write the same pieces of codes on several big data system.

In all the technology stack (Bigata, RDBMS, Visualisation tools, Networks), the one thing common is SQL.

The data have been growing, database technologies have been growing; but SQL is still the King.

You can also check out my online course here.