Hive is great for large scale data warehousing applications. In one of my recent projects I was handed over the interesting and challenging task of making Hive behave like an OLTP system i.e., support update and delete. To be more specific, the Hive database needed to be kept in near real time synchronization with multiple OLTP systems. In this post, I will discuss the high level features of the solution ba ...

Big Data is the result of practically everything in the world being monitored and measured, creating data faster than the available technologies can store, process or manage it. Since it is a lot more intuitive to represent information as a “file” than a relational object, there has been a surge of unstructured data, making up as much as 80% of new data we must manage. Organizations are struggling to manage ...

Hive is an abstraction on Hadoop Map Reduce. It provides a SQL like interface for querying HDFS data, whch accounts for most of it’s popularity. In Hive, table structured data in HDFS is encapsulated with a table as in RDBMS. The DDL for table creation in Hive looks very similar to table creation DDL in RDBMS. In one of my recent projects, I had a need for storing and querying JSON formatted hierarchical da ...

The potential of “big data,” the massive explosion of sources of information from sensors, smart devices, and all other devices connected to the Internet, is probably under-appreciated in terms of its eventual business impact. However, to take maximum advantage of big data, IT is going to have to press the re-start button on its architecture for acquiring and understanding information. IT will need to const ...

New to the Open Source Conference this year is OSCON Data, for developers pioneering the evolving architectures and tools to manage data. See how Hadoop is used to optimize scalability and reliability at Yahoo. Find out how Facebook utilizes HBase to manage real-time messaging. Why Netflix moved from relational DBs to NoSQL cloud systems for personalized movie choosing. The in-depth sessions at OSCON Data, ...

Big data is getting bigger, and Hadoop is becoming the tool that will allow companies to make full use of their growing amounts of data, says Peter Fenton of Benchmark Capita. Fenton is a partner at the venture capital firm which owns the majority stake of HortonWorks, a startup that separated from Yahoo’s Hadoop dev team, and is offering services around the open-source big data analytics platform. In an in ...

Is the day of reckoning for big data upon us? To many observers, the growth in data is nothing short of incomprehensible. Data is streaming into, out of, and through enterprises from a dizzying array of sources-transactions, remote devices, partner sites, websites, and nonstop user-generated content. Not only are the data stores resulting from this information driving databases to scale into the terabyte an ...

When your relational database takes longer to process your data than to collect it, it's time to call in big data technology, said panelists at Interop. Not everyone is sure whether they have big data or not, or whether they need a NoSQL system to handle it. One way to find out, said one adopter of a NoSQL approach, is to ask yourself whether it is taking you longer to process your data than it did to colle ...

There is a lot of buzz around Big Data and the NOSQL movement these days and rightly so. The issues with data have essentially been two-fold: find cost effective ways to store ever increasing amounts of data and information, and find ways to mine this information to extract meaningful Business Intelligence. This problem has been compounded by the emergence of web 2.0 technologies whose legion of loyal fans ...

My last post was on location data query and indexing using MongoDB. Location data query and index support is an unique and powerful feature of MongoDB. Continuing along the same thread, I will dig into Map Reduce framework built right into MongoDB. Some NOSQL database systems provide built in map reduce framework. When the query engine is not enough for complex aggregate queries or other complex computation ...