Esgyn Corporation is a newcomer in the world of high tech companies. Our mission is to help create and nurture Enterprise-class transactional and operational SQL in the Apache® Hadoop ecosystem. But there are many companies doing SQL-on-Hadoop, so it is fair to ask, what are our credentials? I will explore that topic in this post.
Esgyn is a little bit different from your usual startup, in that we already have created and open-sourced a product (the incubating Apache Trafodion database management system), and we are already 30 employees strong. We are different too in that we already have a cohesive team: some of us have been working together on relational database for as long as 27 years!
Our roots in database go back to the Tandem Database Group at Tandem Computers. That group, which included such illustrious database pioneers as Jim Gray, Don Slutz and Franco Putzolu, developed NonStop™ SQL, the first high-performance, linearly scalable and fault-tolerant SQL engine in the industry. Some of Esgyn’s most experienced members joined that team in the late 1980’s, working on the second and third releases of that notable engine.
NonStop SQL (and Tandem in general) was very successful in the online transaction processing (OLTP) market. By the mid-1990’s, its linear scalability and ability to parallelize queries led to it being applied in the then-emerging decision support market place. We found that its architecture of bottom-up optimization and iterator-style run-time flow were not optimal for large query, so a rewrite, NonStop SQL/MX was created. (The first NonStop SQL engine was re-labeled “SQL/MP”.) This second engine boasted an optimizer based on the Cascades framework, which was created by Goetz Graefe. The run-time engine used a data flow architecture of operators communicating via queues. In principle, this idea makes it easy to introduce parallelism at any point in a query plan because a queue can always be replaced by a parallel distribution operator (“exchange operator” in our parlance). Pedro Celis, later an important leader in the development of Microsoft SQL Server™, was the lead architect of the NonStop SQL/MX effort.
Microsoft became very interested in the NonStop SQL/MX work, and helped finance development of a clustered solution on Windows NT™ using a SQL/MX port. Unfortunately, this effort did not work out from a business perspective and was withdrawn from the market in 1999. Even so, NonStop SQL/MX continued its development on the NonStop Kernel platform.
By 2006, Tandem had been acquired by Compaq, and Compaq in turn acquired by Hewlett-Packard. HP saw an opportunity in the Enterprise Data Warehousing (EDW) space for NonStop SQL/MX technology. The HP Neoview product was built on this engine, scaling to up to 512 nodes. Scaling to this size required substantial investment in optimization, mixed workload and query management technologies. On an OLTP system, queries are usually small with the occasional large reporting query thrown in. On EDWs, the difference in number of rows between the smallest of queries and the largest expands to a factor of a billion or more. Keeping good performance for all classes of queries requires advanced tools and techniques.
In 2011, HP decided to exit the EDW market, withdrawing HP Neoview. However, HP’s IT arm was running the business on a trio of HP Neoview machines, and needed a replacement. Work had been going on to port the HP Neoview software stack to clustered Linux. This solution, code named “SeaQuest”, went into operation in 2012 and has since replaced all of HP-IT’s Neoview systems.
In 2011 also, Rob Schauble joined the Neoview/SeaQuest development organization, now squirreled away within HP-IT, as its new Vice President. Rob saw the potential in the technology, and vowed to find a way to commercialize it once again. Some experiments were tried, and after awhile it became clear that the most promising application is in the Hadoop space. SQL-on-Hadoop solutions heretofore focused mainly on analytics. But it was clear that over time there would be a desire to move transactional and operational applications to Hadoop as well. HBase seemed to be a promising foundation, but it lacked a query language and its transaction support was limited to just single rows. This seemed like a stellar opportunity for a seasoned database engine development team and its mature technology!
With funding from HP Labs, the team ported much of the SeaQuest engine on top of HBase, giving birth to Trafodion, an enterprise-class transactional and operational SQL-on-Hadoop solution. A private beta version was first announced in December, 2013, followed by an open source version in June, 2014. We continue to develop and improve Trafodion, and in May 2015, Trafodion was accepted into the Apache Incubator.
Only one step was left to make. In July 2015, Esgyn Corporation was formed, consisting of 30 of the members of the Neoview/SeaQuest team. With this change, there is now an organization dedicated to the furtherance of the Trafodion technology.
It’s been quite a ride for us: Tandem NonStop SQL, NonStop SQL/MX, HP Neoview, SeaQuest and now Trafodion. We are a dedicated group of database management system engineers with deep experience in many marketplaces. Database is a fascinating area to work, never boring, which is why we keep at it. We look forward to much interesting work to come.