White Labeled Ticketing Provider TicketSocket builds a Data Warehouse to simplify its reporting needs
TicketSocket is a true white-label Software-as-a-Service Platform that allows tenants to build their own ticketing platforms to deliver events.
Reporting is one of the most important aspects in any SaaS Platform.
It took quite an effort to write Mongo Aggegation queries to perform reporting in real-time. After all, the MongoDB was being used to full capacity for logging transactions.
TicketSocket’s existing MongoDB transactional database resources were being contended for live Orders traffic as well as the reporting metrics. HyperTrends decided to separate the Read and Write Databases using the classic CQRS tenet.
A more robust reporting platform requires moving the Order Data into a SQL Server Data Warehouse on Microsoft Azure.
Chief Technology Officer, Anup Marwadi leveraged his Software Development company HyperTrends Global Inc. to rapidly develop a Data Warehouse and bring the system into production.
When developing the Data Warehouse, we got estimates of approx 3-6 months from numerous experts. We didn’t have that kind of time at hand. I turned to my company HyperTrends to offer a DW solution. We got a Data Warehouse up in 3-weeks!TicketSocket CTO, Anup Marwadi
Data Warehousing Star Schema to the rescue
HyperTrends utilized the Star Schema approach for building the Data Warehouse.
Since majority of the reporting revolved around Orders, Tickets and Transactions, those were modeled as Facts and majority of the other elements were build as dimensions.
Given HyperTrends’ prior experience building high quality Data Warehouse solutions, many tenets like Dimensions for Dates, Months, Years etc. were readily available for deployment.
Microsoft Azure allows Rapid Deployment
Using Microsoft Azure, HyperTrends was very easily able to deploy the Data Warehouse to SQL Azure. Various security features like Column Masking and Key-Vault Encryption were utilized to maintain Data Warehouse Security.
HyperTrends was also able to turn on Threat Detection, lockdown access to the database by Firewall setup all using the Azure Management Console.
Azure Functions for ETL
HyperTrends used Azure Functions for ETL tasks. Upon successful Order, an entry was simply made into an Azure Functions Queue. The queue processor would then invoke the TicketSocket API to deliver Order data in near real-time for Data Warehouse ETL.
With Azure Functions, you only pay for what you use. This comes with added cost reductions.
Using the Azure Functions, we were able to simplify a sophisticated ETL process into a near real-time effort.We were quoted 5x the costs to build an ETL that would run nightly to load Order data. Using simple Modern programming techniques and using MicroServices Architecture, we were able to build a sub-second ETL processing that could be scaled infinitely.TicketSocket CTO, Anup Marwadi
Another Great Product Delivered
Using Data Warehousing, MicroServices Architecture, REST API and Azure Functions, HyperTrends was able to deliver a fully functional Data Warehouse in about 3-4 weeks.
If you are looking to build Data Warehouses, you may want to look into Azure Functions (or AWS Lambda) to simplify your ETL needs. It seems to work out really well.