As part of our mission to create a world-class stock market research platform, three of the most critical aspects of our service are platform stability, speed, and fresh data. To ensure we deliver them consistently as we grow we’ve just completed the first step in a major transformation of our software engineering capability - and there are some immediate benefits for users.
As part of this year-long project, we’ve introduced a completely new set of processes and technologies. We have also migrated all of our infrastructure from traditional hosted servers in the UK to the global cloud platform run by Amazon Web Services (AWS).
The final part of this move happened on Sunday 13th February, which means the service you’ve been using for the last few weeks has being delivered via the same cloud technologies that power companies like Netflix and Zoom. While it’s still early days for Stockopedia, the benefits to users are already becoming apparent...
The Benefits to You
Faster platform speed - navigate the site quicker than ever
Many of our systems were rewritten, heavily refactored, or replaced ahead of the AWS migration. Much of this was done to make it easier for us to consistently improve the platform over time. But in tandem with being accessible on AWS, that work has also made the site noticeably quicker and even more enjoyable to use.
Our tracking shows that substantial improvements to the code and the migration to AWS has led to a 27% improvement in browser response times on average. This means that you can navigate the site with less time spent waiting for pages to load.
Page loading speeds are just one part of a broader focus on making it quicker for users to get around. Our teams use a “21 Step User Journey” benchmark based on many of the actions we all tend to take in areas like Today, Search, Screens, StockReports, Charts, Folios and Discuss.
In June 2020, just as the Covid pandemic was escalating, that journey took an average time of 1 minute 20 seconds. But after a number of substantial navigation improvements - together with the faster page loading speeds - it has fallen to an average of 39.5 seconds. Against this benchmark, the platform has become much quicker - meaning that research times for users are faster and more enjoyable than ever.
Quicker global site speeds - ideal for research on the go
Stockopedia has subscribers in 61 countries. We also know that wherever you are in the world - whether it’s travelling for work or relaxing by a pool - global site speed is an important issue. With the migration to AWS, we’re no longer reliant on the speed of colocated servers in Berkshire, UK. With a global cloud platform, the service is faster across the world.
Faster processing times - fresh data is now available sooner
Stockopedia systems have to ingest and process financial data from all the main global regions in time for the market-open every day. This huge task takes several hours for each region, and until recently the data was being published to the site between 7.00am and 8.00am.
As part of the latest infrastructure improvements, hardly any of Stockopedia’s original code was left untouched. A vast re-engineering of the back-end means that data processing times are now much faster. On average, the time it takes to ingest, process and publish each data batch has halved to 3.5 hours from 7.0 hours previously.
This improvement means regional data batches are now being published “before you wake”, with fresh data typically reaching the platform between 5.00am and 6.00am (compared to between 7.00am and 8.00am previously) - well in time for your morning research to start.
New efficiencies mean resources can be invested elsewhere
Part of the appeal of migrating to the cloud is that the servers that deliver Stockopedia are now dynamic. Capacity expands and contracts depending on demand, which provides a much more reliable platform that you can depend on - and one that will scale with us as we grow.
But it’s no secret that cloud providers, such as AWS, are much more expensive than managing your own physical infrastructure. To simply lift-and-shift our current tech stack from a regular hosting provider to AWS would have resulted in higher costs but none of the benefits.
So it was imperative to streamline our infrastructure technologies and development practices with a view to making our systems more stable and our developers more productive. This required a substantial effort from the engineering team, and some of the technical highlights have included:
- Replacing all server-based application hosting to container orchestration using Kubernetes
- Re-engineering Stockopedia to remove two database technologies
- Replacing our core application mysql database with AWS Aurora
- Moving our build and deployment pipeline to a full GitOps model
- Rebuilding the data-import orchestration using Argo Workflows and allowing for better horizontal scaling
- Modifying the system architecture to better support dynamically scaling infrastructure based on user load
- Improved system observability using Prometheus and other instrumentation
- A significant number of smaller performance improvements and application adjustments
These changes were made to improve development productivity (measured by the rate of deploying new features and fixes) while keeping the service stable and capable of growing as we do.
Pleasingly, under regular conditions, the platform is already working with all the new speed improvements but using substantially less power to do it. In terms of CPU (processing power) we’ve seen reductions of around 48%, together with reductions of 69% in Memory.
The completion of this project represents a major milestone for the engineering efforts at Stockopedia, and are a boost for site speed, stability and data timeliness. But rather than it being the end-state, we think of it as a strong foundation on which we can build and deliver an exceptional product and user experience for years to come.
