On the right Paf: The big move to AWS

As an online gambling operator, Paf has often remained on the back foot in the way of technical development. Paf, which splits its tech and dev teams between the Åland capital city of Mariehamn, Finland’s Helsinki and Tallinn in Estonia, has faced the threat of lagging behind its industry peers as the majority move slowly towards increasing scale and automating systems.

Thus, in a move to update its operations and bring its technical abilities up to date, the firm is shifting its legacy tech stack to sit on Amazon’s extremely popular and widely-used AWS cloud system.

Historically, Play Among Friends, as Paf is very seldom known, was an early adopter of egaming technology when the company launched its first gambling site in 1999/2000.

Paf build and release engineer Lars-Göran Hakamo acknowledges the operator has fallen behind somewhat in recent years, particularly in moving into new markets (Sweden) without carrying out any housekeeping in terms of its tech stack.

“The past two or three years we realised that in order to meet the demands to be able to scale into further markets we had to change the set-up we have. [AWS] has the scale and flexibility we needed to start using cloud providers.

“Our CTO and lead architect [previously] worked heavily with AWS so it was a natural decision to go with that one,” says Hakamo. The DevOps model It’s a seismic move for any tech-first firm, but for Paf, which manages payments and data and customer journeys in real-time (as any other B2C operator does), the new system will have far-reaching gains across the entire operation, not least in ensuring teams are on the same wavelength and operating under a new DevOps model.

“When we move to the cloud, we also want to make the work more DevOps orientated, which means the teams will be more self-sufficient all the way from gathering requirements and formulating them within the business to having a better service within production,” Hakamo adds.

Paf’s Åland headquarters

“Having the development teams working with the business will ensure we are able to move features faster into production and on to our end customers. It also means if we notice the customers don’t react to a feature as the business intended, it is easier and should be faster for the development team to tweak the feature to get the maximum usage out of it – or even kill the feature off completely if it doesn’t fulfil the needs of the consumer.”

The overall tech team itself is made up of 120 developers, with 20 to 30 working on the migration directly. The majority of the development team is based in Helsinki with an additional 20 in the main office in Åland. Satellite offices are also found in Stockholm and Tallinn, with an additional marketing office in Madrid.

One major benefit of the move is that it will extinguish any threat of downtime across Paf’s site, as moving into the cloud allows for much more flexibility and scalability, and any concern over technical debt and progressively outdating chunky legacy tech is eradicated.

Hakamo anticipates all operations will be fully migrated onto AWS by spring 2020. He admits it is an ambitious objective, “but it’s the goal”, he notes.

“In the end I would say everything will be hosted on the cloud; there is no real need or business case for us to retain our in-house data centre. We have a secondary data centre which is operated by a different company,” he adds.

Road to migration It would be naïve for the firm to put all its trust in AWS or any cloud provider for that matter. Having existed for 20 years and watching the outside world of technology evolve so vastly, the tech team at Paf knows it must adopt a multi-cloud failover capability, which Hakamo says will likely be put in place upon completion of the migration roadmap.

The cloud migration process is known to be a disruptive one and understandably Paf cannot afford to sustain any downtime or major changes to its real-time tech processes. To offer an insight into the progress of the migration, Hakamo says it commenced a general availability of its internal service framework a month ago – it goes without saying that a public software release confirms the company is moving swiftly and taking the process seriously.

From a front-end perspective, the shift has caused minimal disruption, says senior front-end developer and UX designer Johan Westling. “But recently it’s been more impactful as we have moved everything including the source code to the cloud and there has been a more open-source mentality about everything in the company.” The introduction of DevOps has come with a new approach to cross-team collaboration and automating varying systems.

Westling says processes have been made more openly available across the different teams, which is no doubt invaluable for a tech team that spans three offices and two countries. Ewan Maley, lead architect, notes: “The communication between the teams has always been to do handovers and that always slows everything down. Using the DevOps approach, we’ve tried to implement a lot of autonomy within the teams, which has reduced the handovers and let the teams deliver a lot faster.”

Now widely adopted by many egaming firms, not least Paddy Power Betfair and Sky Betting & Gaming, DevOps-style management structures were popularised and quickly became ubiquitous among leading tech giants like Uber and Google, and it was those exact companies, speaking at a DevOps conference a couple of years ago, that planted the seed at Paf.

“Google and its SRE teams have been a big inspiration but we don’t claim we should be like them in any way. [However], the concepts they use are something we want to adopt,” Hakamo says.

Paf operates its own in-house games development arm

It’s in the game

The rumble of the migration has been as far reaching as the in-house games development team, which churns out up to eight games a year. It’s uncommon for a B2C operator to create its own exclusive games, particularly for a specific market.

But Paf does just that, and recently launched a relatively niche slot offering celebrating 100 years of Finnish skiing. Technical lead for the Paf games studio Pawel Stepanov considers the games studio a benefit in allowing product developers and games producers to work together on perfecting the UX and website design. “When we’re building the rest of the site we also have that knowledge in-house to help us with the rest of the design,” Stepanov remarks.

“If you work with local brands we can create features specifically for the brand which is also quite unique in the market.” AWS offers its partnering companies a range of in-built tools to improve their operations, and Maley insists much of the tech on offer will aid game development processes. He says if you’re looking from the front-end, the actual games themselves or the deployments, it really comes down to building deployment pipelines and having a very structured and continuous integration for them. “You can spin up new environments when you need them. It’s very easy to automate, but then when it comes to reliability you are going to struggle to beat any of the cloud providers for that as they have very good reliability and mitigation against any attacks you might get on the site,” he adds.

Data tracking

Similarly, throughout the migration, Paf has worked with third-party data analytics service Sumo Logic to aid its migration process specifically by analysing log and metric data, performing root-cause analyses, and monitoring apps and infrastructure in real time. Westling says the Sumo Logic partnership enables “not just developers but also graphic designers and artists” to react faster and obtain customer feedback more quickly.

“We are able to deliver games faster because we can get some features for free from the new architecture, so we can go straight onto the core business when adding site features of games.” The games production team is also likely to benefit from the new DevOps structure in the long run as cross-team collaboration is slowly being bettered under the new system.

Gone are the days of inefficient handovers and long meetings, as automation takes over and allows teams to deliver on their KPIs much faster.

“By decoupling the whole system, each team has been able to work on its own part very well without having the bigger effect on everybody else we’ve had previously,” Maley explains. “We also have an architecture board to try and keep it moving in the right direction.

We have representatives from each area that provide an update on how they are going with architectural work in their own areas to make sure there is no duplication and they are all aligning to the architecture principles we’ve set.”

A similar board has been established to oversee UX movements and discuss design and development. Westling, who is part of the team, says they meet every week to discuss how to implement new projects into the site and align all the different teams’ implementation.

The benefits of the new system and management structure are evident, but the shift is not without its challenges. One example Maley alludes to is network failures that arise within a distributed application. “In-house it’s been under one big application server,” he says.

But from a personal and more operational perspective it’s the huge change in mindset and understanding of internal processes staff have to take on. One way to ease the move is to adopt the cloud in stages and have staff get used to one new process at a time.

“What we’ve tried to do internally is do it sustainably; doing it gradually step by step and introducing the different parts of AWS. If we did it all in one go we would become less reliable as we wouldn’t have the skillsets in-house,” Maley points out.

The real long-term benefits and challenges of the move to AWS will only arise once the roadmap is complete and everything at Paf sits on the cloud. And as the company looks to the future to be more tech-savvy and have a deeper focus on responsible gambling and compliance, the cloud will help keep it afloat and move alongside its industry peers.