, communication is more than just a feature—it’s a fundamental part of the social experience. Whether you’re leading your clan into battle, organizing a village, or simply chatting while gathering resources, the way you connect with others shapes the world around you.

seamless, fully integrated chat and voice system

We're also truly honored that Mainframe is among the pioneers of this project, along with a few other publishers, making Pax Dei one of the first games to implement 

 (Software Development Kit), bringing Discord’s social and voice features directly into the game in a way that feels natural and immersive.

Several Mainframe developers collaborated with the Discord team on the Social SDK as an early partner. They developed a specific plug-in for Pax Dei, aiming to also help other developers use Discord as an in-game social platform in the future.. 

Our Discord integration has been live in Pax Dei for several months now. While we continue refining the experience, we’re excited to finally share some of the behind-the-scenes work that made it all possible!

—functional, but not quite at the level we envisioned. We wanted something better, something that truly elevated how players connect. 

 because it’s the place with the world's largest and most engaged community of players, and we wanted our community to feel “at home” when using our chat system.

, ensuring players can communicate effortlessly.

, so you can stay connected even outside of Pax Dei, and can still be in touch with your non-Paxian friends while playing Pax Dei.

, using Discord’s server-to-server APIs to power Clan, Party, and Zone chat.

Discord_Friends.jpg

Discord_InGame 4.jpg

chat isn’t just a background tool—it’s part of the game itself

. Whether you’re trading goods, forming alliances, or roleplaying your character’s journey, communication is central to your experience. That’s why we designed our system to allow you to:

Chat with your Pax Dei friends and non-Paxians friends alike: 

this way you don’t miss a conversation, even in the middle of a dungeon.

, allowing players to communicate naturally with those nearby.

, ensuring every interaction feels fluid and immersive.

Discord Side By Side.jpg

https://www.youtube.com/watch?v=TRkSq89TcCM

One of the biggest challenges we faced was 

managing in-game identity across multiple avatars

. Since Pax Dei allows players to switch between different avatars under one Discord account, we needed a way to keep identities clear while maintaining the flexibility players expect.

 with the avatar’s name and ID, so it’s always clear 

, ensuring clarity even outside of the game.
✔ 

, so players always see the right name, not just a Discord username.

This system ensures that whether you’re chatting in-game or through Discord, your Pax Dei identity remains consistent.

Discord_InGame 1.jpg

A great social system isn’t just about communication—it’s also about 

 to display where players are in-game. However, we found that many players disable Rich Presence, and it didn’t distinguish between in-game friends and general Discord friends. Discord added in-game friends recently, and we’re looking forward to adding this feature to Pax Dei users.

keep players connected without compromising security

, and we’ll continue refining how much of a player’s Discord identity is visible within Pax Dei.

. We’re continuing to improve the experience, tweak features, and make communication in Pax Dei feel as immersive and intuitive as the world itself.

Tech Insights: how we integrated Discord’s Chat & Voice in Pax Dei (en anglais)

 Hello Paxians, In  Pax Dei , communication is more than just a feature—it’s a fundamental part of the social experience. Whether you’re leading your ...

We hope you enjoyed celebrating April Fools with us — and more importantly, we hope you had a blast interacting with Velune the Will-o'-Wisp! With its sharp wit and mischievous charm, Velune certainly kept us all entertained throughout the day.

A special thank you to everyone who joined in and solved the riddles shared on Discord and our social channels. Your creativity and cleverness truly impressed us! Here are the answers to the riddles from Discord:

I illuminate the gloom, follow me to your doom, I am the dancer in the dark. What am I?

I am the mercy of Zeb, I am the reflection in dark waters and I am the peace of the Warmaster. What am I?

I am older than kings, yet I have no crown. I bend for no lord, all must kneel before me. I am silent, yet I convey the echoes of time. What am I?

I am silent in stone but sing when set free. Some fear me, some wield me. Without me, kingdoms would fall. What am I?

By hammer and hand, I shape the land. Without me, thou art unarmed. What am I?

And if you missed the riddles shared on social media, you can catch them all on 

As a token of our gratitude for your participation, we’re excited to unveil an 

 — a mesmerizing concept art piece featuring a 

. We hope you’ll enjoy this magical creation as much as we do.

Now, things may be back to normal... but who knows? Perhaps Velune will return next year — or even sooner — to stir up more fun and challenge your wits once again.

From April Fools to April Fay (en anglais)

 Hello Paxians, We hope you enjoyed celebrating April Fools with us — and more importantly, we hope you had a blast interacting with Velune the Will-o...

We know some of you are eager to learn what’s happening on the backend. Today, we’d like to shed some light on the recent changes we made to our database, why we made the upgrade and the consequences for Pax Dei.

To improve the reliability and scalability of our system, we recently migrated from our legacy Amazon RDS Postgres system to Amazon Aurora (Postgres compatible). This migration required us to schedule two maintenance breaks to make necessary database changes with a double goal.

[Unsupported inline type: unordered-list]

With the migration to Amazon Aurora now complete, we now have a better uptime and enhanced reliability with faster failover, as well as seamless scalability to support future growth. 

This migration positions us to better serve you, our community, by providing a more robust and flexible database infrastructure.

Stay tuned for more insights on Pax Dei development!

Tech Insights - Migration Update: Upgrading from Amazon RDS Postgres to Amazon Aurora (en anglais)

 Hello Paxians, We know some of you are eager to learn what’s happening on the backend. Today, we’d like to shed some light on the recent changes we m...

Ever wondered how medieval artifacts, clothing, or tools made their way into the world of 

, we explore something intimate, functional, and surprisingly adventurous: 14th-century undergarments.

 don't quite resemble those you wear today. That's because, in the 14th century, boxer briefs and brassieres were not part of everyday life.

Instead, people relied on more straightforward, less glamorous options: 

. Here's a look at how these historical garments came to life in the world of 

BraiseAndHoes.jpg

Subligars were a type of loincloth worn by both men and women. Typically crafted from linen, they were designed to be tied around the waist and wrapped around the groin. This straightforward design made them practical for physical activities and labor.

epidaure-epidavros-museum-726.jpg

 — a simple breast-binding garment consisting of a strip of cloth wrapped tightly around the upper body to provide support during strenuous activities.

strophium statue.jpg

These garments weren't worn all the time. They were reserved for sports, labor, or combat. In fact, even before the medieval era, some Roman gladiators would enter the arena dressed in nothing but a subligar, ready for battle.

Bringing 14th-Century Undergarments to Life

Recreating historically accurate undergarments in 

 was a challenge we embraced wholeheartedly. We wanted the details to be authentic, right down to how the garments were worn.

: Since the subligars and the strophiums were made of linen and used no modern stitching techniques, we ensured that each fold and wrap in the fabric was consistent with historical practices.

: In the 14th century, these garments weren't sewn together like modern clothing. Instead, they were folded, tied, and wrapped. This meant we had to be meticulous about avoiding seams and stitches in our models.

Pax Dei - Undergarments 3D and IG.jpg

: We designed undergarments in Pax Dei with a focus on their functional nature. The simplicity of the subligar and strophium allowed for ease of movement, reflecting their use during labor or combat scenarios.

Pax Dei - Undergarments UV and IG.png

We believe that even the smallest details contribute to building a rich, immersive world. By including historically accurate undergarments, we add another layer of authenticity to the experience. These aren't just cosmetic choices; they're a window into the daily lives of people who lived in medieval times.

So, the next time you spot a character sporting a subligar or a strophium, remember the craftsmanship, history, and attention to detail that brought these garments to life.

How it was made: Undergarments (en anglais)

 Hello Paxians! Ever wondered how medieval artifacts, clothing, or tools made their way into the world of  Pax Dei ? You are in the right place! In th...

Les magnifiques structures que vous avez construites depuis la première Alpha continuent de nous émerveiller un peu plus chaque jour. Vous, les joueurs, avez su vous approprier le système de construction de façon extraordinaire. Nous avons donc pensé que vous souhaiteriez en savoir un peu plus sur la technologie derrière ce système, d’autant plus qu’elle est probablement l’une des plus sophistiquées de l’histoire du jeu vidéo.

Actuellement, les zones du Domaine les plus prisées comptent environ 400 000 pièces de construction. À raison de 24 Domaines par monde, il y a plusieurs millions de pièces dans chacun d’entre eux. Le taux de construction standard est de 10 000 nouvelles pièces par semaine pour un seul Domaine.

Dans un jeu multijoueur en réseau, le système de réplication permet de s’assurer que l’état du jeu dont disposent les joueurs est cohérent et à jour. En d’autres termes, ce système reçoit les informations envoyées par un joueur et les distribue aux autres joueurs.

La taille des données de construction pour un Domaine est actuellement d’environ 30 Mo. Ça ne paraît pas beaucoup comme ça, n’est-ce pas ? Mais imaginons que l’on ait 100 joueurs ; ce sont maintenant 3 Go de données qui doivent être gérés par le serveur, le client (la machine du joueur) et notre système de réplication.

Un tel volume de données nécessite un traitement très particulier, et ce, à toutes les étapes de la chaîne de production du jeu. Nous pouvons utiliser le système de réplication d’Unreal Engine pour le gameplay, mais pas pour les bâtiments. La réplication des bâtiments repose sur un serveur dorsal de réplication (qui porte donc bien son nom) qui envoie les données de construction de la base de données d’un monde aux clients et aux serveurs.

Lors de la connexion, le client comme le serveur se connectent au serveur de réplication, qui commence par envoyer l’état de la zone. Comme mentionné précédemment, les données initiales pèsent actuellement environ 30 Mo pour les zones les plus étendues. Après la réplication de l’état initial, le client et le serveur reçoivent les nouveaux événements de construction de la part du serveur de réplication. Nous utilisons le flux gRPC comme canal de communication entre les événements de construction et le serveur de réplication. Nous n’entrerons pas dans les détails concernant le gRPC. Pour la faire courte, il garantit que toutes les parties concernées, soit le serveur, le serveur de réplication et les clients, communiquent correctement et se comprennent mutuellement.

In the matrix.jpg

Ajouter une pièce de construction : sous le capot

 de ce qui se passe après qu’un joueur s’est servi de son marteau de construction :

Le client envoie une demande de construction au serveur d’Unreal ;

Le serveur vérifie les autorisations pour s’assurer que la demande de construction est autorisée ;

Si le serveur le permet, il envoie ensuite la demande de construction au serveur dorsal ;

Le serveur dorsal effectue ses propres vérifications pour autoriser ou non la demande de construction ;

Une fois autorisée, une pièce de construction est ajoutée à la base de données du monde principal ;

L’événement de construction est envoyé par la base de données à notre serveur dorsal de réplication personnalisé ;

Le serveur de réplication envoie l’événement de construction au serveur et au client ;

Le serveur et tous les clients de la zone (ici, le Domaine) font apparaître la pièce de construction ;

Le serveur vérifie l’intégrité de la structure ;

Le client calcule également l’intégrité afin de l’afficher dans l’interface utilisateur.

La réplication massive des données est le premier défi auquel nous sommes confrontés. Une fois que le client a reçu les 400 000 pièces du Domaine, le paquet de 30 Mo que nous évoquions précédemment, nous devons les rendre visibles. Votre ordinateur doit les dessiner, et c’est là que réside le deuxième grand défi.

Il est inconcevable d’utiliser un système qui dessine 400 000 pièces les unes après les autres. Nous avions besoin de quelque chose qui puisse s’adapter à nos besoins. Les deux éléments principaux que nous utilisons sont les maillages statiques instanciés (ISM) et les gestionnaires de zone.

Les ISM servent à dessiner des milliers de pièces en un seul signal de dessin au lieu de les dessiner une par une.

Les gestionnaires de zone, les parallélépipèdes que l’on voit dans la vidéo ci-dessous, contrôlent tous les accessoires et les pièces de construction qui se trouvent dans leur zone. Ils se chargent de la création et de la destruction des pièces de construction et des accessoires, des calculs d’intégrité et de bien plus encore, notamment des ISM.

Le gestionnaire examine toutes les pièces de construction standard dans sa zone (pas les stations d’artisanat ni les rangements) et les trie en fonction de l’espace. Il les insère ensuite dans des maillages statiques instanciés personnalisés. La magie peut désormais opérer.

Ceci étant dit, ce n’est pas encore fini. En plus des bâtiments, il faut également dessiner les rangements, les stations d’artisanat et les pièces de décoration, en sachant que la plupart de ces éléments ont leur propre fonctionnalité de gameplay. Tous ces accessoires sont des acteurs standard d’Unreal, mais contrairement aux bâtiments que l’on peut voir d’assez loin, ils n’apparaissent qu’autour du personnage, dans une certaine zone d’affichage.

Dans la vidéo ci-dessous, vous pouvez voir comment les gestionnaires fonctionnent et la façon dont les accessoires apparaissent ou non. Chaque boîte que vous voyez correspond à un gestionnaire. Comme expliqué ci-dessus, chaque gestionnaire contrôle les ISM et l’intégrité de sa zone propre, mais il contrôle également la visibilité des acteurs d’accessoires : stations d’artisanat, rangements et autres éléments de décoration. Selon la position du personnage et la distance d’affichage du client, le gestionnaire activera ou désactivera la visibilité de ses acteurs d’accessoires. 

Dans la vidéo, les gestionnaires qui désactivent la visibilité des accessoires sont indiqués en rouge et ceux qui l’activent sont indiqués en cyan. Le violet indique ceux qui sont en attente d’activation et le magenta ceux qui sont en attente de désactivation.

https://www.youtube.com/watch?v=ZQG9MZJavwo

Il existe un autre type d’accessoire qui nécessite un traitement particulier : les sources de lumière. Lorsque votre personnage s’en approche, les lumières fonctionnent grâce à Lumen, le système d’éclairage dynamique et global d’Unreal Engine 5, afin d’améliorer l’immersion du joueur. Cependant, à une certaine distance, les lumières proviennent de projecteurs de lumière plus légers afin d’améliorer les performances du jeu. Certains Domaines disposent de plus de 20 000 sources de lumière que vous pouvez observer simultanément !

Lights in the night.jpg

Évidemment, il y a aussi quelques contraintes au niveau du serveur. Un serveur d’Unreal Engine peut supporter 150 à 200 joueurs maximum. Si un nombre plus élevé de joueurs souhaitent être dans le même village, alors nous devons créer des instances de serveur. Voici donc un autre domaine où l’existence de systèmes personnalisés est particulièrement utile. Notre système de réplication permet aux bâtiments d’être visibles sur toutes les instances du serveur en temps réel ! Si un joueur construit quelque chose dans une instance de serveur, tous les autres joueurs se trouvant dans les autres instances du serveur verront le bâtiment changer en temps réel.

L’apparition de tous ces bâtiments autour du joueur sans qu’il y ait de lag est compliquée et s’étale sur plusieurs secondes, voire plusieurs minutes, découpées en plusieurs images. Par ailleurs, comme vous l’avez probablement deviné en lisant les différentes étapes ci-dessus, les calculs d’intégrité représentent un processus très complexe. Nos serveurs diffusent du contenu provenant du monde entier ; le calcul de l’intégrité doit donc être soigneusement configuré pour que tous les paysages soient disponibles avant même qu’ils ne soient affichés. La création d’états physiques pour les pièces doit se faire sans lag, et la recherche de chevauchements entre la physique et les calculs d’intégrité réels doit être effectuée de manière itérative

Vous savez maintenant pourquoi, lorsque vous vous connectez, il arrive souvent que vous voyiez d’abord le paysage, puis les bâtiments, et enfin les accessoires. 

Les joueurs de Pax Dei construisent bien plus rapidement que ce que nous avions prévu. L’ensemble du système a déjà fait l’objet de nombreuses optimisations, et nous avons encore de nombreux tours dans notre sac. Le système actuel devrait fonctionner correctement jusqu’à environ 500 000 pièces. Au-delà de ce chiffre, de nouveaux problèmes à résoudre risquent d’apparaître. Toutefois, nous avons une bonne nouvelle à vous annoncer ! Nous prévoyons déjà d’étendre le système à 1 million de pièces par zone.

En attendant, vous pouvez continuer à construire et

 qui ont réussi à maîtriser les pièces et l’environnement pour créer de superbes structures !

Aperçu technologique : notre système de construction

 Bonjour, Paxiens ! Les magnifiques structures que vous avez construites depuis la première Alpha continuent de nous émerveiller un peu plus chaque jo...

After a few days without major issues, most of the team started feeling relieved and enjoying a fresh start to the week. However, tech issues also know about Mondays, and a little mistake can create huge problems. This is our third Story from the War Room, and we call it the DNS Debacle - yeah, the title spoils it a bit, but as some people would say, ‘It is always the DNS’.

-------

On June 24th, at 16:45 UTC, some players began to encounter login issues. They were abruptly disconnected and found themselves unable to log back in. As the affected players were scattered across various regions, shards, and zones, our monitoring systems only detected a slight decrease in user activity and did not sound an alarm. Keeping a watchful eye on our active Discord community, one of the Mainframers spotted the troubling trend and quickly decided to raise the alarm and activate the War Room.

------

By 17:00 UTC, we were in diagnostic mode. The issue was with logins, and it was affecting a significant (and growing) portion of our player base. Although we still had a large number of active players, the curve was flatlining or starting to decline—right when it should be on the rise.
Our initial investigation through monitoring, metrics, graphs, and alerts pointed to a couple of potential smoking guns. One of our central services that handles logins was seeing increased CPU load, and our main authentication system logs were spewing errors. Additionally, we noticed issues with the system handling entitlements (determining who owns the game and what type of license they have) and decided to keep an eye on it while focusing on the most likely offenders.

-----

By 17:20 UTC, we decided to restart the authentication system, which seemed to be at the heart of the issues. In hindsight, this was a mistake. With players already facing login problems and constantly retrying, restarting the system exacerbated the issue. We ended up needing to close off all authentication requests temporarily to allow the system to stabilize, and as the game needs to refresh its authentication tokens regularly, this meant that over time all players were unable to continue playing.


TotalUsers.png

Users playing across our three regions.
Troubles started at 16:45 UTC, and were subsequently made much worse by ourselves at 17:20 UTC

----

With our authentication system back on its feet, we decided to re-allow all requests to the system. As we now had all our players attempting to log in simultaneously, the previously noticed problem with the entitlement system became glaringly obvious. The system was overwhelmed and went completely offline, meaning even those users who managed to connect appeared as if they no longer owned the game. At this point, we knew that we needed to fight on two fronts at the same time. The issue with the entitlements system couldn’t wait for the root cause behind the disconnections to be resolved. Thankfully, we were on the verge of solving this one.

---

So, what triggered this bombardment you will ask? Unfortunately, it was a self-inflicted wound. During our diagnosis, we found that some DNS hostnames were not resolving correctly. One of the engineers on the call had identified the problem - a missing DNS record, specifically a critical NS delegation record. Another engineer immediately realized what had happened.
Earlier in the day, a cleanup task involved removing unnecessary infrastructure, and this crucial record had been mistakenly deleted. The last commands in the cleanup had been issued around 16:40, and it so happens that the default TTL (time-to-live) for many DNS records is exactly 5 minutes. A few manual steps later, the record was recreated, and we had to wait patiently for DNS caches around the world to update.

--

The root cause, however, is only the triggering moment, and with the on-call staff in our War Room assessing the situation and a couple of senior backend engineers ready for action, we still needed to tackle the entitlement service issue. We decided to fight it from two fronts at the same time:


The escalation and support route. We contacted and escalated the issue up the support chain of the provider responsible for our entitlement storage. They confirmed they were overwhelmed and promised a solution would be deployed soon. The clock started ticking at 17:40 UTC.


The hotfix path. We modified our entitlement code to include a fallback method favoring our players. If we had issues resolving entitlements, but the player had previously owned a certain license type, we would assume they still did and let them pass. We already had a code path similar to this one, but it was handling the case of being rate-limited. Adding the case to have a similar result for when the entitlement service was unresponsive was an easy task, and the hotfix was reviewed, compiled, built, and rolled out within 10 minutes.

-

We now had three racehorses competing in the same race: our hotfix letting players in while the service suffered, the entitlement service itself stabilizing with fewer requests coming in and fixes being deployed, and the DNS caches worldwide updating to recognize the recreated record.
By 18:35 UTC, the situation had vastly improved. Our players were logging in successfully, the entitlement system was catching up, DNS propagation was nearing completion, and the War Room team had turned a dire situation around.

The DNS Debacle had been resolved, and we were ready for the next challenge on the horizon. 

Stories from the War Room: the DNS Debacle (en Anglais uniquement)

 After a few days without major issues, most of the team started feeling relieved and enjoying a fresh start to the week. However, tech issues also kn...

For us, D-Day started relatively calmly. All servers in all 3 geographical zones were up and running, pre-warmed, and waiting for the expected onslaught. At Noon UTC, the first players started streaming in, and during the ramp-up, all systems were holding well, with no critical automatic alerts being generated in our monitoring systems.

-----

After about an hour, we started noticing that servers handling some of the more populated zones were starting to grow an alarming number of NPCs with a commensurate effect on performance. At the same time, early signals from the ground that we obtained through Discord started mentioning heavy rubber-banding in some zones which would be an expected consequence.

Those familiar with our 

 from Alpha might remember that a runaway population of rabbits once brought us to our knees. In this case, it wasn’t specifically rabbits, but a more general swarming of mobs that apparently was triggered by a dense and very active player population where our population control was not able to keep up with the spawning of mobs.

As a temporary fix, while we were troubleshooting, we started identifying those servers that were on the brink and selectively restarting them. This meant a brief disconnect for players connected to those zones, but at least the server temporarily went back to a sane state.
Fortunately, we had been working on possibly fixing this in the previous days though we had never been really able to reproduce this under lab conditions.

We decided to fast-track the fix, which was a server-only fix that did not require a patch from the client. Building and deploying the fix to our staging servers took roughly 2 hours. We then decided to deploy it selectively to a few of the worst zones to see if it fixed the issue before deploying it in general.

Deploying a hotfix live to our whole armada of servers is always a risky proposal, and in this case, we prefer applying it to a subset of servers out of caution. Furthermore, once deployed on a zone, we would have to wait at least an hour before we could ascertain that the fix was actually working. Until then we continued to manually restart zones in distress, which was an acceptable stop-gap solution.

----

But in an invasion of this scale, we rarely have to face just one issue. From Discord we also started to receive reports of problems with recipe unlocks, with heavy delays involved in related operations.
Looking at our server, we already knew about one weak link in that system: our Redis server, which was used for replicating recipe notifications to all players (along with various other notifications, such as the damage state of party members, etc.).

This specific server can not scale simply by adding more hardware, so if it was reaching breaking limits, we would have to address it at a more fundamental level.

Again, we could temporarily mitigate the problem by restarting some related services, but we immediately started work on a more permanent fix, but that fix requires some more thorough testing before we can deploy it, so we will probably have to live with this instability one more day.

---

This distraction held our attention while a more fundamental problem was brewing under the surface and partially hidden in the fog of war. At around 16:50 UTC we started receiving reports from Discord that people had trouble placing plots and some were not seeing what they built while resources were still consumed. Originally, starting in the US region, this state also spread to the EU region but strangely left the SEA region unscathed. This battle ended up being the one that took most of our time and was the most difficult to troubleshoot, not resulting in a fix until around 02:00 UTC (D-Day + 1).

--

The culprit in this case was a link between our database and a service responsible for replicating building pieces to all players as they get built. The latter service is a managed service that has some pre-configured quota limits, which we hit at some point, leading to a throttling of the service.

Normally, this wouldn’t be a problem as we can dynamically increase the quota, but in this case, when the quota was increased and the database started communicating again with it, it had to restore a very large backlog of transactions that had piled up. Even though it was doing it diligently, it simply could not catch up, leading to very long delays between creating building blocks and replicating them to clients.

At some point, we could see a whole hour's delay. So for players, it looked like they could not build a plot, but the reality is that they didn’t see the operation conclude until after more than an hour delay. This is all very obvious with the clarity of hindsight, but it took a while for us to troubleshoot this and find the root cause. As we were still in discovery mode, we opted on the side of caution and allowed the service to drain itself at its pace to make sure no player state got lost, which took a couple of hours.

-

Once we found the root cause, we had some simple fixes to the configuration that we applied and restarted the services involved. The building state was nearly immediately restored and all regions were back to normal after this.


MapBlocks.png



During all this, we had nearly forgotten about our initial worry: zone server performance due to mob population but we were happy to observe that after having applied the hotfix to selected zones, they seemed to be well under control and actually performing much better than before, which was great news. So, in parallel to all of the above, we started a rolling update of all zone servers across all our regions, which resulted in a brief disconnect for players while zones restarted but resulted in a much healthier CPU profile for all the worlds.


NPC count.png


All in all, at 03:00 UTC (D-Day + 1) last night, after a nearly 24-hour shift, everyone in the War Room felt tired but happy. We could still hear some distant rumblings and we knew that the war was not over, but at least we had established our beachhead and were ready to move forward. 

Stories from the War Room: D-Day (en Anglais uniquement)

 For us, D-Day started relatively calmly. All servers in all 3 geographical zones were up and running, pre-warmed, and waiting for the expected onslau...

So we are now emerging from the first 24 hours of our second Alpha test. As was to be expected, many lessons were learned, and pride swallowed, but we also got some pleasant surprises.

One of the big objectives of this Alpha is to test our whole tech stack under pressure and at increasing scale with real players because no amount of automated testing and simulations can expose the various critical events that can occur in different sections of our stack and what kind of domino effect they can have on the whole service.

From a Liveops point of view, the whole Alpha test is like firing a huge rocket brimming with fuel out in space with the hope of it reaching orbit and preferably returning to Earth more or less intact, ready for another flight. The whole thing is rigged up with all the telemetry that we can so that we can get a glimpse of what is happening in real-time and be able to address issues, preferably before they lead to catastrophic failures.

Our stack is quite complex, consisting of a variety of servers running a multitude of services across multiple geographical regions. For this test, we are running around 150 virtual servers distributed between the EU and the US. About two-thirds of these are Unreal servers running specific zones in specific worlds and are automatically started as players enter those zones. The remainder are backend servers acting as API endpoints for various game services. These include inventory, avatar, groups, building, and authentication. Behind all of these are a few large databases that handle all persistence, both for customers, their characters, and associated persistent state.

All of these servers and systems, as well as selected samples of clients, generate telemetry that is gathered in an observability platform called Datadog as well as in our analytics stack, which allows us to visualize in real time the state of all components and also generate alerts automatically when certain critical conditions are observed.

During launch events such as now, we assemble a LiveOps crew that is monitoring Datadog as well as monitoring social media channels working on shifts all around the clock. This crew consists of our most seasoned site reliability engineers, software engineers, and community managers. The composition of this team is such that it should be able to troubleshoot critical conditions in a live environment, find an acceptable solution, test and deploy the said solution, and then monitor that it has indeed resolved the issue, all of this while maintaining community updated as much as possible and making sure that anything that is done affects the least amount of people. This crew all convene in the so-called War Room, which is a video conferencing channel where issues are discussed and resolved collectively. Think of this as the Apollo Flight Control (or at least that is what they like to compare themselves to).

Here are a few stories from the War Room.

On this fine Tuesday morning, we lit our engines at 11 UTC and saw an immediate influx of players across all worlds and zones. All systems were behaving normally. During such ramp-ups, the systems to keep an eye on are the authentication servers and, subsequently, the various zones that are launched on demand as players connect to different parts of the worlds. Even though this ramp-up can be quite rapid, it is still relatively organic, as players are joining in a random fashion.

As we were reaching a population of about 5K players around 12:30 UTC we started to get our first warnings. A Redis server was beginning to come dangerously close to 100% CPU. Redis servers are used for caching and fast persistence of volatile state from Unreal servers across multiple worlds in the same AWS region. They are super efficient and usually not something that easily buckles under load but you still need to have some educated initial guess about what kind of CPU capacity it will need. We usually are able to form educated guesses by simulating traffic on various endpoints but, in the case of the Redis server, it did not fall under those types of tests but instead relied on limited stress tests we did using internal players as well as special test bots. These tests never came close to simulating the current load that was happening and it was clear that it needed to be switched over to a more performant node. Usually, this is something that you can do live and transparently, but, in this case, the service had not been configured to run with a readily available replica (called Multi-AZ mode).This meant that it needed to be taken down and then restarted. This is where we were approaching unknown territory because it wasn’t clear if all systems would handle that disruption gracefully and reconnect transparently to a new Redis instance without an orderly shutdown and restart. As we didn’t have much of a choice as the server would be failing eventually, we pulled the trigger on updating the Redis instances to more performant nodes. It quickly became evident that proper reconnection did not happen, and we started getting errors and abnormal behaviour from different parts of the stack that were relying on the Redis server.

Parallel to this, we had been keeping an eye on the memory consumption of a few Unreal zone servers that were trending above the norm, resulting in poor performance and lag for players connected to those nodes. Some investigations by actually connecting to these suspicious nodes revealed the culprit. Rabbits. Rabbits everywhere. It’s always rabbits, isn’t it? A resource distribution rule that was being used to control the rabbit population was behaving badly in certain biomes resulting in a proliferation of rabbits. Now even though a rabbit seems to be an innocent creature, it actually counts as a full creature in our NPC count. When a server is at its full player capacity, NPCs need to be kept in control as they compete for the same CPU resources of the server for things like collision detection and pathfinding. Updating the resource distribution to kill off these rabbits was easy enough, but in this case, it would require a reboot of the zones affected. As we didn’t have a clear view of which node might be affected by rabbit infestations and considering that we were already fighting another disturbance due to the Redis restart, as well as needing to apply the same changes to different regions of the world, it was decided to put the whole game into maintenance mode so that we could reboot everything properly.

Maintenance mode essentially cuts access to all players, so it immediately brought the service down, allowing us to reboot the world and finalize some of the changes we wanted to apply. We want to avoid this as much as possible because it is 100% disruptive to our player experience. Still, there is also another challenge associated with it: going out of maintenance mode.

The problem here is that going into maintenance mode with many thousands of players already connected means that when you want to reopen the service shortly after, there is a crowd of thousands of players reconnecting nearly at the same instant. This is very different from the organic growth pattern of connections across a typical day but more like a tsunami of simultaneous connections happening within a few minutes. Most of our stack has built-in protection for that in terms of auto-scaling and load balancing, but there are still some bottlenecks that can quickly become problematic in these scenarios. These bottlenecks can then have a domino effect on other systems that are waiting for connections and might eventually time out, creating a perfect storm. Usually, this all irons itself in the end though the player experience during that phase might be less than optimal. Our goal in the future is to improve our maintenance mode to include some form of throttling of incoming connections, allowing us to have an orderly ramp-up of connections, but for now, it is instructive for us to see which services become bottlenecks and helps us identify where we might want to have different configurations to handle these surges better.

After about 20 minutes of downtime, we were back on the saddle with players ramping steadily as US customers started to join the game in addition to the EU. At around 17:30 UTC we had reached what we anticipated to be our peak connected players for the day.

At this point, we started observing some turbulence in the force and in this case, it appeared to be related to database congestion. It seemed to affect mostly one region but because some of our DB queries straddle multiple DBs in multiple regions, the effect could be felt everywhere. Our DBs seemed to be properly provisioned and shouldn’t be going anywhere close to 100% CPU even under this high load, but something was causing a DB to crawl to a halt. We finally narrowed it down to one rogue DB connection that was hanging and keeping a critical table locked causing any queries using that table to block. This again had a domino effect as the requests to the backend that were initiating those queries eventually timed out, leading to either retries or failures of specific player activities. This was a good example of a single loose screw having radiating effects across the whole stack and regions. Once we released the lock on the table, most services started recovering gracefully, and we quickly came back to prior player levels.

During the rest of the night, we had a few alerts that needed to be dealt with, but fortunately, these were all issues we could fix live without disrupting the whole service and without needing to wake up more people.

The lessons learned are that usually, once you manage to find the root cause of a problem, the cure is relatively obvious and you might also be slightly embarrassed of not having caught that specific problem before going live. The truth, however, is that the type of issues that come up and the combination of multiple different systems exhibiting different behaviour under real player traffic is difficult to predict in totality, and often it can be challenging to discern what the real cause is because of the interdependencies. At the same time, there is a certain thrill to the hunt and a lot of gratification once a resolution is reached.

And this is why we are running a test like Alpha - and are so grateful to the community participating in an event like this, helping discover these hard-to-find issues and create more robust systems to handle them in the future. Once again, we want to extend a sincere thank you to our whole community for being so supportive and passionate about Pax Dei.

Stories From the War-Room (en Anglais uniquement)

 So we are now emerging from the first 24 hours of our second Alpha test. As was to be expected, many lessons were learned, and pride swallowed, but w...

Notes de patch