, communication is more than just a feature—it’s a fundamental part of the social experience. Whether you’re leading your clan into battle, organizing a village, or simply chatting while gathering resources, the way you connect with others shapes the world around you.

seamless, fully integrated chat and voice system

We're also truly honored that Mainframe is among the pioneers of this project, along with a few other publishers, making Pax Dei one of the first games to implement 

 (Software Development Kit), bringing Discord’s social and voice features directly into the game in a way that feels natural and immersive.

Several Mainframe developers collaborated with the Discord team on the Social SDK as an early partner. They developed a specific plug-in for Pax Dei, aiming to also help other developers use Discord as an in-game social platform in the future.. 

Our Discord integration has been live in Pax Dei for several months now. While we continue refining the experience, we’re excited to finally share some of the behind-the-scenes work that made it all possible!

—functional, but not quite at the level we envisioned. We wanted something better, something that truly elevated how players connect. 

 because it’s the place with the world's largest and most engaged community of players, and we wanted our community to feel “at home” when using our chat system.

, ensuring players can communicate effortlessly.

, so you can stay connected even outside of Pax Dei, and can still be in touch with your non-Paxian friends while playing Pax Dei.

, using Discord’s server-to-server APIs to power Clan, Party, and Zone chat.

Discord_Friends.jpg

Discord_InGame 4.jpg

chat isn’t just a background tool—it’s part of the game itself

. Whether you’re trading goods, forming alliances, or roleplaying your character’s journey, communication is central to your experience. That’s why we designed our system to allow you to:

Chat with your Pax Dei friends and non-Paxians friends alike: 

this way you don’t miss a conversation, even in the middle of a dungeon.

, allowing players to communicate naturally with those nearby.

, ensuring every interaction feels fluid and immersive.

Discord Side By Side.jpg

https://www.youtube.com/watch?v=TRkSq89TcCM

One of the biggest challenges we faced was 

managing in-game identity across multiple avatars

. Since Pax Dei allows players to switch between different avatars under one Discord account, we needed a way to keep identities clear while maintaining the flexibility players expect.

 with the avatar’s name and ID, so it’s always clear 

, ensuring clarity even outside of the game.
✔ 

, so players always see the right name, not just a Discord username.

This system ensures that whether you’re chatting in-game or through Discord, your Pax Dei identity remains consistent.

Discord_InGame 1.jpg

A great social system isn’t just about communication—it’s also about 

 to display where players are in-game. However, we found that many players disable Rich Presence, and it didn’t distinguish between in-game friends and general Discord friends. Discord added in-game friends recently, and we’re looking forward to adding this feature to Pax Dei users.

keep players connected without compromising security

, and we’ll continue refining how much of a player’s Discord identity is visible within Pax Dei.

. We’re continuing to improve the experience, tweak features, and make communication in Pax Dei feel as immersive and intuitive as the world itself.

Tech Insights: how we integrated Discord’s Chat & Voice in Pax Dei (auf englisch)

 Hello Paxians, In  Pax Dei , communication is more than just a feature—it’s a fundamental part of the social experience. Whether you’re leading your ...

We hope you enjoyed celebrating April Fools with us — and more importantly, we hope you had a blast interacting with Velune the Will-o'-Wisp! With its sharp wit and mischievous charm, Velune certainly kept us all entertained throughout the day.

A special thank you to everyone who joined in and solved the riddles shared on Discord and our social channels. Your creativity and cleverness truly impressed us! Here are the answers to the riddles from Discord:

I illuminate the gloom, follow me to your doom, I am the dancer in the dark. What am I?

I am the mercy of Zeb, I am the reflection in dark waters and I am the peace of the Warmaster. What am I?

I am older than kings, yet I have no crown. I bend for no lord, all must kneel before me. I am silent, yet I convey the echoes of time. What am I?

I am silent in stone but sing when set free. Some fear me, some wield me. Without me, kingdoms would fall. What am I?

By hammer and hand, I shape the land. Without me, thou art unarmed. What am I?

And if you missed the riddles shared on social media, you can catch them all on 

As a token of our gratitude for your participation, we’re excited to unveil an 

 — a mesmerizing concept art piece featuring a 

. We hope you’ll enjoy this magical creation as much as we do.

Now, things may be back to normal... but who knows? Perhaps Velune will return next year — or even sooner — to stir up more fun and challenge your wits once again.

From April Fools to April Fay (auf englisch)

 Hello Paxians, We hope you enjoyed celebrating April Fools with us — and more importantly, we hope you had a blast interacting with Velune the Will-o...

We know some of you are eager to learn what’s happening on the backend. Today, we’d like to shed some light on the recent changes we made to our database, why we made the upgrade and the consequences for Pax Dei.

To improve the reliability and scalability of our system, we recently migrated from our legacy Amazon RDS Postgres system to Amazon Aurora (Postgres compatible). This migration required us to schedule two maintenance breaks to make necessary database changes with a double goal.

[Unsupported inline type: unordered-list]

With the migration to Amazon Aurora now complete, we now have a better uptime and enhanced reliability with faster failover, as well as seamless scalability to support future growth. 

This migration positions us to better serve you, our community, by providing a more robust and flexible database infrastructure.

Stay tuned for more insights on Pax Dei development!

Tech Insights - Migration Update: Upgrading from Amazon RDS Postgres to Amazon Aurora (auf englisch)

 Hello Paxians, We know some of you are eager to learn what’s happening on the backend. Today, we’d like to shed some light on the recent changes we m...

Ever wondered how medieval artifacts, clothing, or tools made their way into the world of 

, we explore something intimate, functional, and surprisingly adventurous: 14th-century undergarments.

 don't quite resemble those you wear today. That's because, in the 14th century, boxer briefs and brassieres were not part of everyday life.

Instead, people relied on more straightforward, less glamorous options: 

. Here's a look at how these historical garments came to life in the world of 

BraiseAndHoes.jpg

Subligars were a type of loincloth worn by both men and women. Typically crafted from linen, they were designed to be tied around the waist and wrapped around the groin. This straightforward design made them practical for physical activities and labor.

epidaure-epidavros-museum-726.jpg

 — a simple breast-binding garment consisting of a strip of cloth wrapped tightly around the upper body to provide support during strenuous activities.

strophium statue.jpg

These garments weren't worn all the time. They were reserved for sports, labor, or combat. In fact, even before the medieval era, some Roman gladiators would enter the arena dressed in nothing but a subligar, ready for battle.

Bringing 14th-Century Undergarments to Life

Recreating historically accurate undergarments in 

 was a challenge we embraced wholeheartedly. We wanted the details to be authentic, right down to how the garments were worn.

: Since the subligars and the strophiums were made of linen and used no modern stitching techniques, we ensured that each fold and wrap in the fabric was consistent with historical practices.

: In the 14th century, these garments weren't sewn together like modern clothing. Instead, they were folded, tied, and wrapped. This meant we had to be meticulous about avoiding seams and stitches in our models.

Pax Dei - Undergarments 3D and IG.jpg

: We designed undergarments in Pax Dei with a focus on their functional nature. The simplicity of the subligar and strophium allowed for ease of movement, reflecting their use during labor or combat scenarios.

Pax Dei - Undergarments UV and IG.png

We believe that even the smallest details contribute to building a rich, immersive world. By including historically accurate undergarments, we add another layer of authenticity to the experience. These aren't just cosmetic choices; they're a window into the daily lives of people who lived in medieval times.

So, the next time you spot a character sporting a subligar or a strophium, remember the craftsmanship, history, and attention to detail that brought these garments to life.

How it was made: Undergarments (auf englisch)

 Hello Paxians! Ever wondered how medieval artifacts, clothing, or tools made their way into the world of  Pax Dei ? You are in the right place! In th...

Seit Beginn der Alpha habt ihr viele wunderschöne Bauwerke errichtet, die uns jeden Tag aufs Neue zum Staunen bringen. Das Bausystem ist das bisher meistgenutzte Feature im Spiel. Deshalb dachten wir, dass es euch vielleicht interessieren könnte, ein bisschen mehr über die Technologie zu erfahren, die hinter einem der wahrscheinlich komplexesten Systeme der Videospielgeschichte steht.

Derzeit gibt es in den beliebtesten Heimattal-Zonen rund 400.000 Bauteile. Bei insgesamt 24 Heimattälern pro Welt kommen so in jeder Welt viele Millionen Bauteile zusammen. Die durchschnittliche Baurate liegt bei 10.000 neuen Teilen pro Woche in einem einzelnen Heimattal.

Das Replikationssystem sorgt in netzwerkbasierten Multiplayer-Spielen dafür, dass der Spielstatus aller Spieler einheitlich und aktuell ist. Mit anderen Worten: Es empfängt die Input-Informationen eines Spielers und gibt sie an die anderen Spieler weiter.

Die Größe der Baudaten für ein einziges Heimattal beträgt derzeit etwa 30 MB. Ziemlich wenig, oder? Aber stellt euch einmal vor, es spielen 100 Spieler gleichzeitig – das sind schon circa 3 GB an Daten, die vom Server, dem Client (dem PC der Spieler) und unserem Replikationssystem verarbeitet werden müssen.

Die Datenmenge ist so groß, dass sie auf ganz besondere Weise gehandhabt werden muss, und das in allen Phasen des Spielprozesses. Wenn wir die Replikation der Unreal Engine schon für das Gameplay verwenden, können wir sie nicht mehr für die Gebäude nutzen. Daher läuft unsere Gebäudereplikation über einen Repli-Backend-Server, dem wir den kreativen Namen „Repli“ gegeben haben. Er sendet die Baudaten aus der Weltdatenbank an die Clients und Server.

Beim Einloggen stellen sowohl der Client als auch der Server eine Verbindung zum Repli her, der ihnen zunächst den aktuellen Gesamtstatus der Zone sendet. Wie schon erwähnt liegen die Anfangsdaten in den größten Zonen derzeit bei etwa 30 MB. Nach der Replikation des Anfangszustands warten der Client und der Server auf neue Bau-Events vom Repli. Wir verwenden gRPC-Streaming als Kommunikationskanal, um die Bau-Events vom Repli zu übertragen, aber wir wollen uns jetzt nicht in Details über gRPC verlieren. Kurz gesagt: Es stellt sicher, dass alle beteiligten Parteien (der Server, der Repli und die Clients) richtig kommunizieren und sich gegenseitig verstehen.

In the matrix.jpg

Ein Bauteil hinzufügen: hinter den Kulissen


Das alles passiert, nachdem ein Spieler den Bauhammer geschwungen hat:

Der Client sendet eine Bau-Anfrage an den Unreal-Server.

Der Server führt ein paar Berechtigungsprüfungen aus, um festzustellen, ob die Bau-Anfrage zulässig ist.

Wenn der Server die Bau-Anfrage genehmigt, sendet er sie an das Backend.

Das Backend führt seine eigenen Prüfungen aus, um die Bau-Anfrage zuzulassen oder abzulehnen.

Sobald die Genehmigung erteilt ist, wird das Bauteil der Hauptdatenbank der Welt hinzugefügt.

Dann wird das Bau-Event von der Datenbank an unseren Repli-Backend-Server gesendet.

Der Repli sendet das Bau-Event sowohl an den Server als auch an den Client.

Der Server und alle Clients in der Zone – in diesem Fall das Heimattal – erzeugen das Bauteil.

Der Server prüft, ob die strukturelle Integrität des Bauwerks gegeben ist.

Der Client berechnet ebenfalls die Integrität, um sie über die Benutzeroberfläche anzuzeigen.

Wie wird das Gebäude für alle Spieler sichtbar?

Die massenhafte Datenreplikation ist nur eine von vielen Herausforderungen. Nachdem der Client alle 400.000 Bauteile des Heimattals (das eben erwähnte 30-MB-Paket) empfangen hat, müssen wir sie sichtbar machen. Der Spieler-PC muss sie also zeichnen und hier kommt die zweite große Herausforderung.

Es ist nicht möglich, ein System zu verwenden, das 400.000 Teile hintereinander zeichnet. Wir brauchten etwas, das auf unsere Bedürfnisse zugeschnitten ist. Die beiden wichtigsten Elemente, die wir dafür verwenden, sind sogenannte instanziierte Static Meshes (ISM) und Bereichsmanager

Mithilfe des ISM können wir mehrere, ja sogar Tausende Teile auf einmal zeichnen lassen, anstatt alle einzeln nacheinander abzuarbeiten.

Die Bereichsmanager (die kastenförmigen Strukturen in dem Video unten), kontrollieren alle Bau- und Zubehörteile in ihren Bereichen. Ihre Aufgabe ist es, Bau- und Zubehörteile zu erstellen und zu zerstören, die Integrität zu berechnen und andere Funktionen zu verwalten, wie z. B. das ISM.

Der Manager sieht sich alle Standardbauteile in seinem Bereich an – nicht die Werkstätten oder die Behälter – und sortiert sie räumlich. Dann fügt er sie in benutzerdefinierte instanziierte Static Meshes ein. Und das macht die Magie möglich.

Aber wir sind noch nicht fertig. Neben den Gebäuden müssen auch die Behälter, Werkstätten und Dekorationen, von denen die meisten ihre eigenen Spielfunktionen besitzen, gezeichnet werden. Alle diese Zubehörteile sind Standardakteure der Unreal Engine, aber im Gegensatz zu den Gebäuden, die man schon von Weitem sehen kann, tauchen sie in Streaming-Distanz direkt um den Charakter herum auf.

In dem Video unten seht ihr, wie die Manager arbeiten und wann die Zubehörteile sichtbar werden oder nicht. Jeder „Kasten“, den du siehst, ist ein Manager. Wie wir oben erklärt haben, kontrolliert jeder Manager das ISM und die Integrität in seinem eigenen Bereich. Zusätzlich dazu kontrollieren sie die Sichtbarkeit der Zubehörteile, d. h. Werkstätten, Behälter und andere Dekorationsteile. Je nachdem, wo sich ein Charakter befindet und wie weit die Streaming-Distanz seines Clients reicht, aktiviert oder deaktiviert der Manager die Sichtbarkeit aller Zubehör-Akteure. 

Im Video werden die Manager, welche die Sichtbarkeit der Zubehörteile deaktivieren, rot dargestellt, während die aktiven türkis sind. Lila zeigt an, dass sie noch aktiviert werden müssen und Magenta, dass sie noch deaktiviert werden müssen.

https://www.youtube.com/watch?v=ZQG9MZJavwo

Es gibt noch eine Art von Zubehörteilen, die speziell gehandhabt werden muss: die Lichtquellen. Wenn sich dein Charakter in ihrer Nähe befindet, werden die Lichter mit Unreal Lumen – dem dynamischen globalen Beleuchtungssystem der Unreal Engine 5 – gerendert, um die Immersion zu verbessern. Ab einer bestimmten Distanz werden die Lichter nur noch mithilfe einfacher Lichtkarten dargestellt, um die Leistung zu verbessern. In manchen Tälern gibt es mehr als 20.000 Lichter zu sehen!

Lights in the night.jpg

Natürlich gelten auch auf Serverseite einige Einschränkungen. Ein einzelner Unreal-Engine-Server kann nur maximal 150–200 Spieler verarbeiten. Wenn sich mehr Spieler im selben Dorf aufhalten wollen, müssen wir Serverinstanzen erstellen. Das ist ein weiterer Bereich, in dem uns unsere eigenen, speziell für unsere Bedürfnisse entwickelten Systeme sehr helfen. Denn dank unseres Replikationssystems sind die Gebäude auf allen Serverinstanzen in Echtzeit sichtbar! Wenn ein Spieler in einer Serverinstanz ein Gebäude baut, sehen alle Spieler in allen anderen Serverinstanzen gleichzeitig, wie sich das Gebäude verändert.

Es ist ganz schön kompliziert, all diese Gebäude und Objekte um den Spieler herum zu erzeugen, ohne dass es zu sichtbaren „Aufhängern“ kommt, und es dauert mehrere Sekunden oder sogar Minuten über viele Frames. Und wie ihr vielleicht schon anhand der Schritt-für-Schritt-Beschreibung oben erraten habt, ist die Integritätsberechnung ein sehr komplexer Prozess. Unsere Server streamen außerdem die Weltinhalte – das bedeutet, dass die Integritätsberechnung so eingerichtet werden muss, dass alle Landschaften schon vorhanden sind, bevor die Berechnung überhaupt beginnt. Wir müssen die physikalischen Zustände für die Bauteile erstellen, ohne dass sich das Spiel aufhängt, und dann iterativ abfragen, wo sich diese Zustände mit den eigentlichen Integritätsberechnungen überschneiden.

Jetzt wisst ihr, warum ihr beim Einloggen oft zuerst die Landschaft, dann die Gebäude und am Schluss die Zubehörteile seht. 

Die Baumeister von Pax Dei haben viel schneller gebaut, als wir ursprünglich erwartet hatten. Das ganze System hat schon viele Optimierungen hinter sich und wir haben noch viele Tricks in petto. Das aktuelle System sollte gut funktionieren, bis etwa 500.000 Bauteile erreicht sind. Danach werden neue Probleme auftreten, die wir lösen müssen. Aber es gibt gute Neuigkeiten! Wir planen bereits, das System auf 1.000.000 Bauteile pro Zone zu erweitern.
Aber das ist eine Geschichte für ein andermal.

Bis dahin wünschen wir euch viel Spaß beim Weiterbauen.

 für euch, die die Kunst des Bauens gemeistert haben, damit auch ihr viele prächtige Bauwerke erschaffen könnt!

Technische Einblicke: Unser Bausystem

 Hello, Paxians! Seit Beginn der Alpha habt ihr viele wunderschöne Bauwerke errichtet, die uns jeden Tag aufs Neue zum Staunen bringen. Das Bausystem ...

After a few days without major issues, most of the team started feeling relieved and enjoying a fresh start to the week. However, tech issues also know about Mondays, and a little mistake can create huge problems. This is our third Story from the War Room, and we call it the DNS Debacle - yeah, the title spoils it a bit, but as some people would say, ‘It is always the DNS’.

-------

On June 24th, at 16:45 UTC, some players began to encounter login issues. They were abruptly disconnected and found themselves unable to log back in. As the affected players were scattered across various regions, shards, and zones, our monitoring systems only detected a slight decrease in user activity and did not sound an alarm. Keeping a watchful eye on our active Discord community, one of the Mainframers spotted the troubling trend and quickly decided to raise the alarm and activate the War Room.

------

By 17:00 UTC, we were in diagnostic mode. The issue was with logins, and it was affecting a significant (and growing) portion of our player base. Although we still had a large number of active players, the curve was flatlining or starting to decline—right when it should be on the rise.
Our initial investigation through monitoring, metrics, graphs, and alerts pointed to a couple of potential smoking guns. One of our central services that handles logins was seeing increased CPU load, and our main authentication system logs were spewing errors. Additionally, we noticed issues with the system handling entitlements (determining who owns the game and what type of license they have) and decided to keep an eye on it while focusing on the most likely offenders.

-----

By 17:20 UTC, we decided to restart the authentication system, which seemed to be at the heart of the issues. In hindsight, this was a mistake. With players already facing login problems and constantly retrying, restarting the system exacerbated the issue. We ended up needing to close off all authentication requests temporarily to allow the system to stabilize, and as the game needs to refresh its authentication tokens regularly, this meant that over time all players were unable to continue playing.


TotalUsers.png

Users playing across our three regions.
Troubles started at 16:45 UTC, and were subsequently made much worse by ourselves at 17:20 UTC

----

With our authentication system back on its feet, we decided to re-allow all requests to the system. As we now had all our players attempting to log in simultaneously, the previously noticed problem with the entitlement system became glaringly obvious. The system was overwhelmed and went completely offline, meaning even those users who managed to connect appeared as if they no longer owned the game. At this point, we knew that we needed to fight on two fronts at the same time. The issue with the entitlements system couldn’t wait for the root cause behind the disconnections to be resolved. Thankfully, we were on the verge of solving this one.

---

So, what triggered this bombardment you will ask? Unfortunately, it was a self-inflicted wound. During our diagnosis, we found that some DNS hostnames were not resolving correctly. One of the engineers on the call had identified the problem - a missing DNS record, specifically a critical NS delegation record. Another engineer immediately realized what had happened.
Earlier in the day, a cleanup task involved removing unnecessary infrastructure, and this crucial record had been mistakenly deleted. The last commands in the cleanup had been issued around 16:40, and it so happens that the default TTL (time-to-live) for many DNS records is exactly 5 minutes. A few manual steps later, the record was recreated, and we had to wait patiently for DNS caches around the world to update.

--

The root cause, however, is only the triggering moment, and with the on-call staff in our War Room assessing the situation and a couple of senior backend engineers ready for action, we still needed to tackle the entitlement service issue. We decided to fight it from two fronts at the same time:


The escalation and support route. We contacted and escalated the issue up the support chain of the provider responsible for our entitlement storage. They confirmed they were overwhelmed and promised a solution would be deployed soon. The clock started ticking at 17:40 UTC.


The hotfix path. We modified our entitlement code to include a fallback method favoring our players. If we had issues resolving entitlements, but the player had previously owned a certain license type, we would assume they still did and let them pass. We already had a code path similar to this one, but it was handling the case of being rate-limited. Adding the case to have a similar result for when the entitlement service was unresponsive was an easy task, and the hotfix was reviewed, compiled, built, and rolled out within 10 minutes.

-

We now had three racehorses competing in the same race: our hotfix letting players in while the service suffered, the entitlement service itself stabilizing with fewer requests coming in and fixes being deployed, and the DNS caches worldwide updating to recognize the recreated record.
By 18:35 UTC, the situation had vastly improved. Our players were logging in successfully, the entitlement system was catching up, DNS propagation was nearing completion, and the War Room team had turned a dire situation around.

The DNS Debacle had been resolved, and we were ready for the next challenge on the horizon. 

Stories from the War Room: the DNS Debacle (nur auf Englisch)

 After a few days without major issues, most of the team started feeling relieved and enjoying a fresh start to the week. However, tech issues also kn...

For us, D-Day started relatively calmly. All servers in all 3 geographical zones were up and running, pre-warmed, and waiting for the expected onslaught. At Noon UTC, the first players started streaming in, and during the ramp-up, all systems were holding well, with no critical automatic alerts being generated in our monitoring systems.

-----

After about an hour, we started noticing that servers handling some of the more populated zones were starting to grow an alarming number of NPCs with a commensurate effect on performance. At the same time, early signals from the ground that we obtained through Discord started mentioning heavy rubber-banding in some zones which would be an expected consequence.

Those familiar with our 

 from Alpha might remember that a runaway population of rabbits once brought us to our knees. In this case, it wasn’t specifically rabbits, but a more general swarming of mobs that apparently was triggered by a dense and very active player population where our population control was not able to keep up with the spawning of mobs.

As a temporary fix, while we were troubleshooting, we started identifying those servers that were on the brink and selectively restarting them. This meant a brief disconnect for players connected to those zones, but at least the server temporarily went back to a sane state.
Fortunately, we had been working on possibly fixing this in the previous days though we had never been really able to reproduce this under lab conditions.

We decided to fast-track the fix, which was a server-only fix that did not require a patch from the client. Building and deploying the fix to our staging servers took roughly 2 hours. We then decided to deploy it selectively to a few of the worst zones to see if it fixed the issue before deploying it in general.

Deploying a hotfix live to our whole armada of servers is always a risky proposal, and in this case, we prefer applying it to a subset of servers out of caution. Furthermore, once deployed on a zone, we would have to wait at least an hour before we could ascertain that the fix was actually working. Until then we continued to manually restart zones in distress, which was an acceptable stop-gap solution.

----

But in an invasion of this scale, we rarely have to face just one issue. From Discord we also started to receive reports of problems with recipe unlocks, with heavy delays involved in related operations.
Looking at our server, we already knew about one weak link in that system: our Redis server, which was used for replicating recipe notifications to all players (along with various other notifications, such as the damage state of party members, etc.).

This specific server can not scale simply by adding more hardware, so if it was reaching breaking limits, we would have to address it at a more fundamental level.

Again, we could temporarily mitigate the problem by restarting some related services, but we immediately started work on a more permanent fix, but that fix requires some more thorough testing before we can deploy it, so we will probably have to live with this instability one more day.

---

This distraction held our attention while a more fundamental problem was brewing under the surface and partially hidden in the fog of war. At around 16:50 UTC we started receiving reports from Discord that people had trouble placing plots and some were not seeing what they built while resources were still consumed. Originally, starting in the US region, this state also spread to the EU region but strangely left the SEA region unscathed. This battle ended up being the one that took most of our time and was the most difficult to troubleshoot, not resulting in a fix until around 02:00 UTC (D-Day + 1).

--

The culprit in this case was a link between our database and a service responsible for replicating building pieces to all players as they get built. The latter service is a managed service that has some pre-configured quota limits, which we hit at some point, leading to a throttling of the service.

Normally, this wouldn’t be a problem as we can dynamically increase the quota, but in this case, when the quota was increased and the database started communicating again with it, it had to restore a very large backlog of transactions that had piled up. Even though it was doing it diligently, it simply could not catch up, leading to very long delays between creating building blocks and replicating them to clients.

At some point, we could see a whole hour's delay. So for players, it looked like they could not build a plot, but the reality is that they didn’t see the operation conclude until after more than an hour delay. This is all very obvious with the clarity of hindsight, but it took a while for us to troubleshoot this and find the root cause. As we were still in discovery mode, we opted on the side of caution and allowed the service to drain itself at its pace to make sure no player state got lost, which took a couple of hours.

-

Once we found the root cause, we had some simple fixes to the configuration that we applied and restarted the services involved. The building state was nearly immediately restored and all regions were back to normal after this.


MapBlocks.png



During all this, we had nearly forgotten about our initial worry: zone server performance due to mob population but we were happy to observe that after having applied the hotfix to selected zones, they seemed to be well under control and actually performing much better than before, which was great news. So, in parallel to all of the above, we started a rolling update of all zone servers across all our regions, which resulted in a brief disconnect for players while zones restarted but resulted in a much healthier CPU profile for all the worlds.


NPC count.png


All in all, at 03:00 UTC (D-Day + 1) last night, after a nearly 24-hour shift, everyone in the War Room felt tired but happy. We could still hear some distant rumblings and we knew that the war was not over, but at least we had established our beachhead and were ready to move forward. 

Stories from the War Room: D-Day (nur auf Englisch)

 For us, D-Day started relatively calmly. All servers in all 3 geographical zones were up and running, pre-warmed, and waiting for the expected onslau...

So we are now emerging from the first 24 hours of our second Alpha test. As was to be expected, many lessons were learned, and pride swallowed, but we also got some pleasant surprises.

One of the big objectives of this Alpha is to test our whole tech stack under pressure and at increasing scale with real players because no amount of automated testing and simulations can expose the various critical events that can occur in different sections of our stack and what kind of domino effect they can have on the whole service.

From a Liveops point of view, the whole Alpha test is like firing a huge rocket brimming with fuel out in space with the hope of it reaching orbit and preferably returning to Earth more or less intact, ready for another flight. The whole thing is rigged up with all the telemetry that we can so that we can get a glimpse of what is happening in real-time and be able to address issues, preferably before they lead to catastrophic failures.

Our stack is quite complex, consisting of a variety of servers running a multitude of services across multiple geographical regions. For this test, we are running around 150 virtual servers distributed between the EU and the US. About two-thirds of these are Unreal servers running specific zones in specific worlds and are automatically started as players enter those zones. The remainder are backend servers acting as API endpoints for various game services. These include inventory, avatar, groups, building, and authentication. Behind all of these are a few large databases that handle all persistence, both for customers, their characters, and associated persistent state.

All of these servers and systems, as well as selected samples of clients, generate telemetry that is gathered in an observability platform called Datadog as well as in our analytics stack, which allows us to visualize in real time the state of all components and also generate alerts automatically when certain critical conditions are observed.

During launch events such as now, we assemble a LiveOps crew that is monitoring Datadog as well as monitoring social media channels working on shifts all around the clock. This crew consists of our most seasoned site reliability engineers, software engineers, and community managers. The composition of this team is such that it should be able to troubleshoot critical conditions in a live environment, find an acceptable solution, test and deploy the said solution, and then monitor that it has indeed resolved the issue, all of this while maintaining community updated as much as possible and making sure that anything that is done affects the least amount of people. This crew all convene in the so-called War Room, which is a video conferencing channel where issues are discussed and resolved collectively. Think of this as the Apollo Flight Control (or at least that is what they like to compare themselves to).

Here are a few stories from the War Room.

On this fine Tuesday morning, we lit our engines at 11 UTC and saw an immediate influx of players across all worlds and zones. All systems were behaving normally. During such ramp-ups, the systems to keep an eye on are the authentication servers and, subsequently, the various zones that are launched on demand as players connect to different parts of the worlds. Even though this ramp-up can be quite rapid, it is still relatively organic, as players are joining in a random fashion.

As we were reaching a population of about 5K players around 12:30 UTC we started to get our first warnings. A Redis server was beginning to come dangerously close to 100% CPU. Redis servers are used for caching and fast persistence of volatile state from Unreal servers across multiple worlds in the same AWS region. They are super efficient and usually not something that easily buckles under load but you still need to have some educated initial guess about what kind of CPU capacity it will need. We usually are able to form educated guesses by simulating traffic on various endpoints but, in the case of the Redis server, it did not fall under those types of tests but instead relied on limited stress tests we did using internal players as well as special test bots. These tests never came close to simulating the current load that was happening and it was clear that it needed to be switched over to a more performant node. Usually, this is something that you can do live and transparently, but, in this case, the service had not been configured to run with a readily available replica (called Multi-AZ mode).This meant that it needed to be taken down and then restarted. This is where we were approaching unknown territory because it wasn’t clear if all systems would handle that disruption gracefully and reconnect transparently to a new Redis instance without an orderly shutdown and restart. As we didn’t have much of a choice as the server would be failing eventually, we pulled the trigger on updating the Redis instances to more performant nodes. It quickly became evident that proper reconnection did not happen, and we started getting errors and abnormal behaviour from different parts of the stack that were relying on the Redis server.

Parallel to this, we had been keeping an eye on the memory consumption of a few Unreal zone servers that were trending above the norm, resulting in poor performance and lag for players connected to those nodes. Some investigations by actually connecting to these suspicious nodes revealed the culprit. Rabbits. Rabbits everywhere. It’s always rabbits, isn’t it? A resource distribution rule that was being used to control the rabbit population was behaving badly in certain biomes resulting in a proliferation of rabbits. Now even though a rabbit seems to be an innocent creature, it actually counts as a full creature in our NPC count. When a server is at its full player capacity, NPCs need to be kept in control as they compete for the same CPU resources of the server for things like collision detection and pathfinding. Updating the resource distribution to kill off these rabbits was easy enough, but in this case, it would require a reboot of the zones affected. As we didn’t have a clear view of which node might be affected by rabbit infestations and considering that we were already fighting another disturbance due to the Redis restart, as well as needing to apply the same changes to different regions of the world, it was decided to put the whole game into maintenance mode so that we could reboot everything properly.

Maintenance mode essentially cuts access to all players, so it immediately brought the service down, allowing us to reboot the world and finalize some of the changes we wanted to apply. We want to avoid this as much as possible because it is 100% disruptive to our player experience. Still, there is also another challenge associated with it: going out of maintenance mode.

The problem here is that going into maintenance mode with many thousands of players already connected means that when you want to reopen the service shortly after, there is a crowd of thousands of players reconnecting nearly at the same instant. This is very different from the organic growth pattern of connections across a typical day but more like a tsunami of simultaneous connections happening within a few minutes. Most of our stack has built-in protection for that in terms of auto-scaling and load balancing, but there are still some bottlenecks that can quickly become problematic in these scenarios. These bottlenecks can then have a domino effect on other systems that are waiting for connections and might eventually time out, creating a perfect storm. Usually, this all irons itself in the end though the player experience during that phase might be less than optimal. Our goal in the future is to improve our maintenance mode to include some form of throttling of incoming connections, allowing us to have an orderly ramp-up of connections, but for now, it is instructive for us to see which services become bottlenecks and helps us identify where we might want to have different configurations to handle these surges better.

After about 20 minutes of downtime, we were back on the saddle with players ramping steadily as US customers started to join the game in addition to the EU. At around 17:30 UTC we had reached what we anticipated to be our peak connected players for the day.

At this point, we started observing some turbulence in the force and in this case, it appeared to be related to database congestion. It seemed to affect mostly one region but because some of our DB queries straddle multiple DBs in multiple regions, the effect could be felt everywhere. Our DBs seemed to be properly provisioned and shouldn’t be going anywhere close to 100% CPU even under this high load, but something was causing a DB to crawl to a halt. We finally narrowed it down to one rogue DB connection that was hanging and keeping a critical table locked causing any queries using that table to block. This again had a domino effect as the requests to the backend that were initiating those queries eventually timed out, leading to either retries or failures of specific player activities. This was a good example of a single loose screw having radiating effects across the whole stack and regions. Once we released the lock on the table, most services started recovering gracefully, and we quickly came back to prior player levels.

During the rest of the night, we had a few alerts that needed to be dealt with, but fortunately, these were all issues we could fix live without disrupting the whole service and without needing to wake up more people.

The lessons learned are that usually, once you manage to find the root cause of a problem, the cure is relatively obvious and you might also be slightly embarrassed of not having caught that specific problem before going live. The truth, however, is that the type of issues that come up and the combination of multiple different systems exhibiting different behaviour under real player traffic is difficult to predict in totality, and often it can be challenging to discern what the real cause is because of the interdependencies. At the same time, there is a certain thrill to the hunt and a lot of gratification once a resolution is reached.

And this is why we are running a test like Alpha - and are so grateful to the community participating in an event like this, helping discover these hard-to-find issues and create more robust systems to handle them in the future. Once again, we want to extend a sincere thank you to our whole community for being so supportive and passionate about Pax Dei.

Stories From the War-Room (nur auf Englisch)

 So we are now emerging from the first 24 hours of our second Alpha test. As was to be expected, many lessons were learned, and pride swallowed, but w...