, communication is more than just a feature—it’s a fundamental part of the social experience. Whether you’re leading your clan into battle, organizing a village, or simply chatting while gathering resources, the way you connect with others shapes the world around you.

seamless, fully integrated chat and voice system

We're also truly honored that Mainframe is among the pioneers of this project, along with a few other publishers, making Pax Dei one of the first games to implement 

 (Software Development Kit), bringing Discord’s social and voice features directly into the game in a way that feels natural and immersive.

Several Mainframe developers collaborated with the Discord team on the Social SDK as an early partner. They developed a specific plug-in for Pax Dei, aiming to also help other developers use Discord as an in-game social platform in the future.. 

Our Discord integration has been live in Pax Dei for several months now. While we continue refining the experience, we’re excited to finally share some of the behind-the-scenes work that made it all possible!

—functional, but not quite at the level we envisioned. We wanted something better, something that truly elevated how players connect. 

 because it’s the place with the world's largest and most engaged community of players, and we wanted our community to feel “at home” when using our chat system.

, ensuring players can communicate effortlessly.

, so you can stay connected even outside of Pax Dei, and can still be in touch with your non-Paxian friends while playing Pax Dei.

, using Discord’s server-to-server APIs to power Clan, Party, and Zone chat.

Discord_Friends.jpg

Discord_InGame 4.jpg

chat isn’t just a background tool—it’s part of the game itself

. Whether you’re trading goods, forming alliances, or roleplaying your character’s journey, communication is central to your experience. That’s why we designed our system to allow you to:

Chat with your Pax Dei friends and non-Paxians friends alike: 

this way you don’t miss a conversation, even in the middle of a dungeon.

, allowing players to communicate naturally with those nearby.

, ensuring every interaction feels fluid and immersive.

Discord Side By Side.jpg

https://www.youtube.com/watch?v=TRkSq89TcCM

One of the biggest challenges we faced was 

managing in-game identity across multiple avatars

. Since Pax Dei allows players to switch between different avatars under one Discord account, we needed a way to keep identities clear while maintaining the flexibility players expect.

 with the avatar’s name and ID, so it’s always clear 

, ensuring clarity even outside of the game.
✔ 

, so players always see the right name, not just a Discord username.

This system ensures that whether you’re chatting in-game or through Discord, your Pax Dei identity remains consistent.

Discord_InGame 1.jpg

A great social system isn’t just about communication—it’s also about 

 to display where players are in-game. However, we found that many players disable Rich Presence, and it didn’t distinguish between in-game friends and general Discord friends. Discord added in-game friends recently, and we’re looking forward to adding this feature to Pax Dei users.

keep players connected without compromising security

, and we’ll continue refining how much of a player’s Discord identity is visible within Pax Dei.

. We’re continuing to improve the experience, tweak features, and make communication in Pax Dei feel as immersive and intuitive as the world itself.

Tech Insights: how we integrated Discord’s Chat & Voice in Pax Dei

 Hello Paxians, In  Pax Dei , communication is more than just a feature—it’s a fundamental part of the social experience. Whether you’re leading your ...

We hope you enjoyed celebrating April Fools with us — and more importantly, we hope you had a blast interacting with Velune the Will-o'-Wisp! With its sharp wit and mischievous charm, Velune certainly kept us all entertained throughout the day.

A special thank you to everyone who joined in and solved the riddles shared on Discord and our social channels. Your creativity and cleverness truly impressed us! Here are the answers to the riddles from Discord:

I illuminate the gloom, follow me to your doom, I am the dancer in the dark. What am I?

I am the mercy of Zeb, I am the reflection in dark waters and I am the peace of the Warmaster. What am I?

I am older than kings, yet I have no crown. I bend for no lord, all must kneel before me. I am silent, yet I convey the echoes of time. What am I?

I am silent in stone but sing when set free. Some fear me, some wield me. Without me, kingdoms would fall. What am I?

By hammer and hand, I shape the land. Without me, thou art unarmed. What am I?

And if you missed the riddles shared on social media, you can catch them all on 

As a token of our gratitude for your participation, we’re excited to unveil an 

 — a mesmerizing concept art piece featuring a 

. We hope you’ll enjoy this magical creation as much as we do.

Now, things may be back to normal... but who knows? Perhaps Velune will return next year — or even sooner — to stir up more fun and challenge your wits once again.

From April Fools to April Fay

 Hello Paxians, We hope you enjoyed celebrating April Fools with us — and more importantly, we hope you had a blast interacting with Velune the Will-o...

We know some of you are eager to learn what’s happening on the backend. Today, we’d like to shed some light on the recent changes we made to our database, why we made the upgrade and the consequences for Pax Dei.

To improve the reliability and scalability of our system, we recently migrated from our legacy Amazon RDS Postgres system to Amazon Aurora (Postgres compatible). This migration required us to schedule two maintenance breaks to make necessary database changes with a double goal.

[Unsupported inline type: unordered-list]

With the migration to Amazon Aurora now complete, we now have a better uptime and enhanced reliability with faster failover, as well as seamless scalability to support future growth. 

This migration positions us to better serve you, our community, by providing a more robust and flexible database infrastructure.

Stay tuned for more insights on Pax Dei development!

Tech Insights - Migration Update: Upgrading from Amazon RDS Postgres to Amazon Aurora

 Hello Paxians, We know some of you are eager to learn what’s happening on the backend. Today, we’d like to shed some light on the recent changes we m...

Ever wondered how medieval artifacts, clothing, or tools made their way into the world of 

, we explore something intimate, functional, and surprisingly adventurous: 14th-century undergarments.

 don't quite resemble those you wear today. That's because, in the 14th century, boxer briefs and brassieres were not part of everyday life.

Instead, people relied on more straightforward, less glamorous options: 

. Here's a look at how these historical garments came to life in the world of 

BraiseAndHoes.jpg

Subligars were a type of loincloth worn by both men and women. Typically crafted from linen, they were designed to be tied around the waist and wrapped around the groin. This straightforward design made them practical for physical activities and labor.

epidaure-epidavros-museum-726.jpg

 — a simple breast-binding garment consisting of a strip of cloth wrapped tightly around the upper body to provide support during strenuous activities.

strophium statue.jpg

These garments weren't worn all the time. They were reserved for sports, labor, or combat. In fact, even before the medieval era, some Roman gladiators would enter the arena dressed in nothing but a subligar, ready for battle.

Bringing 14th-Century Undergarments to Life

Recreating historically accurate undergarments in 

 was a challenge we embraced wholeheartedly. We wanted the details to be authentic, right down to how the garments were worn.

: Since the subligars and the strophiums were made of linen and used no modern stitching techniques, we ensured that each fold and wrap in the fabric was consistent with historical practices.

: In the 14th century, these garments weren't sewn together like modern clothing. Instead, they were folded, tied, and wrapped. This meant we had to be meticulous about avoiding seams and stitches in our models.

Pax Dei - Undergarments 3D and IG.jpg

: We designed undergarments in Pax Dei with a focus on their functional nature. The simplicity of the subligar and strophium allowed for ease of movement, reflecting their use during labor or combat scenarios.

Pax Dei - Undergarments UV and IG.png

We believe that even the smallest details contribute to building a rich, immersive world. By including historically accurate undergarments, we add another layer of authenticity to the experience. These aren't just cosmetic choices; they're a window into the daily lives of people who lived in medieval times.

So, the next time you spot a character sporting a subligar or a strophium, remember the craftsmanship, history, and attention to detail that brought these garments to life.

How it was made: Undergarments

 Hello Paxians! Ever wondered how medieval artifacts, clothing, or tools made their way into the world of  Pax Dei ? You are in the right place! In th...

All the beautiful structures you have built since the first Alpha amaze us every day, and the building system is the players' most amazingly utilized feature so far. As such, we thought you might be interested in knowing a bit more about the tech behind the system - which is probably one of the most complex systems in the history of video games.

Currently, the hottest home valley zones have around 400,000 building pieces. With a total of 24 home valleys per world, there are many millions of pieces in each world. The typical build rate is 10,000 new pieces per week in a single-home valley.

In a networked multiplayer game, the replication system is used to ensure that all the players have a consistent and up-to-date game state. In other words, It receives the input information of a player and distributes it to the other players.

The building data size for a single home valley is currently around 30MB. It does not sound a lot, right? But let’s say we have 100 players; we are now talking about 3GB of data that needs to be managed by the server, the client (the player’s machine), and our replication system.

This amount is, in fact, so massive that it needs very special handling in all stages of the game pipeline. If we can use Unreal Engine replication for gameplay, we can’t use it for the buildings. The backbone of our building replication is the repli backend server - smartly named ‘repli’ - which sends build data from the world database to the clients and the servers.

When logging in, both the client and the server connect to repli, which first sends the current full state of the zone. As said, initial data is currently around 30MB in the largest zones. After the initial state replication, the client and the server listen to new build events from repli. We use gRPC streaming as a communication channel to build events from repli. We won’t go into details about gRPC. Simply put, it ensures all parties involved, the server, repli, and the clients, communicate correctly and understand each other.

In the matrix.jpg

what happens after a player swings their construction hammer:

The client sends a build request to the Unreal server;

The server does some permission checks to verify if the build request is allowed;

If the server allows it, it then sends the build request to the backend;

The backend does its own checks to allow or not the build request;

Once allowed, the building piece is added to the main world database;

The build event is sent from the database to our custom repli backend server;

Repli will send the build event to both the server and the client;

The server and all the clients in the zone - here, the Home Valley - spawn the building piece;

The server does the integrity check that the structure still holds;

The client calculates integrity as well to display it in the UI.

Massive data replication is one of many challenges. After the client has all 400,000 pieces of the home valley, the 30 MB package we discussed earlier, we need to make them visible. Your computer must draw them, and here comes the second big challenge.

It would be impossible to use a system that draws 400,000 pieces one after the other. We needed something tailored to suit our needs. The two key elements we use are instanced static meshes (ISM) and area managers.

The ISM is used to draw several, even thousands, pieces in one draw call, instead of drawing them one by one.

The area managers, the parallelepipeds boxes in the video below, control all props and building pieces in their areas. They handle the creation and destruction of building pieces and props, integrity calculations, and other things, including the ISM.

The manager looks at all the standard building pieces in its area - not the crafters nor the containers - and sorts them spatially. Then, it inserts them in custom instanced static meshes. And now the magic can happen.

That being said, we are not done yet. In addition to the buildings, we have containers, crafters, and decoration pieces that need to be drawn, and most of them have their own gameplay functionality. All these props are standard Unreal actors, but contrary to the buildings you can see from pretty far away, they spawn only around the character inside the streaming distance.

In the video below, you can see how the managers are working and how the props become visible or not. Each box you can see is a manager. As explained above, each manager controls its own area for ISM and integrity, but they also control the visibility of the prop actors - crafters, containers, and other decoration pieces. Depending on the position of a character and the streaming distance of its client, the manager will activate or deactivate the visibility of all its prop actors. 

In the video, red shows the managers that deactivate the props’ visibility and cyan the active ones. Purple shows the ones pending ongoing activation, and magenta the ones pending ongoing deactivation.

https://www.youtube.com/watch?v=ZQG9MZJavwo

There is another special type of props that needs special treatment: the lighting sources. When your character is close to them, the lights are running through the Unreal Lumen lighting - the dynamic global illumination system from Unreal Engine 5 - to improve immersion. However, the lights are switched to lightweight light cards at a certain distance to improve performance. Some valleys have more than 20,000 lights that you can observe simultaneously!

Lights in the night.jpg

Of course, we also have some constraints on the server side. One single Unreal Engine server can handle only a maximum of 150-200 players, and if more players want to be in the same village, then we need to create some server instances. That’s another area where having some custom systems helps a lot. Our replication system allows the buildings to be visible across all the server instances and in real time! If one player builds in one server instance, then all the players in the other server instances see the building changing at the same time.

Spawning all this building stuff around the player without visible stuns is complicated and is done over several seconds, even minutes, sliced into many frames. Also, as you can have guessed from our step-by-step above, the integrity calculation is a very complex process. Our servers also stream world content, so integrity calculation needs careful setup so that all supporting landscapes are there before they even start. Creating only physics states for pieces needs to be done without stuns, and after that, querying overlaps from physics and actual integrity calculations need to be done iteratively.

Now you know why when logging in, you often can see the landscape first, then the buildings, and finally the props. 

Pax Dei builders have been doing building much faster than we anticipated initially. The whole system has seen many optimization passes so far, and we still have many tricks in our sleeves. The current system should work fine until around 500.000 pieces. After that, we will start seeing new issues that must be solved. Good news, though! We already have plans to scale the system to 1.000.000 pieces per zone.

In the meantime, you can keep building and follow 

 who master the pieces and environment to create gorgeous structures!

Tech Insights: our Building System

 Hello, Paxians! All the beautiful structures you have built since the first Alpha amaze us every day, and the building system is the players' most am...

After a few days without major issues, most of the team started feeling relieved and enjoying a fresh start to the week. However, tech issues also know about Mondays, and a little mistake can create huge problems. This is our third Story from the War Room, and we call it the DNS Debacle - yeah, the title spoils it a bit, but as some people would say, ‘It is always the DNS’.

-------

On June 24th, at 16:45 UTC, some players began to encounter login issues. They were abruptly disconnected and found themselves unable to log back in. As the affected players were scattered across various regions, shards, and zones, our monitoring systems only detected a slight decrease in user activity and did not sound an alarm. Keeping a watchful eye on our active Discord community, one of the Mainframers spotted the troubling trend and quickly decided to raise the alarm and activate the War Room.

------

By 17:00 UTC, we were in diagnostic mode. The issue was with logins, and it was affecting a significant (and growing) portion of our player base. Although we still had a large number of active players, the curve was flatlining or starting to decline—right when it should be on the rise.
Our initial investigation through monitoring, metrics, graphs, and alerts pointed to a couple of potential smoking guns. One of our central services that handles logins was seeing increased CPU load, and our main authentication system logs were spewing errors. Additionally, we noticed issues with the system handling entitlements (determining who owns the game and what type of license they have) and decided to keep an eye on it while focusing on the most likely offenders.

-----

By 17:20 UTC, we decided to restart the authentication system, which seemed to be at the heart of the issues. In hindsight, this was a mistake. With players already facing login problems and constantly retrying, restarting the system exacerbated the issue. We ended up needing to close off all authentication requests temporarily to allow the system to stabilize, and as the game needs to refresh its authentication tokens regularly, this meant that over time all players were unable to continue playing.


TotalUsers.png

Users playing across our three regions.
Troubles started at 16:45 UTC, and were subsequently made much worse by ourselves at 17:20 UTC

----

With our authentication system back on its feet, we decided to re-allow all requests to the system. As we now had all our players attempting to log in simultaneously, the previously noticed problem with the entitlement system became glaringly obvious. The system was overwhelmed and went completely offline, meaning even those users who managed to connect appeared as if they no longer owned the game. At this point, we knew that we needed to fight on two fronts at the same time. The issue with the entitlements system couldn’t wait for the root cause behind the disconnections to be resolved. Thankfully, we were on the verge of solving this one.

---

So, what triggered this bombardment you will ask? Unfortunately, it was a self-inflicted wound. During our diagnosis, we found that some DNS hostnames were not resolving correctly. One of the engineers on the call had identified the problem - a missing DNS record, specifically a critical NS delegation record. Another engineer immediately realized what had happened.
Earlier in the day, a cleanup task involved removing unnecessary infrastructure, and this crucial record had been mistakenly deleted. The last commands in the cleanup had been issued around 16:40, and it so happens that the default TTL (time-to-live) for many DNS records is exactly 5 minutes. A few manual steps later, the record was recreated, and we had to wait patiently for DNS caches around the world to update.

--

The root cause, however, is only the triggering moment, and with the on-call staff in our War Room assessing the situation and a couple of senior backend engineers ready for action, we still needed to tackle the entitlement service issue. We decided to fight it from two fronts at the same time:


The escalation and support route. We contacted and escalated the issue up the support chain of the provider responsible for our entitlement storage. They confirmed they were overwhelmed and promised a solution would be deployed soon. The clock started ticking at 17:40 UTC.


The hotfix path. We modified our entitlement code to include a fallback method favoring our players. If we had issues resolving entitlements, but the player had previously owned a certain license type, we would assume they still did and let them pass. We already had a code path similar to this one, but it was handling the case of being rate-limited. Adding the case to have a similar result for when the entitlement service was unresponsive was an easy task, and the hotfix was reviewed, compiled, built, and rolled out within 10 minutes.

-

We now had three racehorses competing in the same race: our hotfix letting players in while the service suffered, the entitlement service itself stabilizing with fewer requests coming in and fixes being deployed, and the DNS caches worldwide updating to recognize the recreated record.
By 18:35 UTC, the situation had vastly improved. Our players were logging in successfully, the entitlement system was catching up, DNS propagation was nearing completion, and the War Room team had turned a dire situation around.

The DNS Debacle had been resolved, and we were ready for the next challenge on the horizon. 

Stories from the War Room: the DNS Debacle

 After a few days without major issues, most of the team started feeling relieved and enjoying a fresh start to the week. However, tech issues also kn...

For us, D-Day started relatively calmly. All servers in all 3 geographical zones were up and running, pre-warmed, and waiting for the expected onslaught. At Noon UTC, the first players started streaming in, and during the ramp-up, all systems were holding well, with no critical automatic alerts being generated in our monitoring systems.

-----

After about an hour, we started noticing that servers handling some of the more populated zones were starting to grow an alarming number of NPCs with a commensurate effect on performance. At the same time, early signals from the ground that we obtained through Discord started mentioning heavy rubber-banding in some zones which would be an expected consequence.

Those familiar with our 

 from Alpha might remember that a runaway population of rabbits once brought us to our knees. In this case, it wasn’t specifically rabbits, but a more general swarming of mobs that apparently was triggered by a dense and very active player population where our population control was not able to keep up with the spawning of mobs.

As a temporary fix, while we were troubleshooting, we started identifying those servers that were on the brink and selectively restarting them. This meant a brief disconnect for players connected to those zones, but at least the server temporarily went back to a sane state.
Fortunately, we had been working on possibly fixing this in the previous days though we had never been really able to reproduce this under lab conditions.

We decided to fast-track the fix, which was a server-only fix that did not require a patch from the client. Building and deploying the fix to our staging servers took roughly 2 hours. We then decided to deploy it selectively to a few of the worst zones to see if it fixed the issue before deploying it in general.

Deploying a hotfix live to our whole armada of servers is always a risky proposal, and in this case, we prefer applying it to a subset of servers out of caution. Furthermore, once deployed on a zone, we would have to wait at least an hour before we could ascertain that the fix was actually working. Until then we continued to manually restart zones in distress, which was an acceptable stop-gap solution.

----

But in an invasion of this scale, we rarely have to face just one issue. From Discord we also started to receive reports of problems with recipe unlocks, with heavy delays involved in related operations.
Looking at our server, we already knew about one weak link in that system: our Redis server, which was used for replicating recipe notifications to all players (along with various other notifications, such as the damage state of party members, etc.).

This specific server can not scale simply by adding more hardware, so if it was reaching breaking limits, we would have to address it at a more fundamental level.

Again, we could temporarily mitigate the problem by restarting some related services, but we immediately started work on a more permanent fix, but that fix requires some more thorough testing before we can deploy it, so we will probably have to live with this instability one more day.

---

This distraction held our attention while a more fundamental problem was brewing under the surface and partially hidden in the fog of war. At around 16:50 UTC we started receiving reports from Discord that people had trouble placing plots and some were not seeing what they built while resources were still consumed. Originally, starting in the US region, this state also spread to the EU region but strangely left the SEA region unscathed. This battle ended up being the one that took most of our time and was the most difficult to troubleshoot, not resulting in a fix until around 02:00 UTC (D-Day + 1).

--

The culprit in this case was a link between our database and a service responsible for replicating building pieces to all players as they get built. The latter service is a managed service that has some pre-configured quota limits, which we hit at some point, leading to a throttling of the service.

Normally, this wouldn’t be a problem as we can dynamically increase the quota, but in this case, when the quota was increased and the database started communicating again with it, it had to restore a very large backlog of transactions that had piled up. Even though it was doing it diligently, it simply could not catch up, leading to very long delays between creating building blocks and replicating them to clients.

At some point, we could see a whole hour's delay. So for players, it looked like they could not build a plot, but the reality is that they didn’t see the operation conclude until after more than an hour delay. This is all very obvious with the clarity of hindsight, but it took a while for us to troubleshoot this and find the root cause. As we were still in discovery mode, we opted on the side of caution and allowed the service to drain itself at its pace to make sure no player state got lost, which took a couple of hours.

-

Once we found the root cause, we had some simple fixes to the configuration that we applied and restarted the services involved. The building state was nearly immediately restored and all regions were back to normal after this.


MapBlocks.png



During all this, we had nearly forgotten about our initial worry: zone server performance due to mob population but we were happy to observe that after having applied the hotfix to selected zones, they seemed to be well under control and actually performing much better than before, which was great news. So, in parallel to all of the above, we started a rolling update of all zone servers across all our regions, which resulted in a brief disconnect for players while zones restarted but resulted in a much healthier CPU profile for all the worlds.


NPC count.png


All in all, at 03:00 UTC (D-Day + 1) last night, after a nearly 24-hour shift, everyone in the War Room felt tired but happy. We could still hear some distant rumblings and we knew that the war was not over, but at least we had established our beachhead and were ready to move forward. 

Stories from the War Room: D-Day 

 For us, D-Day started relatively calmly. All servers in all 3 geographical zones were up and running, pre-warmed, and waiting for the expected onslau...

So we are now emerging from the first 24 hours of our second Alpha test. As was to be expected, many lessons were learned, and pride swallowed, but we also got some pleasant surprises.

One of the big objectives of this Alpha is to test our whole tech stack under pressure and at increasing scale with real players because no amount of automated testing and simulations can expose the various critical events that can occur in different sections of our stack and what kind of domino effect they can have on the whole service.

From a Liveops point of view, the whole Alpha test is like firing a huge rocket brimming with fuel out in space with the hope of it reaching orbit and preferably returning to Earth more or less intact, ready for another flight. The whole thing is rigged up with all the telemetry that we can so that we can get a glimpse of what is happening in real-time and be able to address issues, preferably before they lead to catastrophic failures.

Our stack is quite complex, consisting of a variety of servers running a multitude of services across multiple geographical regions. For this test, we are running around 150 virtual servers distributed between the EU and the US. About two-thirds of these are Unreal servers running specific zones in specific worlds and are automatically started as players enter those zones. The remainder are backend servers acting as API endpoints for various game services. These include inventory, avatar, groups, building, and authentication. Behind all of these are a few large databases that handle all persistence, both for customers, their characters, and associated persistent state.

All of these servers and systems, as well as selected samples of clients, generate telemetry that is gathered in an observability platform called Datadog as well as in our analytics stack, which allows us to visualize in real time the state of all components and also generate alerts automatically when certain critical conditions are observed.

During launch events such as now, we assemble a LiveOps crew that is monitoring Datadog as well as monitoring social media channels working on shifts all around the clock. This crew consists of our most seasoned site reliability engineers, software engineers, and community managers. The composition of this team is such that it should be able to troubleshoot critical conditions in a live environment, find an acceptable solution, test and deploy the said solution, and then monitor that it has indeed resolved the issue, all of this while maintaining community updated as much as possible and making sure that anything that is done affects the least amount of people. This crew all convene in the so-called War Room, which is a video conferencing channel where issues are discussed and resolved collectively. Think of this as the Apollo Flight Control (or at least that is what they like to compare themselves to).

Here are a few stories from the War Room.

On this fine Tuesday morning, we lit our engines at 11 UTC and saw an immediate influx of players across all worlds and zones. All systems were behaving normally. During such ramp-ups, the systems to keep an eye on are the authentication servers and, subsequently, the various zones that are launched on demand as players connect to different parts of the worlds. Even though this ramp-up can be quite rapid, it is still relatively organic, as players are joining in a random fashion.

As we were reaching a population of about 5K players around 12:30 UTC we started to get our first warnings. A Redis server was beginning to come dangerously close to 100% CPU. Redis servers are used for caching and fast persistence of volatile state from Unreal servers across multiple worlds in the same AWS region. They are super efficient and usually not something that easily buckles under load but you still need to have some educated initial guess about what kind of CPU capacity it will need. We usually are able to form educated guesses by simulating traffic on various endpoints but, in the case of the Redis server, it did not fall under those types of tests but instead relied on limited stress tests we did using internal players as well as special test bots. These tests never came close to simulating the current load that was happening and it was clear that it needed to be switched over to a more performant node. Usually, this is something that you can do live and transparently, but, in this case, the service had not been configured to run with a readily available replica (called Multi-AZ mode).This meant that it needed to be taken down and then restarted. This is where we were approaching unknown territory because it wasn’t clear if all systems would handle that disruption gracefully and reconnect transparently to a new Redis instance without an orderly shutdown and restart. As we didn’t have much of a choice as the server would be failing eventually, we pulled the trigger on updating the Redis instances to more performant nodes. It quickly became evident that proper reconnection did not happen, and we started getting errors and abnormal behaviour from different parts of the stack that were relying on the Redis server.

Parallel to this, we had been keeping an eye on the memory consumption of a few Unreal zone servers that were trending above the norm, resulting in poor performance and lag for players connected to those nodes. Some investigations by actually connecting to these suspicious nodes revealed the culprit. Rabbits. Rabbits everywhere. It’s always rabbits, isn’t it? A resource distribution rule that was being used to control the rabbit population was behaving badly in certain biomes resulting in a proliferation of rabbits. Now even though a rabbit seems to be an innocent creature, it actually counts as a full creature in our NPC count. When a server is at its full player capacity, NPCs need to be kept in control as they compete for the same CPU resources of the server for things like collision detection and pathfinding. Updating the resource distribution to kill off these rabbits was easy enough, but in this case, it would require a reboot of the zones affected. As we didn’t have a clear view of which node might be affected by rabbit infestations and considering that we were already fighting another disturbance due to the Redis restart, as well as needing to apply the same changes to different regions of the world, it was decided to put the whole game into maintenance mode so that we could reboot everything properly.

Maintenance mode essentially cuts access to all players, so it immediately brought the service down, allowing us to reboot the world and finalize some of the changes we wanted to apply. We want to avoid this as much as possible because it is 100% disruptive to our player experience. Still, there is also another challenge associated with it: going out of maintenance mode.

The problem here is that going into maintenance mode with many thousands of players already connected means that when you want to reopen the service shortly after, there is a crowd of thousands of players reconnecting nearly at the same instant. This is very different from the organic growth pattern of connections across a typical day but more like a tsunami of simultaneous connections happening within a few minutes. Most of our stack has built-in protection for that in terms of auto-scaling and load balancing, but there are still some bottlenecks that can quickly become problematic in these scenarios. These bottlenecks can then have a domino effect on other systems that are waiting for connections and might eventually time out, creating a perfect storm. Usually, this all irons itself in the end though the player experience during that phase might be less than optimal. Our goal in the future is to improve our maintenance mode to include some form of throttling of incoming connections, allowing us to have an orderly ramp-up of connections, but for now, it is instructive for us to see which services become bottlenecks and helps us identify where we might want to have different configurations to handle these surges better.

After about 20 minutes of downtime, we were back on the saddle with players ramping steadily as US customers started to join the game in addition to the EU. At around 17:30 UTC we had reached what we anticipated to be our peak connected players for the day.

At this point, we started observing some turbulence in the force and in this case, it appeared to be related to database congestion. It seemed to affect mostly one region but because some of our DB queries straddle multiple DBs in multiple regions, the effect could be felt everywhere. Our DBs seemed to be properly provisioned and shouldn’t be going anywhere close to 100% CPU even under this high load, but something was causing a DB to crawl to a halt. We finally narrowed it down to one rogue DB connection that was hanging and keeping a critical table locked causing any queries using that table to block. This again had a domino effect as the requests to the backend that were initiating those queries eventually timed out, leading to either retries or failures of specific player activities. This was a good example of a single loose screw having radiating effects across the whole stack and regions. Once we released the lock on the table, most services started recovering gracefully, and we quickly came back to prior player levels.

During the rest of the night, we had a few alerts that needed to be dealt with, but fortunately, these were all issues we could fix live without disrupting the whole service and without needing to wake up more people.

The lessons learned are that usually, once you manage to find the root cause of a problem, the cure is relatively obvious and you might also be slightly embarrassed of not having caught that specific problem before going live. The truth, however, is that the type of issues that come up and the combination of multiple different systems exhibiting different behaviour under real player traffic is difficult to predict in totality, and often it can be challenging to discern what the real cause is because of the interdependencies. At the same time, there is a certain thrill to the hunt and a lot of gratification once a resolution is reached.

And this is why we are running a test like Alpha - and are so grateful to the community participating in an event like this, helping discover these hard-to-find issues and create more robust systems to handle them in the future. Once again, we want to extend a sincere thank you to our whole community for being so supportive and passionate about Pax Dei.


Stories From the War-Room

 So we are now emerging from the first 24 hours of our second Alpha test. As was to be expected, many lessons were learned, and pride swallowed, but w...

Game Content