So this is exciting news!

I'm back to being able to full time work on my projects again.

As some of you may know I moved to Europe in March. This was an incredible amount of work that I really don't ever want to do again lol. It involved dissolving my life in the US (which doesn't sound like a lot until you actually have to do it).

It took months to get rid of all the stuff from my old life, pack, and get the legal paperwork done to export my parrot to Europe.
That last part is a massive undertaking and I pity any human or bird that needs to do it. Since COVID the laws for transporting birds got a lot more restrictive. You basically need two major permits between the US and Europe which can take months. Then you need a ton of other permits, quarantine your bird for 30 days, and then you have to find an airline that will transport her... There's more rules and laws between countries, and airlines, than you can shake a lawyer at.

It was A LOT. I'm glad to say that she's in Europe now. A happy EU birdy. I love my birdy.

(Random IRL view from my new home now, not in-game lol I wish!)

It also took a lot of work to get settled in. I think that part may take years to end... Either way. It's done!
I expected moving to take maybe 3 months. It's involved almost a full year of complete focus.

All that said. That explains why development was slow.

(Say what you will about the use of WPO, but I love the subsurface lighting on foliage that I made!)

Now that I can work on this project again, a major "to do" involved upgrading the project to the most recent version of Unreal.

I know this is by no means an easy thing to do. The bigger a project gets, the more things break.
I am also not certain if this is a thing that people working in Unreal regularly do with their projects? Is it better to just keep working in the version that the project is built in?
It seemed appropriate to upgrade so I can take advantage of newer features...

Upgrading was by no means an easy task. There were points where I honestly wanted to give up. Toward the end I realized that I couldn't because the original 5.1 version stopped building, and it would probably have taken as much work to figure out what that's about rather than just staying the course with getting settled into 5.4.
For a while it seemed like as soon as I fixed one thing, ten other things broke as a result.
By now I'm settled into 5.4 enough to understand why certain things broke... It took some meddling.

Someone asked for examples as to what things were hard, and I thought it would be a good idea to write this and cover a few of the major hurdles. Maybe someday I'll look back at this and be proud of how far I've come!

(helpful screenshot I saved from the many forum posts)

On the outset, upgrading Unreal is a difficult thing to do because there's this performance drop that happens. It takes a lot of fiddling with the Project Settings to find out what you should enable or disable to fit your project, and keep performance at peak.
A big one of mine was avoiding lumen. I don't understand it or why it's a good idea (yet). Both Lumen and Nanite are the next big thing that helps performance, but out of the box that's difficult to see. You still have to understand what's going on.
I don't fully understand both, but I'm getting there!

Upgrading the project is always a cold shock because you see how much is just going... full crazy in your level. After converting the project from 5.1 to 5.4, I opened up to levels that had broken lighting, weird lighting glitches, shadows that had ghosting problems, the volumetric fog system changed so that took a lot of tweaking, foliage was hovering in all directions... and things generally seemed out of control in a way that I wanted to shout "feature not bug!", accept the new state of things, and call it a day... But I can't. It needs to be perfect otherwise how can I call myself a self-respecting gamedev?

Ok then...

(Screenshot taken from the forum)

The first obvious problem was a simple pleasant one to fix. There were changes to the wind system, and overall foliage that used this needed to be re-examined. I found this post "Nanite Foliage with wind/sway material effects placed via procedural volumes (UE 5.1.0) may seesaw/sway together with all other foliage in the same procedurally generated volume" ... weirdly I didn't have this issue in 5.1, it happened in 5.3 and on.
I fixed this by changing the TransformPosition's Source to "Instance & Particle Space"... That was probably the most pleasant bug to figure out.

There were a ton of issues in most of my skies. Volumetric fog seemed to have changed and it took a lot of tweaking to fix these. I'm making a note to myself to keep my skies simpler, going forward.
Volumetric effects in general had differences in 5.4 and, from what I could gather, my setup in earlier levels was not correct so plenty of effects "broke" or changed. It was quite a journey to look back at earlier levels (when I knew less) and see all the mistakes I made.
I think I came far!

I also had a weird ghosting issue on ONLY the landscape textures that would happen whenever the camera moved. Nothing else had this problem. Only the landscape texture. This also only happened in 5.4 not 5.3...
First I thought it was my virtual texture setup, but that wasn't the case. It turned out to be the combination of WPO and nanite on the landscape... This seems to be an actual bug in the engine. (Sorry for saying this. I know that it's probably a popular conclusion to jump to if you don't know what you're doing... which is my case, but also others are saying this so it's probably fair to say!)
There's a few other's with the same issue. Disabling "Output Velocity due to vertex deformation" in the project settings seems to "fix" this, but I'm not sure why or what that means. For now, it's fixed.
Some links to what I'm talking about...
* https://www.reddit.com/r/unrealengine/comments/1cxox9i/54_landscape_rendering_is...
* https://forums.unrealengine.com/t/landscape-rendering-issue-when-using-world-pos...
* https://issues.unrealengine.com/issue/UE-210893

There was a very obscure issue that kept happening whenever I would build. A Niagara error would occur that made the build process error, but this error was incredibly vague. It took a while to find out that, when the project got ported to a newer version of unreal, the Sky Creator plugin used in some of the levels lost the connection to the "Weather FX" system and this was causing the crash. This was the case for any levels that used this sky system. Just going in the "Weather FX" settings and reconnecting the Niagara system fixed this.
The error was so vague it took a while to figure out that this was the case. It didn't happen in any other contexts but for updating. I'm glad to say that this was an easy fix, once figuring out what actually broke.

There's also some brand new weird things that I don't completely understand yet, but OK... For example the "Resizing Virtual Texture Pools" message that kept popping up (as described here and here). The actual documentation about it, here, is pretty straightforward so I followed the instructions. I hope this is what I was supposed to do. It scares me that I don't really completely understand this, but so far so good...

Some consistent errors with the player falling through the floor in 5.3 would happen, and keeps happening. Searching around plenty of other people had this issue (for example, here), so it was obvious to me that I should use 5.4. 5.4 had a bunch of other issues that took a while to get around (like the WPO issue) but overall I got it to work.

I also had strange ghosting and flickering that happened with some of the lighting. Turning off "dynamic shadows" in some areas seemed to have fixed it, and then I found this explaining that "This happens when I have multiple static or movable rect lights with large or overlapping attenuation radiuses". Disabling the shadow denoiser seems to "fix" this but I didn't do that. I just adjusted the lights in my case.
Little things like this worry me because I don't know what I'm doing, so it's a bit of a guessing game when you adjust the settings. I don't want to disable/enable too much in the settings because I KNOW that eventually that's going to cause problems when the project grows and I forget what I did or didn't do.

Weirdly between 5.1 and 5.4 "one media opened" stopped working for video in my specific case. It still worked in 5.3, although it would fire more than once... in 5.4 it completely stopped. There seemed to be weird quirks in general with video but I didn't really need "on media opened" so I changed how I was implementing that.

("Assertation" is a both vague and strangely specific word choice but it stopped happening once I restarted my computer.)

At this point you can imagine that I think I will have a good cry before ever doing this again. It honestly felt like fixing one thing would break a ton of other things and it took a lot of searching through forums to figure out why. I don't like not understanding why a fix "fixed" something because you know that it will come back to haunt you.
Most of the issues came down to having to refresh, and rebuild everything. Restarting my computer also worked in some cases... weirdly enough.

For example, there was some inexplicable issue that I could not recreate for the life of me so I'll just take the victory that it stopped happening but... When I upgraded directly from 5.1 to 5.4 the WPO on the landscape was flat out broke. No WPO at all. Looking around nobody seemed to have this issue. So I thought that 5.4 was probably not a good version (others said it was buggy, but in general people say that about things)... and I tried settling into 5.3.
There was no issue with landscape and WPO in 5.3... but eventually, in working on fixing everything, I upgraded from 5.3 to 5.4 and the WPO issue was gone. I did nothing to fix it. This should keep me up at night, but I'm assuming that it must have been some cache or saved thing somewhere that got rebuilt.

Other issues were interesting, and I'm happy to say that I think it's gotten me closer to understanding the weird little quirks in Unreal.
For example, my latest one...

If I deleted something from my level it would still show up when built or while play testing. These were things that were also set to "hidden in game" and "is editor only"... They'd just show up after deleting.
Rebuilding my HLOD's fixed this, so it's good to know that you have to do that regularly in open worlds... also cleaning up references helps. Little things like this are new to me because I'm used to just deleting something and it's properly gone.
Here there seems to be all sorts of caching so you have to be careful.
I probably sound ignorant... Yes, I'm still learning... Here's to logging my journey!

There are still plenty of issues I'm facing with my runtime virtual textures on the landscape but I don't really care at this point. There's plenty of documentation out there to help me understand. I'll take the win. I'm glad to say that I can continue building on BlueSuburbia using 5.4 now!

*Knock on wood* :)