So, it turns out, this is a feature in macOS Monterey.

When you install an application from the App Store, if it is labeled or categorized as a game, it will actually install in a specific folder called Games. This also puts the Chess.app in the folder as well but for some reason leaves it in the “Other” folder at the same time.