If you want to be totally portable with your system, PA.c is a good place to start, and you can always use their launcher if you want to keep everything in one place. The only difference a PA.c app has from, say the portable version of Kdenlive, which is a zip folder with the program and everything it needs to run inside of it, is integration into the PA.c ecosystem (which amounts to a custom splash screen and some organizational tags for the PA.c menu). It's not really installing in the way we're used to it, because it isn't writing to the registry and making folders elsewhere on your system. In effect, the PA.c installer is just an unzipper that places that folder wherever you indicate.

Instead of existing in folders like AppData or Program Files, and relying on configuration files somewhere else on your system, everything lives in one folder that you can move around without issue. Portable apps run entirely out of a single directory. Truth be told, that's not even particularly bad, but there's no such thing as good clutter, so enter portable apps. This isn't particularly problematic until you go to uninstall an app, and it leaves some data behind. That involves making changes to the registry, modifying context menu entries, or assigning filetypes to themselves.

Regular app installations integrate themselves into your system. Since I didn't see it clearly explained in the comments, here's the deal with portable apps and (PA.c).

Portable app creator download