I am upgrading from 2023.2.1 -> 2024.2.1
Can you upgrade without a reboot? Yes. But ideally you run it with no pending restarts, and during the upgrade the application will be offline.
Agents require a restart when SolarWinds has to upgrade/install the version of .NET. I'm not aware of any changes in .NET between the versions that you mentioned. Outside of that, agents typically only require a service restart (SW Agent service) for upgrades