Custom reports are reports you build using the Report Manager. You can copy a canned report and customize the copy, or you can create new reports from scratch. There is a query builder that you can use to select the data you want in your report (very much like the alert trigger builder in Alert Manager). You can also write your own SQL or SWQL queries to select data for your report.
SNMP monitoring requires the SNMP service or daemon on the node that you're monitoring. I like it because SNMP tends to be very stable. Solarwinds can monitor all of the basics using SNMP: CPU utilization, memory utilization, disk status/utilization, etc. It's possible to monitor specific processes and services on the node with SNMP. If the node exposes a property via SNMP, Solarwinds can monitor it. SNMP does not let you run scripts on the monitored node, though. If you're monitoring with SNMP, you will need to use another method (like SSH) to run scripts on the node.
Agent monitoring provides the same features as SNMP monitoring. Because the Agent runs on the monitored node, you can use the agent to run scripts on the node. This is useful if you need to get performance data that is not visible from outside the node. It can also make it easier to manage the monitored node (reboot it, restart services). I have found Agents to be a little less stable than SNMP. Sometimes Agents don't upgrade themselves properly, and an admin must log on and manually uninstall the old agent and install the new agent.
Custom properties are additional properties that you create for Nodes, Applications, and so on. They're useful if you need to keep track of data that Solarwinds doesn't collect by default. For instance, I added a custom property called "InServiceDate" to my Nodes. I use "InServiceDate" to store the date we started using the Node. You can use Custom Properties in reports and alerts.