If you want to use RPC method (Performance Counter Monitor), then you can use perfmon on target machine to make sure that category "MSMQ Queue" exists and gives you information you are interested in (e.g. value of counter "Messages in Queue") for instance with desired queue name.
Example of Performance Counter Monitor configuration:
Category: MSMQ Queue
Counter: Messages in Queue
Instance: Computer Queues
Example of WMI Monitor configuration:
WMI Namespace: root\CIMV2
Query: SELECT MessagesinQueue FROM Win32_PerfFormattedData_msmq_MSMQQueue WHERE Name='Computer Queues'
Both methods should give you the same information. There should be wizards for both these monitor types that can help you to start monitoring using browsing method.
To the heart of your question though, have you tried using the following application templates for monitoring MSMQ? They're included in SAM 5.5 so you should see them listed once you upgrade.