cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post

Orion multi-tenant deployments

Level 15

This blog complements the information available here, related to what SolarWinds has to offer to the Managed Service Providers (read also the FAQs) and focuses on the multi-tenancy capabilities of SolarWinds NPM network monitor (even if some of this document is actually applicable to other products relying on the SolarWinds Platform, aka Orion Core.

First, what is multi-tenancy?

Wikipedia defines it as a software architecture where a single instance of the software runs on a server, serving multiple client organizations (tenants). Multi-tenancy is contrasted with a multi-instance architecture where separate software instances (or hardware systems) are set up for different client organizations.

With a multi-tenant architecture, a software application is designed to virtually partition its data and configuration, and each client organization works with a customized virtual application instance.

The drivers behind multi-tenancy are mainly (but not only) cost savings (only one platform – HW and SW – is needed to support multiple customers) and easier release management (tasks like upgrade, high availability, database backup are performed only once and benefit all supported customers in a single operation).

Can SolarWinds products be deployed in a multi-tenant way?

Yes, and here is how.

For more clarity, let’s break down this discussion into 2 separate topics:

·         Visualization
How do I configure SolarWinds
products so a particular customer can safely login into the SolarWinds product portal and get a view restricted/limited to THEIR IT resources, even though they are being supported by a multi-tenant platform that also supports other customers like them?

·         Deployment and Data Collection
What should be my deployment strategy, so I can collect management information from my customer’s IT environment and feed it back in to a single shared (i.e multi-tenant) SolarWinds
instance? How do I deal with duplicate IP addresses that my customers have?

This discussion could be extended to other levels, such as storage, and this might be explored in further editions of this blog, but for now, let’s stick to this two main topics: visualization and data collection.

Configuration of the visualization in multi-tenant environments

How to create for my customers, the perception that they are on their own private IT management platform, while they are actually being hosted on a single instance, shared with other customers?

The SolarWinds platform provides two types of “Limitations” that will allow you to isolate the view and the data that each customer has access to:

  • Account limitations.
    This type of limitation is the most commonly used by MSPs and allows for the restriction of the Nodes that a given login will be able to view (or not). Simply put, the Account Limitation controls visibility that Accounts/Login have of Nodes. We will use this notation in the rest of this blog: Account => Nodes
  • View Limitations.

This type of limitation allows the MSP to restrict the Nodes that a given View will be able to display. This will be noted View=>Nodes. Of course, if all your accounts have access to the same Views (e.g. a TOP 10 view), this is not very helpful in the context of isolation, because then all your accounts will have access to all the nodes that this view has visibility on. This is useless for multi-tenancy, unless you can control visibility of Accounts on Views (Accounts => Views); then by combining this with Account Limitations above, you can configure this type of visualization model: Accounts => Views => Nodes.

What is a View?: A view is an HTML page displaying multiple resources (single graph, table, display widget), that have a common theme. A product usually comes with multiple out of the box views. Customers can create new or customize views. Views are selected by clicking on their names right below the product tabs.

Capture.PNG

Now that we understand the theory behind these 2 features, let’s look at concrete examples and how to configure them.

Account limitations: As said above, MSPs typically use this to control the Nodes visible by their Accounts (i.e. login) across all generic views (or regardless of the view).

To configure Account => Nodes, go to Settings>Manage Accounts>Select an Account>Edit>Account Limitations>Add Limitation:

Capture1.PNG

As you can see, there are many different ways to express the list of nodes that will be visible by the account/login that you have selected. Let’s explore a few:

  • Single Network Node does what you would expect (you can select a single node). Easy for testing but fairly limited.
  • Group of Nodes (understand “list” of nodes), is more interesting: it proposes the list of Nodes so you can check one node or multiple nodes that your Account/Login will have visibility on. This is the most commonly utilized.
  • Single Group allows you to define a group and use its content to limit what a user shoud be able to view. This is an easy way for an MSP to define the views of each of their customers.
    Just create a group (Settings>Nodes & Group Management>Manage Groups) and use this group to configure your Single Group limitation.
    Any user associated to this group, will see his/her views restricted to the content of this group.

    Note that this was fixed in NPM v10.4, and used to be more tricky in previous versions , because it does not do what most people expect. Here is the behavior experienced pre-v10.4 abd how to workaround it. This section below is NOT required in NPM v10.4.

You would expect it to give visibility to all Nodes that have previously been put in a group, which would be very easy for MSP’s: put your nodes assigned to each customer in a group representing the customer and you are done, then just assign each Customer groups to the Account/Login allocated to this customer.
Unfortunately it does not work this way. Doing this restricts access to the Group itself but the visibility limitation does not extend to the content of the group. So this does not help you.
Here is how to make it work: make sure you have a Custom property for each Node in NPM network monitoring software, populated with the Customer name (you would probably do this to create the group anyway), but instead of creating your Account Limitation based on the group (which does not work), you create it based on a Custom Account Limitation based on a property. Creating a Custom Account Limitation based on a property is easy: Start Menu>App Programs>SolarWinds Orion>Grouping and Access Control>Custom Property Editor>Orion>Account Limitation Builder> Add>name your custom limitation, select your Custom property (e.g. CustomerTag in example below):
Capture2.PNG
Once done, go back to your Account Limitation screen and notice your Custom Account Limitation method:
Capture3.PNG
This is functionally equivalent to what you think it would do if you created your Account Limitation to a Group that was based on the Custom property “CustomerTag”. Just don’t use the group.

View Limitations: As seen above, MSPs typically use View limitation in this context: Accounts => Views => Nodes, and the most common use case is when Views are actually Home>Summary Views that need to be different per customer, because they have maps that are specific to customers. In other words, for the map to be specific to a customer, you need to create a view that has the correct map and assign it to the Account / Login attached to this customer.

Capture4.PNG

Here is how to assign a Summary View to a account: go to Settings>Manage Accounts>[select an account]>Edit>Account Limitations>Default Menu Bar and Views:

Capture5.PNG

Then, once you have assigned a view to your account, you can now restrict what Nodes will be visible from this view by using a View Limitation: Settings>Manage Views>[your custom view]>Edit>View Limitation> Edit:

Capture6.PNG

From there, the same discussion as above applies.

Let’s take a concrete example and make sure you understand how the system behaves. Let’s say you have used this model Accounts => Views => Nodes to deploy your 2 customers ACME and EuroCust, respectively based in US and Europe, as follows:

  • Account ACME => US_View – use the home page view configuration
  • Account EuroCust => Europe_View – use the home page view configuration
  • US_View=>Chicago_Router, Austin_Router – use the View Limitation feature
  • Europe_View=>Frankfurt_Router, London_Router – use the View Limitation feature

This will provide the expected view, but you will want to know about the following behavior of the system: If the Account ACME user can modify the URL of his US_View page and figure-out what URL corresponds to the Europe_View page, he will not only be able to see the map of Europe (not big deal) but more importantly, see the Frankfurt_Router, London_Router, because these are attached to the Europe_View, as per the View Limitation.

If you are afraid of this behavior (some MSP consider this a back door), then you should NOT use the View Limitation and rather model your system like this:

  • Account ACME => US_View – use the home page view configuration
  • Account EuroCust => Europe_View – use the home page view configuration
  • Account ACME => Chicago_Router, Austin_Router – use the Account Limitation feature
  • Account EuroCust => Frankfurt_Router, London_Router - use the Account Limitation feature

The conclusion of this part is that you will want to think about how you will model your environment and customers, do some preliminary testing of SolarWinds products, based on the ones that you have deployed. Remember that different products may honor the Account and View limitations differently.

Ultimately, you will want to make your own opinion about the pros and cons that the various options offer. Hopefully the section above will help you understand the most important points, so you can figure out the details related to your particular environment.

Deployment and Data Collection in multi-tenant environments

The main issue at this level is dealing with duplicated IP addresses, which the MSP customers will usually have. Today, there are two recommended ways to deal with this: NAT and EOC-based deployments

NAT-based deployment: Network Address Translators translate the customer domain addresses, so that they are all unique from a SolarWinds product perspective. Basically, NAT eliminates the overlapping IP addresses.

Capture7.PNG
Note that this makes the identification of managed devices more complex because the translated IP’s don’t necessarily make sense to report readers. This can be addressed by populating custom properties with IP’s or Names that will not be affected by any translation.


EOC-based deployment: a full instance of SolarWinds product is deployed per Customer (usually large ones) and they are consolidated at the MSP level, by EOC (Enterprise Operations Console)

Capture8.PNG

Other deployment considerations

SolarWinds does not recommend the deployment of a central (NOC) instance of our network management product with one Polling Engine deployed on each customer’s network, because the WAN that would then connect the remote Polling Engines and the central database, would likely be the cause of lower reliability and higher latency which is not a certified environment.

Other solutions such as managing an entire customer environment via a unique IP address and differentiate their different managed devices (e.g. servers) using different SNMP ports (port forwarding) does not work.

You can find more on Orion architecture (non MSP-specific) here.

39 Comments
Level 21

We use Orion (NPM & SAM) in a multi-tenant environment and for the most part it works very well.

There is one significant security concern to be aware of when using custom maps for customers where the background image is a network topology or has any other private information, details regarding that can be found on an earlier post HERE.

Level 15

Hi Byron,

You are correct. The limitation you highlight is the one I was refering to in this section

If you are afraid of this behavior (some MSP consider this a back door), then you should NOT use the View Limitation and rather model your system like this [...]

In your case, did you use View limitation? Because if you do, the customer who changed is URL can see an other Customer's page PLUS the Nodes that are "attached" to teh view by the View Limitation.

This is why I recommend to use theAccount Limitation (and not teh View limitation), because in this case, the customer who changed is URL can see an other Customer's page but node the Nodes, which is the most confidential info.

Does this make sense?

Level 21

We use the Account Limitation model which does properly restrict viewing of the nodes on the map; however, you can still see the background image which is often a network topology or something of that nature.

Currently we assign each customer a CustomerID number which functions as their login name to Orion.  We also have a Custom Node Property called CustomerID where we associate every node with a specific customer.  We then assign the CustomerID as an account limitation so that the customer can only see the nodes associated with their CustomerID.

How would it work if you used both an Account and View Limitation?

Level 15

In this case, yes you are absolutely correct, the background images of the other customer become visible, but not the nodes.

The way you use CustomerID is exactly the CustomerTag property in my example.

On your last question, I don;t know never tried, it's a precedence question. Between the Account Limitation saying you can't see some nodes and getting access to a View attached to some Nodes and therefore making them visible, which one "wins"?

I would need to try.

Level 19

When a user with an account limitation loads a view with a view limitation, the set of visible nodes is the intersection of the set allowed by the account limitation(s) and the set allowed by the view limitation. The limitations all are applied simultaneously - there is no notion of one taking precedence over another.

Level 21

So it sounds like the security concern I expressed is still a valid issue with no current solution, is that correct?

Level 15

So what you are saying Tim, is this:

- My Account is limited (Account Limitation) to see nodes A, B, C

- I tweak my URL and gain access to a View limited (View Limitation) to nodes D, E, F

- The result is that I will see only the nodes that are in both, i.e. nothing in this example, because there is no Node that is in both lists

Level 15

Personnaly, I consider it a security concern, but if you are aware of it, and are making sure that you are not putting any customer confidential info in your maps (e.g. stick to cities, coutries, avoid buildings, don't put IP addresses hardcoded in the maps, that type of things), I think this is an acceptable concern, for which, you are right, there is no solution.

Level 21

We are aware of it and have therefore chosen to not do maps for customers until such a time where this is resolved (as I have been told it's being worked on). With that aside, we have had significant success with Orion in a multi-tenant environment and our customers are always very impressed with our monitoring capabilities.

Level 10

This is correct.  You get the minimum of all possible nodes (intersection).  And in your example, the intersection is null, so no nodes can be seen.

Level 21

It also seems that limitations are all based on a Node level.  What I mean is that you can't give somebody access to see an object associated with a node such as a volume or interface if they don't also have the ability to also see that entire node.

Level 12

Fcaron,

We aren't an MSP but have a need for our users to be able to update their nodes and maps only.  Is this functionality scheduled for a future release?  Do you currently have a Feature request in the system for this capability?

Level 11

This would be wonderful - limited administration.  We could really put this to good use as well.

Level 15

You should be able to leverage what we have now, even if you are not an MSP.

We tagged this blog MSP because this type requirement usually comes from this segment of the market, but nothing prevents you from using these features to "isolate" your users (e.g. per dept or even individual users) and limit their visibility, pretty much like MSP's isolate their customers.

Level 12

Please advise how this is possible with the existing system. I need the capability to restrict admins to only be able to manage their assigned nodes and maps. Currently, there is no way to prevent an Administrator from changing anything on the system.

Level 7

What about the approach described at the following link on how to address duplicate IP addressing using source policy based routing: http://seann.herdejurgen.com/resume/samag.com/html/v11/i12/a6.htm

Level 15

Could you use users that are not Admins, and assign them just what they need to see, in terms of viewsa dn nodes?

Level 15

As they mention, once the packets coming from different overlapping customers, reach the Orion Polling Engine, then the rest is fine, because nodes are not keyed based on their IP. So the whole issue is how to route SNMP packets between Polling Engine installed on the same site (e.g. NOC) to customers that have Overlapping IP's?

This article does a good job recap'ing the various solutions.

I think the best diagram, is their Figure 4 where they show on the right, NMS Tier 1, which would be Poling Engines, installed on the same host (VM's) and assigned to a virtual interface, so the can reach each overlapping customers, via an access router that performs source-based routing.

I hear more about the NAT solution than this one, but since this happens outside of Orion, there might be alot of implementations that we are not hearing about.

I'd love to hear more about them here.

Thanks for this great link Mark!

Level 12

Unfortunately not, because they would not be able to update/create maps, views, etc.  What we are trying to do is to decentralize administration and give them the ability to maintain their own view, etc utilizing standardized guidance for naming, etc.

Level 21

I completely understand what you are looking for as it's something that we would really like as well.  Having the ability to create organizations (logical grouping of nodes and users) and assign an administrator within that organization allowing them the full node/application management functions for all of the objects in their organization.

Level 15

OK, got it, you need a user with same privileges as an Administrator, but limited in scope to his/her "domain" (e.g. group of nodes, views, group of users).

We have opened a request against this (156610)

Level 15

Specifically for you, byrona, as an MSP, would you use this "Administrator limited to a group" to delegate full admin right to each of your customers? I suppose the answer is yes, but would like to be sure.

Level 21

Currently we restrict users to see only their nodes based on CustomerID (a custom node property) though I suppose this could be done though Groups.  In a perfect world I would like to control how much administrative access they have including the option to allow them to create and manage users within their group/organization.

What I have found is the more access I give somebody the more that they want.  Currently giving customers view only access is working fine the way we have it setup; however, I was trying to add more scope to the feature request based on how I have seen it done in other products.

Level 15

Very clear, thanks.

BTW, limiting access per group does not work (see this section in my post above: "Single Group: this one is tricky, because it does not do what most people expect..."

The way to make it work is use a custom property (as described in the blog too), which I think is what you do. So you are good there.

Level 9

As far as I can see, this does NOT address the issue that drives me crazy: there is no way to have different Interface Details views for different audiences.

We have two separate networks, and a shared NPM/NCM/SAM/etc implementation. If I modify the Interface Details view for Type=Ethernet to reflect the features/needs of my team (using Adva Optical network devices) this also modifies the view seen by the other network team for their Cisco network devices. Please explain how to fix this?!? [In practice, I need to be able to define "Interface Details - by Vendor - by Type", not just "Interface Details - by Type" as is presently supported.

To my mind this is a fatal flaw with the whole MSP multi-tenant approach detailed here --- it assumes that all tenants are happy with the same Interface Details (and really Node Details) settings.

Excellent post!

Im interested to find out how you would approch multi-tennant where the alerting engine in SAM is concerned. I would like to be able to create a single alert (node down for example) and be able to target specific distribution lists depending on what nodes are effected. The only way possible I see would be to create a custom advanced alert per client which seems very cumbersom and a pain to maintain.

Any suggestions or advice?

Thanks, Dale

Level 21

My suggestion would be to add a custom node property called something like "Cust Email" and put in the email address for the customer associated with that node.  Then in your advanced alert email address field add the variable for the custom node property that you created.  By doing this it will automatically send the alert to what ever email address you have in the custom property associated with the node.  At this point you can tell your customers that it's their job to maintain an email list on their end (presumably associated with the email address you add to the nodes).  Hope this helps!

Thanks for the response.

Sorry one last question. While adding a custom property to the node such as 'CustomerEmail' works well. How would this work for application monitors on nodes. I have attempted calling the $(CustomerEmail) variable on application alerts however this doesn't work. Any further suggestions?

Level 9

Walker.daleandrew,   I do something similar for contact emails as a custom property.  Initial guess is that you should call ${Node.CustomerEmail} though.

Here is one of my alerts that use a similar custom property:    Good luck

jake

Level 7

Hi! Is it possible to devide one device to 3-4 virtual devices witj defenite interfaces, allowing to see only tenant's interfaces?

Level 8

Hi guys.

I intend to start to use Solarwinds in my envinronment as MSP in replace an another tool where the multi-tenant concept is related to have a server on each customer/tenant. So, all data came from that server is tagged as the server name, for example. So when I query the data at the database I can do this using this tag as a key.

I used this tag for view and report.

Can I assume this same concept for Solarwinds, I mean, for each customer I have, I need an APE at the customer premisses ?

Thank you.

Clecimar

hey clecimar

It depends on how geographically diverse your customers are, and how you plan to get the metrics polled into the central Orion database.

If you are a traditional co-lo MSP, you wouldn't need one APE per customer environment, you would use Custom Properties to achieve the segregation (simlar to the tag concept you mention).

If your customers each have their own private Datacenter, then you'd have far more reliability going down the APE/client method. Fair warning though, this can get expensive as each additional polling engine has a cost. You'd need to build that in to your contract so you're not out of pocket (it's circa £10K per polling engine license, with associated maintenance renewal YoY).

You'd need to have site to site VPNs between each customer site and your management network, to facilitate communications between the APEs are Orion database, and the circuits must have sufficient bandwidth to cope, which varies depending on modules used.

Feel free to PM me if you have any other questions. I used to run a NOC for a MSP, using Orion as the monitoring tool, so I have experience in exactly what you're looking to achieve

Level 8

Hi Silverback.

Yes, they are geographically disperse around the country.

So, it seems the multi-tenant feature is regardless of have or not a server at the customer premises. Is possible do this using what was said at the beginning of this discussion. Right ?

So, the APE need a specific license ? I thought buy the number of licenses to the number of itens monitored would be enough.

Related to the VPN I´m already aware about this. Wonder if there a "magic spreadsheet" where I can know how much bandwidth is necessary ?

Thank you for your availability.

Clecimar

Level 10

Depending on the whole of your monitoring plans, NOM, NAM, AppCentric at least help mitigate the cost of APEs (and HA if desired) as they are effectively free, but the whole licensing model changes to Node Based for each of the modules.

Level 8

Hi Jay.

I´m new on Orion Platform, so I don´t know yet what does mean NOM, NAM, AppCentric.

At the first moment I intend to monitor network devices only, so I believe I´m going to use NPM. So, there are different licensing models ?

Thank you.

Clecimar

MVP
MVP

Network Automation Manager (NAM) includes NPM, NTA, NCM, UDT, VNQM, and IPAM as well as unlimited additional polling engine and high availability licenses.

When you hit a certain number of devices and multiple modules, there is a point where it is more cost effective to purchase a NAM license (based on number of nodes monitored) instead of multiple modules and separate APE/HA licenses. From memory though, this doesn't happen until you need about 6+ polling engines but there are a number factors that can affect that point.

Level 12

Just because you can do something doesn't mean you should

...

There are a lot of cons to this...instead of only one customer going down when SW takes a dive, you bring down 2+...this is an administrative nightmare that can go wrong in a lot of ways again affecting multiple clients...don't even get started on the multiple levels of Security and compliance being a huge issue...NDAs that need to be signed with all of the "tenants"

The alternative to single instance multi-tenancy is using multiple instances, but managing them via Enterprise Operations Console.

This would give you similar benefits to multi-tenancy, without the potential of going blind if your main instance goes down.

You can mitigate this with proper use of Orion High Availability (license extra), but it's another option SolarWinds has for you, within their own stable.

About the Author
Francois has joined the SW product management team in Dec 2010. He has been in the network management space for about 15 years, first in a startup company, then in one of the big 4 and back to a human-size company. Despite his bizarre accent, he is a decent guy to talk to.