cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post

Everything You Ever Wanted to Know About Regular Expressions in NCM But Were Afraid to Ask

Level 18

Regular expressions are used in many places in the Network Configuration Manager (NCM). Because of technical and historical reasons, NCM uses a few different flavors of regular expressions for different functionality. In this post, I would like to explain the differences and provide references to more detailed documentation.

What Are Regular Expressions

Informally, a regular expression is a search pattern (sequence of characters) used mainly for string pattern matching and string finding. The term originated in formal language theory where regular expressions are exactly defined and related to other concepts of the same power (regular languages, finite automata, etc.). Although the current tools still use the term 'regular expression' (regex), it would be more precise to call them something like 'search patterns'; the expressive power of the search engine is, in fact, often much higher than that of the original regular expressions. (Many modern tools support patterns that are even more powerful than context-free languages.).

Regular Expressions for Comparison Criteria

When you compare two config files for changes, you may want to ignore some of them. (Examples: certificates, last modification time.) Defining comparison criteria enables you to filter out of comparison results lines that you do not need NCM to evaluate; this saves both processing time and, more importantly, makes the review of compared files easier.

Comparison-Criteria.PNG

The regular expressions you create and enable in the settings (Settings -> NCM Settings -> Comparison Criteria) are used throughout NCM — for example, in performing scheduled jobs — wherever the software needs to compare config files as part of its work. A few criteria that are often used come with NCM out of the box; you can also define your own. The lines you typically want to ignore are often informational comments like when was the configuration changed last.

NCM uses the well-known diff tool (from the GNU Diffutils package) for config file comparisons. As you can find in Diffutils documentation, diff uses grep-style regular expressions; then it's not difficult to learn from GNU grep manual that the default regex flavour for grep are basic regular expressions (BRE). Please note that the diff program works line by line -- you cannot define a regex that would match multiple lines.

Let's review some implications of BRE being the regex kind used in NCM comparison criteria.

  • Meta-characters ‘?’, ‘+’, ‘{’, ‘|’, ‘(’, and ‘)’ lose their special meaning; you must use the backslashed versions ‘\?’, ‘\+’, ‘\{’, ‘\|’, ‘\(’, and ‘\)’. In other words, you will use backslashes to give certain characters special meaning, while many other regex implementations use backslashes to take the special meaning away.
  • You have to use '%%' in the pattern when you want to match '%'.

More details on BRE syntax can be found e.g. on this page.

Device Templates

The parameter RegEx may be added to certain commands to recognize the string that is received when the command is complete. For example, if the command is complete when the device responds with System Characteristic, then you must add the following attribute to the command: RegEx="System Characteristic".

Another example: If you login on a device and must switch user context to execute a command, resulting in a different command prompt, use the following example to guide you when switching context and recognizing the new command prompt:

<Command Name=”Reset” Value=”appropriateSwitchContextCommands” RegEx=”newPrompt”/>

(You should also specify that the template logic should run in CLI mode: <Command Name=”MenuBased” Value=”false”/>)

NCM uses the Microsoft VBScript Regular Expression 5.5 engine to parse the value of the RegEx parameter. This engine implements Perl-style regular expressions with some limitations that you will probably never notice. Please see the details on this page.

Using Regular Expressions for Compliance Checks

Policy reports help ensure device configurations conform to both internal business practices and federal regulations, such as Sarbanes-Oxley Act (SOX), Health Insurance Portability and Accountability (HIPAA), and Computer Inventory of Survey Plans (CISP). Policy reports scan configuration files and report any discovered rule violations. For example, a rule requires configurations not to include the read-only community string public. You can run a report on your configuration files, and then display any configurations that violate the rule.

If you want to search the device config for a text block, or by a regular expression, you should use the Advanced Config Search. Compliance checks use the Microsoft .NET Framework regex engine. The syntax is described in Appendix B of the NCM Administrator Guide (http://www.solarwinds.com/documentation/orionNCM/docs/orionNCMAdministratorGuide.pdf) and also in Regular Expression Language - Quick Reference and Using Regular Expressions with .NET - C# and Visual Basic.

A Basic Example

By Tim Nelson.

When you create an advanced rule with multiple patterns, it simply takes the true/false of the patterns and then performs an AND operator on them to see if any are violated based on the overall criteria, not the individual rule criteria. Second, you can use parentheses to make rules inclusive of each other. This is needed when you want conditional statements with dependencies. Think of it as a creative way around if-then statements. So to explain this in better detail let’s cover what will and will not match in a policy.

If you create an advanced rule and you want to do a conditional based on bandwidth, you can say the following (keep in mind you should test ALL rules with the single regex you want to match before creating a complex multi-line rule, or use a regex tester):

       Must contain “string”: bandwidth 1536

and    Must not contain “regex”: class ipp1[^\r\n]

or     Must not containt “string”: bandwidth 1536

In order for this to actually work the top two rules must be contained within parentheses to make them inclusive of each other, i.e. one requires the other. What we are actually doing with the middle pattern is matching anything BUT “class ipp1\r” or “class ipp1\n” which is why we used, Must not contain. Lastly, all of this is wrapped within a container of “Must not contain” for rule violation, this is what actually decides if the rule is violated or not.

Policy-Create-Rule.PNG

What you will notice when this violates is that the “bandwidth 1536” is matched within the config, because you put “must contain” and it matched, NCM will add a plus sign where ever there is a pattern match. You must keep this in mind as you write rules, as without being the creator of the rule, this can become difficult to parse through.

NCM will show the first line on a multi-line match as the violation, so for end-user readability we can create rules that identify the exact line in error and therefore have more granular reporting. You should keep in mind that if your standards change in your network environment, you will have to be able to update these rules. So it is best to fully document your logic for every advanced rule you create.

25 Comments
Level 13

can you please make a FAQ life document the list and details dozen of examples

this with descriptions of how best to use them

Level 18

Hi,

I can't promise anything, but I hope to find some time to start with it.

Regards,

Jiri

Level 11

The screenshot is wrong btw, it has must contain with a negation, which is not what I typed above

Level 18

Thanks, will correct that.

Level 13

Are there any examples on thwack?

Level 18

Not a dedicated post, but I would say thwack is full of regex examples.

Jiri

I rely on this information almost daily--in the background.  Each time something new is required, it may mean digging into the RegEx reference materials--some of which can be found here:

RegExr: Learn, Build, & Test RegEx

http://www.zytrax.com/tech/web/regex.htm

Swift Packets!

Rick S.

Level 8

Hi,

I've been searching all around to find a regex example which will ignore 'blank' interface configurations.. Does any body know what regex I should be using?

My rule looks like this:

Capture.JPG

The report works fine, apart from reporting interfaces which have not been configured as missing the strings above... because they look like this:

interface GigabitEthernet0/1

!

Thanks for your help, happy to answer any questions.

From a SQL standpoint, have you tried:

^$

?

Some other options:

^\s*$

Also, check here:  RegExr: Learn, Build, & Test RegEx

\s* will match zero or any number of White space

Level 8

Hi, Thanks for this..

I had seen something similar, but couldn't get it working. I think I just needed to try a little harder, because its working now

Thanks again!

Level 10

Has anyone had any luck with trying to stop this comparison?  This is the crypto portion in a cisco config....  Every time I run a compare between the startup and the running I see this....  I have loaded my entire config file into the RegEx test site mentioned above and I can get the RegEx expression to work there but it never works when run in NCM....  my latest changes were to the Quit line RegEx which are [ \t\r\n\v\f]*quit[ \t\r\n\v\f]*  But this doesn't work either.

Any suggestions would be greatly appreciated.

comparison.jpg

Check your configuration comparison criteria in NCM's Settings and look for these lines:

pastedImage_0.png

The top line likely is the one that can help you out, but the other lines have also helped clean up my configs.

Level 11

These use BRE's basic regular expressions.  You will find that if you try to match the certificate data you will end up breaking other things with the configuration comparison.

My recommendation is to leave it as default and not attempt to expand on them.  Ignoring the hex MAY work on certain devices but in fact break other configuration comparisons depending on what's in your environment.   So it's very specific as to what ALL of your configurations look like and not just routers with certificates.

- Tim

Level 12

Hi george.subnet​, how did you get NCM to ignore "blank interfaces" or even interfaces which have nothing configured on them? TIA

Level 8

Hi, I'm not sure if it will fulfil your requirements, but I added what @rschreder posted and used ^$

You can see the extra line I added at the bottom of the image:

string.JPG

I hope this helps

Level 12

OK excellent, I'll have a go with that. Thanks very much!

Level 12

So NCM's policy reporting tool uses .Net's flavour of Regex? Can it do multi line matching?cvachovecj

Level 12

Hi rschroeder​, do you find that the regex tool you provided a link to 'RegExr' is a decent replication of the regex engine used by NCM specifically the policy reporting tool? What I am trying to say is if I can match a string with that tool should the same regex work to find a string using the NCM policy reporting tool's "create rule"

I'd LOVE to say yes or no to your question, but my expertise in the RegEx world is incomplete, and I apologize for not being able to provide a definitive answer for you, noobes​.

The info at the link is useful for understanding RegEx, but I must refer you to a Solarwinds expert if you need information indicating that Solarwinds followed ALL of the RegEx examples and standards in their implementation.

Sorry.

Swift Packets!

Rick schroeder

Level 12

Cheers rschroeder​, thanks for the reply - the thing is Solarwinds don't offer support for regex, it's not that I need help creating the regex I would just love to have a testing ground that Solarwinds could stand over and say if it works on tool X it will work in Solarwinds. A tool/interface like Regexr or regex 101 would be so useful when building policy report rules. Thanks again!!

That scenario WOULD be nice!

In my travels, I have seen many people use regex successful with the NCM Compliance.  For those programmers that love regex, it is easy for them.  For those of us that like it and trying to learn, it takes a bit more time.  I major issue I see out there, is the willingness to share their regex code with Thwack from non-Thwack users(Yes! there are still those people out there).

I have put some regex into my DISA STIGs, not prefer to do actually CLI commands because it is easier to explain to those that are new to NCM Compliance or do not have a programming background.

For those attending my class (through Loop 1), we will cover some regex.

Thanks,

Eric

Level 12

Cheers. I'll ask support if there is any tool already available online that might be a decent replication of how the .Net regex engine used by NCM works.

Level 12

I use regex a lot in my compliance reports for example if I want to make sure that default SNMP string is not being used I set up a rule to alert if the regex community(|-name) (Public|public|Private|private) is found the major thing I am struggling with is trying to get NCM to match strings over multi line. I can do it using the tools rschroeder has mentioned above but when I use the same regex in NCM I am getting unexpected results. Anyone know what regex character to use to match over multi lines? I've been using \n or \n\r to no effect