cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post

Normalcy is boring, or is it?

Product Manager
Product Manager

itsnormmal.jpg

Normalcy is boring, or is it?

          Something that I have been working on is helping to come up with a baseline security plan for an IT team and their infrastructure.  What I have ran into is that having a basic template and starting point really helps.  Fantastic right?  Well, when I start off by giving them credit for monitoring they look peculiar at me as in why would monitoring be a starting point?  To be fair and accurate a few high five me as they are like SAWEETNESS (meant to be spelled wrong as that literally is how I speak, ok back to the blog ) check that off the list of things to come!  Today, I'm going to go over this one portion of the plan and show why "knowing normal" is actually a starting point for a great security best practices and policies.

     First things first,my favorite quote "If you don't know what's normal how the heck do you know when something's wrong?".  Baseline and accurate monitoring history will show you whats normal.  This also will show you how your infrastructure handles new applications and loads when you are monitoring so its not just for up down that is just a side perk honestly.

Ok, now once you know what normal is the following will help you to see issues easier and be aware.  So remember the below is once you have monitored and understand your normalcy of your devices your monitoring.

Monitoring security features

  • Node -  up/down
    • This will show you if there is a DoS happening or a configuration error with no ability to ping a device. 
    • Will show you areas within your monitoring that are being possibly attacked.
    • Allows you to have a clear audit of the event that are taking place so you can use for management and your team for assessments.
  • Node - CPU/Memory/Volume
    • CPU will show you if there is an increase spike as that will help to show where to look for what increased or caused this spike that never went away.
    • Memory allows you to know if there is a spike obviously something is holding it hostage and you need to address this and prevent or resolve. 
    • Volume if you see a drive increase its capacity OR decrease quickly and are alerted to this you may be able to stop things like ransom ware quickly.  The trick is to be monitoring AND have alerts setup to make you aware of drastic changes.
  • Interface - utilization
    • Utilization will show you if a sudden increase of data is transferring into or out of an interface.
  • Log File monitoring
    • Know when AD attempts are failing.
      • This is something I see a lot of times and the person monitoring just states "yes, but its just an old app making the request no biggy".  Ok, to me I'm like fix the old application so this is no longer NOISE and when you have these coming in from outside this app you are more inclined to investigate and stop the whole.
    • Encryption know if files are being encrypted on your volumes
    • Directory changes if directory/file changes are happening you need to beware period
  • Configuration monitoring
    • Real-time change notification that compares to the baseline config is vital to make sure no one is changing configurations outside of your team.  Period end OF STORY.  (I preach this a lot I know.  #SorryNotSorry)
  • Port monitoring
    • rogue devices plugging into your network needs to be known when and who immediately

          This is obviously not all the reasons you can use against normalcy but its once again a start.  Understanding normal is vital to set up accurate alerts, reports, and monitoring features.  As you hone in your skills on assessing what you are monitoring and alerting you'll see things drop off while others will increase within your environment.

          Don't be shy to ask questions like, why is this important?  I seen this article on an attack, how can we be alerted in the future if this happens to us?  Some of the best monitoring I've seen is due to looking through THWACK and reading articles on what's going on in mainstream.  Bring this knowledge to your monitoring environment and begin crafting an awesome arsenal against, well, the WORLD.

HTH

~Dez~

  

22 Comments
Level 21

After working nearly all night last night (unexpectedly) due to a back-end storage system filing up and taking down an entire virtual environment (including Orion) I would love to have some normal!

I point this out because if somebody would have setup the monitoring properly we would have seen the storage filling up, the issue would have been resolved and I would have been able to sleep last night.  In other words (and in support of your great post here), better monitoring configuration would have kept my life more normal.

Level 13

It still surprises me how many senior IT professionals don't bother with baselines...how do you know if something is acting weird if you don't have a baseline?

MVP
MVP

Dez​ nailed it !

without a baseline or a "present normal" you have no idea of when or where you "might" have a problem at least based on what you know.

Level 13

Exactly...

Level 20

I like the new automatic baseline features in Orion.

I agree, that was a great addition

Product Manager
Product Manager

Ugh!  THIS right here is why I love helping people do "audits" on their monitoring systems.  Regardless of vendor, the need is there and we should re-evaluate often our setups.

~Dez~

Product Manager
Product Manager

Preach it!  No really preach it as my voice is horse... 

~Dez~

Product Manager
Product Manager

So much yes here!

MVP
MVP

In the end I cannot say Normalcy is boring.  It is a constant challenge to try and keep things normal so how can that be boring ?

Being complacent is boring....the end result is chaos and not on your terms.

Level 12

Normal is two things.

Normal is sleeping overnight undisturbed because all is well.

Normal is wondering what you've missed in your monitoring because we all know there is never a time when something isn't not normal.

Baselining is an ongoing process as well, not just considering time of day, day of week, day of month, and day of year, but also OS, hardware, and software upgrades (which one hopes improve the baseline.)

It's why they hired us, really, in the end. "Make certain the users never know we almost had a problem before the problem occurred."

Monitoring without baselining might be like stating humans do or do not have impact global warming.  How can you make an accurate assessment without baselines and trend analyses?

xkcd: Earth Temperature Timeline

Product Manager
Product Manager

Yes!! 

I haven't read all the reply's so if this is redundant please forgive me.   I was told once that Sysadmin's don't make good DBA's.   I took offense but they might be right.   As an admin we just to fix everything right now, but as a DBA, you have to be able to be patient, make a change, watch for improvements and or degradation.  Then you have to decide how to move forward.  So you ask how does this apply to your topic?  Depending on your approach to monitoring it could be the same way, not every admin is right to monitor everything.  I think you have to have a good understanding of the functional area, then make policy to assist in resolution of alerting.  Normal is never boring, if you are doing your job, often there is not enough hours in the day, even when everything is working like its supposed to. 

Level 13

I don't know...I find DBA's don't have the same troubleshooting skills as a sysadmin...

MVP
MVP

Sleeping all night undisturbed is something I have not had in a long time...

The joys of being a firefighter on the side.  There it is more normal to be disturbed while sleeping.

A common problem with silo'ed IT organizations: DBA's, Network, Systems, Storage, Applications... (especially without a centralized NOC) is that they baseline differently and rarely alert others when there is a problem. So core elements like baselining, UX, KPI's, are not shared. This fosters an unhealthy "Not It!" environment.

Product Manager
Product Manager

However, they do when it comes to their databases and out of character issues that might be happening.  This will assist the sysadmin and network admin to better protect and have normal behavior matching all around.

Everyone has skillz knowing how to leverage them is key.

~Dez~

Level 13

I so agree.  You need a baseline to figure out is something is wrong.  Great comments.

Product Manager
Product Manager

This is another reason that I like to bring up security talks with all teams.  it seems to take the silo part out as everyone feels like they are contributing to the greater good.  Other outside areas feel like they are able to use their expertise with others and vice versa.  I remember a few years ago security was a cuss word.  Now if you're not using that word you're literally hindering yourself in your daily life.

~Dez~

Level 14

Love this.  You can't identify abnormal, if you don't know what normal looks like.  Also, get rid of the weeds.  If the logs are showing a misconfigured server doing anything incorrectly, fix the problem!  Get rid of the noise so you can see the grass through the weeds.

MVP
MVP

The fun part about IT is that the only true normal is change.

About the Author
I started in networking and security around 2002 by taking Cisco Certified Network Associate and Security+ courses from Central Vo-tech. This is where I fell in love with technology in general. From there I venture out to internships and started using the Engineers Toolset from SolarWinds which made me wonder about software. The company I was with purchased Cirrus which is now Network Configuration Manager (NCM) and I was officially hooked. I searched out for SolarWinds and well you guessed it I started working for them and believe it or not in sales. That was the only position open but I knew I wanted to be here. So I quickly worked my way in to the support side and became the first Sales Engineer and then the first Applications Engineer. Since I am a very curious person I have since in my 9 years of being at SolarWinds decided to pursue more education. Security is always a fascination to me so I started taking classes on INFOSEC Assessment Methodology (IAM) and INFOSEC Evaluation Methodology (IEM) of the NSA. Then I went and took the CIW Masters for web development and ventured to databases. MCITP SQL Server and Development certifications that led me to a database development degree in college. I’m pretty much a jack of all trades and LOVE IT! This all applied to my work with SolarWinds as I wanted to be able to help customers solve their issues or needs. So knowing more information allowed me to do this successfully. I also dabbled in Cisco UCS management and currently taking classes to venture toward a CCIE (crossing fingers). NCM is a product that I have worked with since its beginning. I even had the opportunity to fly to the NSA to create templates for some of their devices. I used to be the sole MIB database controller so I’m definitely your huckleberry on MIBs and OIDs. As an Applications Engineer I focused on Network Performance Monitor, Network Configuration Manager, Web Performance Monitor, Enterprise Operations Console, Patch Manager, User Device Tracker, and the Engineers Toolset. See why I like to constantly learn new things I had a lot to be on top of! SolarWinds is a passion of mine still to this very day. My new role as a Product Manager for NCM is home to me. Funny how I circled around back to my favorite product that got me here in the first place. :) My goal is to educate and work with customers to leverage our products to their fullest degree!