Skip navigation
1 2 3 Previous Next

Product Blog

34 Posts authored by: nicole pauls

(updated on November 12, 2015)

 

As a part of helping untangle compliance initiatives, a popular request on the federal side is FISMA (Federal Information Systems Management Act) Compliance and support for the Risk Management Framework (RMF). In this post, I'll outline what FISMA compliance is, we'll walk through FISMA bit-by-bit, and we'll talk about where SolarWinds products can help.

 

FIS-WHAT? What is FISMA AND RMF? And how does NIST play into it? And FIPS?

 

What it actually means to take on what's commonly referred to as "FISMA Compliance" is described in several NIST (National Institute of Standards and Technology) publications. It's pretty impressive the amount of NIST publications out there, but there's really only a few we're interested in. A couple of these are FIPS (Federal Information Processing Standard) publications - usually when we think of FIPS we think of encryption, but here we're mostly focused on risk analysis.

  1. NIST 800-37: Establishes the Risk Management Framework as the security life cycle approach.

  2. NIST 800-53: This is the main "FISMA Compliance" publication. This describes what controls need to be applied to different systems.
  3. FIPS 199 and
  4. FIPS 200: These two documents describe how to perform risk analysis and categorization for systems on the network. You'll need this categorization when you actually go to implement 800-53.

 

Here's a great summary, though wordy, of how all of that fits together:

FIPS Publication 200, Minimum Security Requirements for Federal Information and Information Systems, is a mandatory federal standard developed by NIST in response to FISMA. To comply with the federal standard, organizations follow the Risk Management Framework to determine the security category of their information system in accordance with FIPS Publication 199, Standards for Security Categorization of Federal Information and Information Systems, derive the information system impact level from the security category in accordance with FIPS 200, and then apply the appropriately tailored set of baseline security controls in NIST Special Publication 800-53, Security and Privacy Controls for Federal Information Systems and Organizations

 

Okay, okay, how about the super simple version? In order to implement FIPS 200 with NIST 800-53, you have to first do the risk categorization in FIPS 199. Whew!

 

Navigating and Implementing NIST 800-53 - High Level

 

We'll leave the whole exercise of assigning risk up to you, since it'll be different for each environment. Once you've done that, as you walk through the 800-53 requirements, you'll see different controls that need to be applied at different levels. Generally, you'll have to comply with the "document" and "policy" controls across all risk levels, but some of the finer controls may not need to be applied to all risk levels.

 

NIST 800-53 and the RMF provide a great breakdown of the steps that need to be applied. Of interest to us when it comes to where SolarWinds products can help are:

  • Step 3: Implement controls
  • Step 4: Assess controls are working correctly
    • Our security product portfolio, including NCM, and Log & Event Manager (LEM), can be used to make sure controls have been implemented correctly.
  • Step 6: Monitor
    • Lastly, several products, including LEM, Network Performance Monitor (NPM), and NCM, can be used to make sure that controls are working as expected, bypasses aren't attempted, and produce reports that can be used to prove it.

 

I'll walk through each control and identify relevant products for each category as I go, so you don't have to memorize them all just yet.

 

Key Out of the Box Content for NCM and LEM

 

Before we dig into implementing key controls (Step 3), as a part of assessing and monitoring controls (Step 4 & Step 6), there is out of the box content included in NCM and LEM that is designed to help:

  1. For LEM:
    1. There are hundreds of out of the box reports, many of which are categorized for FISMA specifically. These reports really help address the Assess/Monitor by helping look for exceptions to controls, unexpected changes or activity, or attempts to bypass controls. In the LEM Reports Console, navigate to Configure > Manage Categories, select FISMA, then click OK. To see the list, go to View > Industry Reports.
      1. LEM-industry-reports.png
    2. In addition, LEM includes dozens of correlation rules categorized for different compliance initiatives that can help - and be quickly enabled. From the LEM Console, navigate to Build > Rules, and either launch the Add Rule Wizard or navigate to the categories on the bottom left. I'd recommend starting with General Best Practice, but as we go through the actual controls you should find relevant correlation rules where real-time notifications are useful.
  2. For NCM:
    1. There are several templates included to help (starting with NCM 7.4 - DISA STIG and NIST FISMA Reports Now Shipping with NCM! - earlier versions can download from the Content Exchange):
      1. NIST - Services: identify services exposed on network devices
      2. NIST - Remote Access: identify remote access enabled on network devices
      3. NIST - Management: identify management protocols used on network devices
      4. NIST - Access Lists: identify key access control lists that should be present
    2. In the NCM web console, under CONFIGS, then Compliance, you should see them listed under the NIST category.
      1. NCM-FISMA.png

 

Control-by-Control Details

 

You might want to get a cup of coffee (or tea) while you read through this, as there's a lot here. The entirety of Appendix F of 800-53 actually describes the controls and implementing them in detail. I'm going to skip over a lot of them since they don't apply to implementing SolarWinds products, but I'll include a description for each and more details where they are especially relevant. Got your warm beverage? Let's get going.

 

  • AC-X: Access Control
    • General Notes: In general, there's a few areas our products can help, but a lot of these controls will be implemented at the policy or device level. For some of these, NCM can help you distribute configuration or identify violations where it comes to network devices; LEM can help audit and monitor for potential changes.
    • Of interest:
      • AC-2: Account Management:
        • You could use LEM to identify accounts that are created outside of these controls - e.g. service accounts being added to unexpected groups - either in real-time or via reports.
        • You could use LEM to audit when passwords were changed on accounts, when users were added to groups, etc - either in real-time or via reports.
        • LEM can help satisfy AU-2(2): Automated Auditing for creation, modification, enabling, disabling, and removal, either in real-time or via reports.
        • LEM can assist with AU2(12): Atypical Usage by looking for logon activity or patterns that are outside your environment norms, either in real-time or via reports.
      • AC-4: Information Flow Enforcement
        • LEM can help with AC-4(17) - ensure local authentication is not used by auditing for local authentication activity on systems (logons not to the domain), either in real-time or via reports.
      • AC-6: Least Privilege
        • LEM can help audit where things deviate from least privilege - e.g. when an unexpected user accesses certain files, systems, or commands, either in real-time or via reports.
        • NCM can help audit device policies for existing privileged users as things change, and roll out configuration changes if necessary.
      • AC-7: Unsuccessful Logon Attempts
        • Usually this is implemented in IAM/Domain/system policy, but you can use LEM to confirm this policy is being enforced and see how frequently it is used, generally via reports/historical analysis.
      • AC-8: System Use Notification
        • Where required across network devices, NCM can help distribute config or identify configs that don't match expected settings.
      • AC-9: Previous Logon (Access) Notification
        • Where required across network devices, NCM can help distribute config or identify configs that don't match expected settings.
      • AC-10: Concurrent Session Control
        • Where required across network devices, NCM can help distribute config or identify configs that don't match expected settings.
      • AC-11: Session Lock
        • Where required across network devices, NCM can help distribute config or identify configs that don't match expected settings.
      • AC-12: Session Termination
        • Where required across network devices, NCM can help distribute config or identify configs that don't match expected settings.
      • AC-16: Security Attributes
        • Depending on how controls are implemented, it's possible that LEM can help identify when things deviate from expected policy, either in real-time or via reports.
      • AC-17: Remote Access
        • LEM can help audit/monitor remote access, but not implement controls. LEM can also help audit where remote access is being used outside of expected controls (e.g. controls are being bypassed, or attempts to bypass are being made). As usual, this can be done either in real-time or via reports.
          • Explicitly, LEM can help with AC-17(1) - automated monitoring / control
        • NCM can help audit where and how remote access is being used across network devices, identify violations, and potentially roll out policy changes if necessary.
      • AC-19: Access Control for Mobile Devices
        • You may be able to use User Device Tracker (UDT) to detect usage of devices that are in those classified networks/facilities, and possibly also use LEM to identify authentication from unexpected users or devices.
      • AC-20: Use of External Information Systems
        • LEM can help audit AC-20(2) and AC-20(3) - use of portable storage devices and personal devices with USB-Defender when policy is bypassed/ignored.
      • AC-23: Data Mining Protection
        • You may be able to use LEM with SQL Auditor or Database Performance Analyzer (DPA) to identify when large queries or unexpected activity is being done to a database.
  • AT-X: Awareness Training
  • AU-X: Audit and Accountability
    • General Notes: A lot of this set of controls is about what data you might feed into a system like LEM and how that data needs to be preserved. LEM can help satisfy some controls directly. Some of the comments below are about how LEM treats relevant data within the controls, should be implemented to satisfy the controls, or satisfies these requirements specifically.
      • A really good note from AU-6(10) to keep in mind: remember that you can adjust audit levels depending on organizational needs and risks changing! You don't have to just enable the firehose.
    • Of Interest:
      • AU-2: Audit Events
        • LEM helps serve this, but this control is about what you feed into LEM.
      • AU-3: Content of Audit Records
        • Again, LEM stores this data, but generally this is up to logging sources. Where we normalize data, we preserve these fields.
        • AU-3(2) - Centralized Management of Planned Audit Record Content - about automation. At a low level, you would serve with tools like NCM (for devices), or Group Policy, but LEM can play a factor in automating configuration to ensure the right data is captured from similar systems with connector profiles.
      • AU-4: Audit Storage Capacity
        • Depending on your storage requirements you would need to ensure LEM has enough storage capacity to meet your needs, and can implement archiving as well.
      • AU-5: Audit Processing Failures
        • LEM can generate events when agents go offline, when there's an issue storing or processing data, when running out of disk space, and on behalf of other systems when audit logs are cleared, when there are hardware issues we can detect via log data
      • AU-6: Audit Review, Analysis, and Reporting
        • LEM satisfies this requirement, up to you to decide which systems need to be audited and for what, and ensure the required data is logged for collection
        • Correlation with some data sources (e.g. "non-technical sources" in AU-6(9)) may have to be a manual process done as a part of investigation.
      • AU-7: Audit Reduction and Report Generation
        • LEM satisfies this requirement
      • AU-8: Time Stamps
        • LEM satisfies this requirement (note - we will use timestamps provided by log sources as well, but may only be down to the second)
      • AU-9: Protection of Audit Information
      • AU-10: Non-repudiation
        • For data stored and accessed in LEM, LEM satisfies this requirement
      • AU-11: Audit Record Retention
        • Depending on your retention requirements, you'd need to ensure LEM has enough storage capacity to meet your needs
      • AU-12: Audit Generation
        • LEM helps satisfy this requirement
      • AU-14: Session Audit
        • With AU-14(3), you may be able to satisfy some requirements with DameWare.
      • AU-15: Alternate Audit Capability
        • You may want to set up backup logging for devices that syslog, or architect LEM in such a way that you can go to point systems or syslog servers or servers directly to ensure (prove) you can still access data.
      • AU-16: Cross-Organizational Auditing
        • Potentially, you can use LEM to foster cross-organizational auditing (exporting, providing limited access, etc)
  • CA-X: Security Assessment and Authorization
    • General Notes: for the most part, this isn't an area we can help support, but Continuous Monitoring does fall under this area.
    • Of Interest:
      • CA-7: Continuous Monitoring
        • LEM can help facilitate continuous monitoring (correlating security data, alerting, reporting). We also find many federal government customers utilizing NPM, Server & Application Monitor (SAM), and other parts of our monitoring suite to support enterprise-wide continuous monitoring.
  • CM-X: Configuration Management
    • General Notes: A few products can help here, but primarily NCM when it comes to network devices. Patch Manager and LEM can also pitch in in a few key areas.
    • Of Interest:
      • CM-2: Baseline configuration
        • For devices, NCM (and partially FSM) can help establish and automate comparing configs to a baseline, and retaining configs.
      • CM-3: Configuration Change Control
        • For devices, NCM (and partially FSM) can help test/validate/document, automate changes
      • CM-5: Access Restrictions for Change
        • You may be able to use LEM to audit when changes are made depending on components and policies actually changed. NCM for devices and things like dual authorization.
      • CM-6: Configuration Settings
        • CM-6(1) - automated central management - use NCM for network devices.
        • CM-6(2) - NCM can help for devices, and LEM can potentially alert on relevant events in real-time.
      • CM-7: Least Functionality
        • LEM can help audit when unauthorized software and programs are being executed.
      • CM-8: Information System Component Inventory
        • Patch Manager can help audit software and system status.
      • CM-10: Software Usage Restrictions
        • You can use LEM to audit when P2P and other software is used in general, and Patch Manager to audit what's installed on a system, but it may not ultimately be perfect.
      • CM-11: User Installed Software
        • You can use LEM to audit when much software is being installed, and Patch Manager to know what's on a system.
  • CP-X: Contingency Planning
  • IA-X: Identification and Authentication
  • IR-X: Incident Response
    • General Notes: For the most part, LEM can help when it comes to incident generation and investigation, and also leveraging active response can provide you in-the-moment capabilities to deal with incidents as they occur.
    • Of Interest:
      • IR-4: Incident Handling
        • LEM can support this - including IR-4(4) information correlation, IR-4(5) automatic disabling of information system, and IR-4(9) dynamic response capability.
      • IR-5: Incident Monitoring
        • LEM may generate incidents from correlated activity, and this information can be tracked and stored (reports produced, alerts sent, etc).
      • IR-6: Incident Reporting
        • LEM can help support IR-6(1) - automated reporting to report correlated incidents detected from within LEM. (Where other SW products are used to detect and generate incidents, this is also generally true of them.)
  • MA-X: System Maintenance
    • General Notes: NCM is a key player here to help with controlling and managing approvals where it comes to network devices. LEM can help alert when stuff just doesn't seem according to expected maintenance policies.
    • Of Interest:
      • MA-2: Controlled Maintenance
        • NCM can help with MA-2(2) automated maintenance for network devices, and LEM can help audit when maintenance is taking place outside of expected maintenance windows.
      • MA-4: Nonlocal Maintenance
        • LEM can help audit MA-4(1) - auditing and review of nonlocal maintenance.
        • NCM can help with MA-4(5) - approvals and notifications - when it comes to network devices.
  • MP-X: Media Protection
    • General Notes: Most of this isn't relevant when it comes to SolarWinds products, but there's one area when it comes to removable devices where LEM's USB-Defender can help.
    • Of Interest:
      • MP-2: Media Access
        • LEM's USB-Defender can help with the USB removable media component of this.
  • PE-X: Physical & Environmental Protection
  • PL-X: Security Planning
    • General Notes: Several of the mentioned controls are those which may be supported by LEM, which can be used to centrally manage auditing and monitoring, especially within PL-9. Also interesting when it comes to PL-8 is mention of defense-in-depth techniques.
  • PS-X: Personnel Security
    • General Notes: A lot of this is external and policy-related, but think about using LEM to ensure what should happen did (i.e. Trust, But Verify).
    • Of Interest:
      • PS-4: Personnel Termination
        • May use LEM to audit usage of credentials and ensure attempts to use them do not continue after users are terminated.
      • PS-7: Third Party Personnel Security
        • May use LEM to audit usage of third party credentials and ensure attempts to use them do not continue after users are terminated
  • RA-X: Risk Assessment
    • General Notes: There's a lot of policy and procedure here, and really only one area where LEM and Patch Manager especially can help.
    • Of Interest:
      • RA-5: Vulnerability Scanning
        • Can use Patch Manager to assess vulnerable systems by missing patches
          • RA-5(1) Update Tool Capability and RA-5(2) Update by Frequency/Prior to New Scan/When Identified - Patch Manager is automatically updated with new patches
          • RA-5(6) - automated trend analysis - Patch Manager can report on patch status over time
          • RA-5(8) - review historic audit logs - Patch Manager will include audit activity of what is being patched and tracked
        • Also, you can use LEM with a vulnerability scanner to support RA-5(6) and RA-5(8) as well, along with RA-5(10) correlate scanning information.
  • SA-X: System & Services Acquisition
    • General Notes: There's not a lot that applies here to us, but it's worth mentioning that SA-4(8) speaks to ensuring new systems/apps include activity that can be monitored as part of continuous monitoring planning. Think about how you're going to monitor systems as you implement them, rather than after the fact.
  • SC-X: System & Communications Protection
    • General Notes: SC is a pretty fascinating set of controls, with everything from cryptography, to honeypots, to detonation chambers. There's a few places I made notes where SolarWinds products are relevant.
    • Of Interest:
      • SC-5: Denial of Service Protection
      • SC-7: Boundary Protection
        • Monitoring communications with LEM, NTA/NPM, and NCM/FSM for the configuration side.
        • SC-7(8) - you can also use LEM to monitor attempts to bypass proxy server.
        • SC-7(10) - you can generally use LEM for monitoring here.
      • SC-19: Voice Over Internet Protocol
      • SC-29: Heterogeneity
        • Where you have a heterogenous environment, third party monitoring and management tools like SW (e.g. Virtualization Manager, SAM, NPM, and LEM) are more important!
  • SI-X: System & Information Integrity
    • General Notes: There's a big section for LEM in here specific to auditing (aside from the normal steps for compliance), but also a couple of other smaller areas of note.
    • Of Interest:
      • SI-2: Flaw Remediation
        • Patching - Patch Manager can help with SI-2(1) central management, SI-2(5) automatic software updates, and SI-2(6) removal of previous versions
      • SI-4: Information System Monitoring
        • This is all about LEM - also especially SI-4(2) automated tools for real-time analysis , SI-4(4) inbound and outbound communications traffic, SI-4(5) system-generated alerts, SI-4(7) automated response to suspicious events, SI-4(11) analyze communications traffic anomalies, SI-4(12) automated alerts, SI-4(13) analyze traffic/event patterns, SI-4(16) correlate monitoring information, SI-4(17) integrated situational awareness, SI-4(19) individuals posing greater risk, SI-4(20) privileged users, SI-4(22) unauthorized network services, SI-4(23) host-based devices, and SI-4(24) indicators of compromise.
        • You could also use NPM/NTA where traffic comes into play to potentially detect unexpected traffic patterns or performance issues that indicate security issues
      • SI-7: Software, firmware, and information integrity
        • Can use LEM to detect some unexpected changes, e.g. windows does a system file check initially which can create events, and can also use LEM's FIM to detect critical system changes (files, registry keys).
          • LEM would also support SI-7(5) automated response, SI-7(7) integration of detection and response, and SI-7(8) auditing capability for significant events
      • SI-15: Information Output Filtering
        • You would want to integrate these into LEM, and consider something like LEM's SQL Auditor to detect failures when it comes to databases.

 

Double whew! I bet your hot beverage cup is empty at this point, perhaps I should have warned you to use the large one.

 

Got FISMA?

 

Hopefully at this point we've given you a lot more info on how we can help you get moving with FISMA compliance. If you've got questions, feel free to post them and we'll update the post as things change or more details are necessary.

It's been a while since we talked SolarWinds Patch Manager and patching in general here on the Product Blog, but with VMWorld 2015 right around the corner all things virtual are on our minds. Here's a few quick considerations to make when thinking about patching and maintaining virtual systems.

 

Is patching virtual (guest) systems really different? Yes, and no.

 

At the most fundamental level, patching virtual guest systems isn't really different than patching physical systems. You back the system up (hopefully), you install patches (which you tested first, right?), and if necessary, finish with a reboot. Seems simple enough, but there's points along the way where we can really take advantage of virtual systems - and virtual systems can help back us up when we're being lazy (or hasty).

 

  1. Backing up the system: here we can take advantage of the virtual environment's ability to take snapshots, either by integration with our backup system, integration with our patching system, or by hand. Snapshots can really cover your assets when it comes to making a mistake, or if a patch has unintended consequences (not that vendors ever make a mistake, right?). If a system fails to come back after a patch or you need time to diagnose an issue, reverting to snapshot while you clone and re-test is much more simple than the old school "revert from a backup? sigh..." or relying on Windows' ability to take reliable system restore points.
  2. Testing patches: with snapshots and a virtual environment (or even a hybrid or cloud environment), you can clone a live system into a testbed relatively easily. Gone are the days of drive imaging and system cloning, or having standby hardware in a test environment just because it's identical to production. Now, you can clone a snapshot of a production system, tweak its network and VM configuration to move it over to your test environment, and install and test patches pretty easily.
  3. Installing and rebooting: while systems are patching and rebooting, virtual environment HA configurations can help plug some of the holes of down systems without dealing with operating system clustering technologies directly. Both can be admittedly cumbersome to set up the first time, but virtual HA can save your bacon and minimize impact to your downstream users.

 

Don't forget your hypervisor!

 

When it comes to Hyper-V, patching your hypervisor really is all about patching your OS. Tools like Patch Manager are going to make it easy to stay up to date with Windows patches (AND third party patches, too). With Patch Manager on top of WSUS or SCCM, you can make intelligent groupings of systems, both for status and reporting details and for patching.

 

For vSphere (ESXi)-based systems, patching your hypervisor is a little more complex, and patches have been coming about monthly. There's actually a handy table of build numbers to patches published in their Knowledgebase that shows the patch history, and VMware has a Patch Portal to help you find and download updates that apply to you, plus see which KB articles patches resolve. I'd recommend showing the "Severity", "Category", and "System Impact" columns to help you understand which patches are most critical (keep a keen eye on security updates) and what the impact will be to running systems.

 

VMwarePatchPortal.PNG

 

 

Patching utilities for host<->guest communication is important, too

 

Within virtual guest systems, there are usually utilities that establish good host to guest (and vice versa) communication. These tools let you perform clean maintenance tasks like shutdown, reboot, and snapshot; provide time synchronization (very useful if you're doing any log analysis, troubleshooting, or anything certificate-based where time can matter a lot); and provide insight into what's on a guest or host OS.

 

When it comes to VMware Tools specifically, you won't get the tools "for free" when you bring up a clean guest OS until you install them, though thankfully most modern Linux distributions include open-vm-tools by default (or easily added). For those of you tired of this deployment process on Windows, though, we've got good news! Patch Manager now includes VMware Tools packages in our third party update catalog.  With Patch Manager, you can now automatically download and deploy VMware Tools updates just like Windows (and other third party) updates.

 

For existing Patch Manager customers, you can add the VMware Tools library to your patching catalog by following a few steps:

1. Use the Third Party Updates Configuration Wizard to synchronize available updates from SolarWinds

Administration & Reporting > Software Publishing > Patch Manager Update Configuration Wizard

SynchronizingWizard.PNG

2. Click "Next" when the Wizard completes to see the full list of available updates from all vendors.

DoneSynchronizingWizard.PNG
3. Scroll down and make sure "VMware Tools" and "VMware Tools (Upgrade)" are selected from the list of subscriptions.SelectWizard.PNG
4. Click next and finish to confirm your package synchronization schedule, then Finish.PackageSynchronizationSchedule.PNG
5. To see the available packages and versions, go to Administration and Reporting > Software Publishing, then right click and select "Refresh". After doing so, you should see "VMware, Inc" appear in the list, and see the respective packages.PackagesinList.PNG
6. From here, you can select to publish the packages to your WSUS/SCCM server (click "Publish Packages" on the right). Select x86 if you've got any 32-bit systems out there, otherwise select x64, then click Next.PublishingWizard.PNG
7. You'll watch an awesome progress bar for a little bit as it downloads and pushes the packages... then click Next to continue.DownloadingPackages.PNG
8. What do you know, more awesome progress bars as it pushes the packages to the Patch Manager server... (there will be two at first as it pushes the files, then one warning you to be patient as it publishes.). Once it's done, you can hit "finish" to finish the publishing step.

PublishingWizardtoPAM.PNG

DonePublishing.PNG

9. If you head back up to your Updates view, you'll see the new packages in the list.

Update Services > <your server> > Updates > Third Party Updates (you might have to right click on "Updates" and click "Refresh" first).

UpdatesView.PNG
10. From here, you can do your standard Patch Manager tasks, such as Approve the package for distribution and decide which systems should receive the package/update. Click "Approve", then click on each group to approve to and click the "Approved for Install" button (in my example, I approved the update for my Servers group), then click OK. You'll see another fancy progress bar while things finish, then confirm.ApproveUpdate.PNG

You can also automatically download and approve future versions with the new-in-Patch Manager 2.1 auto-approval feature, if you check out our GA blog post there's a bunch of details on that feature - Announcing General Availability of Patch Manager v2.1 - Automated 3rd Party Patches & More!.

 

What's Next for Patching Virtual Systems?

 

If you check out the Patch Manager What We're Working On, you'll see specific mention of more features we're looking at adding regarding patching virtual systems - including the automated snapshotting (and potentially reverting) mentioned above.

 

What big issues do you have with patching virtual systems? What can we do to help?

We've seen time and again that dividing your security attention between the inside and the outside threat (and unfortunately the blend of both - when outsider leverages or becomes an insider) is an ongoing challenge. If you check out our last 1-2 years of Federal IT Security Surveys, you'll see the insider is still a pretty big concern that's far less understood and harder to solve (more on that - Internal Federal Cybersecurity Threats Nearly as Prevalent as External, SolarWinds Survey Reveals), spreading from training to actual technical controls to the challenges of monitoring. In the interest of giving you a bit of a head start, here's some insight into some ways you can monitor for malicious insiders with Log & Event Manager (LEM).

 

(Note: Anywhere you see a screenshot below, be sure to click to see a full version - they might look fuzzy otherwise.)


Endpoint Monitoring with File Integrity Monitoring (FIM) and USB-Defender

Out of the box, LEM includes both built-in File Integrity Monitoring (FIM) - which can audit for file and registry access/changes - and USB-Defender - which monitors USB device access. On systems where you may have potential exposure - think kiosks, systems with access to confidential data, servers, and shared workstations - deploying FIM and USB-Defender will allow you to:

  • Monitor for unexpected copying of files and data to USB devices that can indicate data is being exfiltrated
  • Attempts to bypass application installation and access policies by running applications directly from USB devices that can put systems at risk
  • Changes to system settings and files that can indicate potential unexpected modifications, either due to malware, policy bypassing, or intentional abuse

 

Out of the box, you'll want to look at the following LEM content:

  • Default FIM Monitors - the Windows Server template can also be applied to workstations as a place to start

FIM Monitors.PNG

  • Filters of interest:
    • Endpoint Monitoring > USB-Defender
    • Change Management > USB File Auditing, All File Audit Activity

EndpointFilters.png

  • Rules of interest can be found in the categories:
    • Activity Types > USB Device Monitoring, File Auditing

EndpointRules.png


System and Endpoint Monitoring for Authentication and Change Events

Beyond tracking files and USB Devices, on servers and workstations alike authentication and changes can offer unique insights into what's happening on the network, and provide critical clues when it comes time to investigate. Windows does not audit the mechanism a user used to log on, or changes made to local system accounts, at a domain controller, so without insight into the actual workstations and member servers directly you'll be missing pieces of the puzzle. Deploy agents to all your critical member servers and that same pool of workstations you need insight into and get to tracking the local Event Logs. With this data, you can see:

  • Users logging on unexpectedly - unused accounts suddenly being used, service accounts being used to access the wrong systems, admin accounts being used incorrectly
  • Remote access - usage of remote desktop vs. interactive logins, access from VPN accounts/addresses, contractors authenticating to unexpected systems
  • Additional users & privileges - users being added to local or domain admins, local users being created

 

Out of the box, you'll want to look for the following LEM content:

  • Filters of interest in these categories:
    • Change Management
    • Authentication
    • Endpoint Monitoring

AuthFilters.png

  • Rules of interest in the following categories:
    • Change Management
    • Authentication
    • Activity Types > Inappropriate Usage

AuthRules.png


Network Device Traffic Monitoring


If we move off of the systems themselves, we should also be able to detect behavior patterns that look abnormal using network traffic events, too. Sometimes putting agents on all workstations is infeasible, not to mention accounting for transient or new devices, and BYOD if you've got that in the mix as well. Log activity from all the devices you can that can monitor traffic patterns and connectivity - IDS/IPS, firewalls, wireless APs/WLAN controllers, routers, switches, VPNs, etc. With network traffic data, we can look for:

  • If you've got a proxy or similar policy in place, users attempting to bypass proxy policies with direct communication on port 80 (i.e. network traffic that's not outbound from your proxy server)
  • Network traffic to/from unexpected hosts or ports - your servers/workstations will generally communicate to a smaller subset of known hosts, traffic outside of this pattern would be unexpected
  • Excessive network traffic - sometimes traffic patterns can become clear without utilizing netflow or deep packet inspection based on sheer event numbers, types, or behavior patterns alone

 

Out of the box, you'll want to look for the following LEM content:

  • Filters of interest:
    • Start from the out of the box filters in IT Operations and Security and build from them, especially the traffic filters

NetworkFilters.png

  • Rules of interest in the following categories:
    • Activity Types > Network
    • Devices > Firewalls

NetworkRules.png

 

Check out our thwackCamp session on using firewall log data, too - thwackCamp 2015 - Digging for Security Gold: Using Firewall Logs to Find Security Issues.


Traditional Malware and Security Event Detection

You can definitely put your existing investments in pure security technology to work for you here, too. The name of the game is defense in depth, and while traditional malware detection, IDS and IPS, and other tools might not be enough alone, each one of them can play an important part in helping detect potential abuse or piecing together fingerprints during an investigation. Infected endpoints are a gateway to the interior of the network and not all of us are victims of zero-days but rather some kind of combination of existing malware and other techniques that gives us a good chance of detecting it somewhere along the way. With these feeds, you'll see things like:

  • Antivirus/anti-malware technology cleaning or having trouble cleaning potential infections
  • IDS and IPS systems detecting potentially unwanted payloads or symptoms of infections or even exfiltration
  • Triggers from any other security systems you've got to put to work for you that generate event streams - wireless security, data leak prevention, etc
  • System errors and crash reports - potential malware causing leaks to affect the system in unexpected ways

 

Out of the box, you'll want to look for the following LEM content:

  • Filters of interest include:
    • Security > Virus Attacks, IDS
    • IT Operations > Windows Error Events

MalwareFilters.png

  • Rules of interest in the following categories:
    • Security > Malware
    • Devices > IDS and IPS (and related device types for your systems)

MalwareRules.png


Threat Intelligence and Dynamic Feeds to Detect Malicious Traffic

Thinking forward, if you've seen our LEM What We're Working on page, you'll note we're talking a little bit about Threat Intelligence Feeds. We're working on adding the capability for LEM to dynamically download a list of known bad actors - potentially infected hosts, botnets, command and control networks, spammers, and general IPs up to no good - and automatically use that to detect communication on your network. This will be a really good way to see:

  • When someone internal is communicating with a potentially malicious host, which can indicate they've already been infected
  • When you're being probed, attacked, or otherwise communicated with externally by a potentially malicious host, which can indicate an incoming attempt
  • Communication to/from spam, denial of service, or similar hosts that can indicate phishing attempts, zombies on your network, or other security issues

 

Watch for more on that here - when we've got more to discuss we'll update this post with how to use it to detect malicious insiders more specifically.

 

Manually, you can create and import lists of potentially unwanted IPs and ports and compare those to traffic as well. If you've got a list of known good ports that should be used to communicate on your network (especially inside>outside), or known applications if you're using Next-Gen firewalls, or known IP addresses when we're talking servers and controlled communication, build User-Defined Groups and rules/filters that compare to them.


What About Other SolarWinds Products? How Can They Help, Too?

Sure! Here are some ideas on using other products to help you detect potential malicious behavior internally:

  • Network Performance Monitor: monitor for unexpected firewall/network performance issues and high bandwidth utilization that can indicate an outbreak or single host is infected
  • Netflow Traffic Analyzer: building on the above unknown traffic patterns, look for possible unexpected hosts, ports, or communication patterns that might give you an idea something is wrong
  • User Device Tracker: useful when tracking and potentially detecting issues at endpoints - the "who" to go with the "where"
  • Server & Application Monitor and even Virtualization Manager: look for systems & applications performing unexpectedly or becoming unstable, these can be early warnings for security issues, too
  • Database Performance Analyzer: building on that, look for batch transactions, long-running queries, and sudden performance issues, identify their sources
  • Network Configuration Manager and Firewall Security Manager: as always, cover your bases with configuration first!
  • Patch Manager: track systems out of compliance with patching policies, out of date systems are MUCH more likely to be victims of malware and other security issues

 

Feel free to let us know if you've got any content you're interested in seeing around detecting malicious insiders, any ideas or successful stories yourselves, or any other questions we can help with in the comments!

We wrote back in 2012 about the challenges of SharePoint auditing and how to address them via Auditing SharePoint with LEM & LOGbinder SP, but the folks over at Monterey Technology Group (the same folks who brought you Ultimate Windows Security) went on to create even MORE useful Microsoft auditing tools. This time around, we've also integrated LOGbinder for Exchange (LOGbinder EX).

 

Without LOGbinder EX or a tool like it, it's very hard to get visibility into the Exchange auditing logs. Audit data is stored as a part of the mailbox instead of the Event Log, and there's no clean way to get the data into the Event Log repeatably and consistently. Even if you were able to do that, there's a ton of coded data, with different types and metadata that you'd have to translate. The LOGbinder system does this automatically, storing data into the Event Log and both making it easy for you to read and for a system like Log & Event Manager to monitor, alert, and store it.

 

Use LOGbinder EX for:

  • Detecting non-owner mailbox access (e.g. delegate or users opening other users' mailboxes)
  • Changes to audit log settings and audit log integrity
  • Permissions, policy, certificate, federation, and IRM changes

Check out the full list of events LOGbinder EX generates for more details.

 

Use LEM + LOGbinder EX together for:

  • Alerting on unexpected client activity (mailboxes accessed from something other than Outlook/OWA)
  • Alerting on unexpected mailbox access (someone opening one or many mailboxes other than their own)
  • Alerting on unexpected changes across Exchange infrastructure
  • Reporting on Exchange audit and change management events
  • Viewing Exchange events in context with other system, network, security, and application events

 

I just uploaded some rules, filters, and reports for LOGbinder EX over at the Content Exchange that provide some additional insight for the LEM side of your configuration. There's an integration guide in the Zip file that will explain how to install the files, which are all tailored to the LOGbinder EX event log data. You will need an agent installed on your LOGbinder EX system, you'll need to make sure you have the latest product connectors installed, then it's just a matter of following the guide to get set up and start monitoring. You can download a free trial of LOGbinder for Exchange from their website, too.

The Patch Manager team is pleased to announce general availability of Patch Manager version 2.1. All customers under active maintenance can find the download in the SolarWinds Customer Portal, and anyone interested in Patch Manager can download from solarwinds.com.

 

We talked about what's new in detail back in the beta days (Patch Manager 2.1 Beta 1) , but let me quickly refresh your memory.

 

Automated Third Party Patches

 

We've added the ability to automatically publish and patch third party updates similar to the way WSUS/SCCM handle Windows updates. With this feature, you can automatically download, publish, and patch third party products from our 3PUP catalog (see: Table of third party patches - updated 4/8/2015 for the latest third party patches).

3rd+party.JPG

Important note: there are some providers that don't allow us to automatically download and publish their updates without a click-through EULA acceptance. For now, these providers (including Oracle and Adobe patches) won't be supported through this feature. Hopefully we'll find a good solution to this in the near future - we're working with the third party vendors directly to hopefully find some options.

 

New Reports

 

Thanks to thwack feedback and discussions, we've pulled in a lot of common custom report requests and now have them included out of the box! These reports should make it easier to identify problem systems and report on updates.

 

Included are:

  • Custom Hardware Report
  • Installed Programs and Features Basic
  • Approved Updates Status Count by WSUS Server and Update Source
  • Computer Update Status - Locally Published Updates
  • Computer Update Status with Aggregate Counts of Install State for Approved Updates
  • Computer Update Status Counts by Classification for Approved - Not Installed
  • Computer Update Status - Approved Updates with ID and Revision

 

New Computer Group Scoping Options

 

A common request is to make it easier to find and manage computers across subnets, Active Directory OUs, or AD sites. We've added several new grouping options:

  • IP subnet or range
  • Active Directory Organizational Unit (OU)
  • Active Directory Site

 

...and more!

 

In addition to the standard bugfixes and improvements, we've also:

  • Made it easier to gather logs for troubleshooting with the support team
  • and, added support for Windows 2012 R2 and Windows 8.1 as options when selecting computer properties

 

As always come over to the Patch Manager space here on thwack if you've got any questions. Happy patching!

Here on the Log & Event Manager product team we often get questions about how we maintain our appliance and ensure integrity of the data we collect. We previously published this KB article, but this post includes a quick recap of the critical areas relevant to LEM's end-to-end security.

 

This diagram is a good overview of the specific areas of interest, with even more detail in the sections below.

LEM security diagram2.png

 

Hardened Operating System

 

LEM applies security elements at just about every level to include the OS. Since the product is deployed as a virtual appliance, several security measures are applied at the OS level to include:

 

  • A Debian Linux core operating system with all unnecessary ports, protocols, processes and services/daemons disabled.
  • Operating system and application maintenance is performed regularly with LEM upgrades, for both security and stability updates.
  • No root level access – in fact, all passwords are randomly generated per-appliance, and even our internal teams don’t have knowledge of them in advance.
  • Minimal access to the appliance - only via virtual appliance console or SSH.
    • For low level appliance configuration (including things like networking and backup configuration), LEM includes a menu driven command line interface that requires a username/password via SSH over a nonstandard port.
  • Remote configuration access can also be disabled or restricted by IP address, and an appropriate usage banner can be displayed.
  • Internal logging and auditing.

 

Web Console Security

 

LEM’s web console provides a secure read-only view of data and access to LEM’s configuration. This access can be restricted and further secured.

 

Security features applied to the web console include:

  • Encrypted Console access that is certificate based over secure HTTP
    • Option to deploy a CA-signed certificate in addition to the included self-signed certificate
    • Option to disable HTTP access in favor of only HTTPS
  • Additional console access limitation applied on a per IP basis.
  • Local or Active Directory users with role based access. The roles available are:
    • Administrator - Users who have full access to the features and capabilities within the web console.
    • Auditor/Guest - Users who have extensive view rights to the system, but cannot modify anything other than their own filters.
    • Monitor - Users who can access the Console, but cannot view or modify anything, and must be provided a set of filters.
    • Contact - Users who cannot access the Console, but do receive external notification.
    • Reports – Users who can only run Reports, but cannot access the Console for real-time monitoring.
  • Password complexity requirement for local users (AD users inherit AD policy).
  • LEM active responses are pre-configured and not "open" scripted (and don't accept input on the client-side).

 

Data Storage Security

 

The crux of securing the LEM appliance is ensuring the data cannot be altered. To that end the following measures have been applied:

 

  • Data storage is encrypted and hashed – should access to the appliance be breached, data can’t be tampered with and served back to LEM as if it were unmodified.
  • All access via LEM’s tools to data storage is read-only, ensuring that data cannot be altered regardless of the role assigned to the user.
  • Data added to the LEM appliance is insert-only, only removed to make room for new data and never “updated” or edited (except accumulated metadata).
  • Access to LEM reports can be further restricted by IP address and leverage certificate-based TLS communication.

 

Log Collection Security

 

As logs are collected, chain of custody is ensured – again, to prevent as much tampering as possible.

 

  • Logs are collected on the agent as close to real-time as possible, ensuring data lives on disk a minimum amount of time.
  • Windows Event Log data is collected using the Event Log API, not relying on data to be written to disk to be collected.
  • Where appropriate, LEM tools avoid collecting personally identifiable information. SQL Auditor, for example, does not include full queries and responses, which could have personal data in either the statement or the response.
  • Logs collected using the Log & Event Manager agent are protected in transit with FIPS-approved TLS/SSL encryption algorithms.
  • When communication is interrupted, data is buffered to disk in a binary format, not the original modifiable log content.

 

Archives

 

Security is not limited to the data that sits on the Virtual Appliance. In the event archives are used and need to be restored, their integrity is also maintained.

 

  • Database and log archives are encrypted and hashed to prevent tampering

 

Internal Auditing

 

Detailed auditing of all appliance activity is enabled by default, and can be leveraged in both alerts and reports. This includes:

 

  • Change Auditing - All changes to rules, reports, filters, searches, widgets and other internal element.
  • Access Audit - Successful and Failed attempts to access the web and reporting console.
  • Rule Activity – Any rule that fires creates a full audit trail of the actions taken.
  • Report Activity – Any time a report is generated automatically or manually.
  • Search – All search activity is recorded and includes the source IP and username.
  • Built-in audit reports that can be scheduled and reviewed.

 

SolarWinds Security Processes

 

Our internal security practices are leveraged with every product release.

 

  • LEM is Common Criteria certified, which includes both product and processes
  • Vulnerability scans are ran during the product development cycle and reported issues addressed
  • Time is allotted during each release cycle to include security updates, patches, and fixes
  • Internal response plans to assess critical vulnerabilities (like Shellshock), reported external vulnerabilities, and reported customer vulnerabilities via your own internal scans

 

 

If you’ve got any questions about how LEM security is maintained, or any features we might have missed, let us know – another part of security is recognizing that our processes must be constantly improved.

The Log & Event Manager team is pleased to announce the official release of LEM 6.1. Features included in this release:

  • NEW: Getting Started experience - configure LEM basic settings and easily walk through all the steps to get LEM up and running from one place
  • NEW: Rules Configuration Wizard - quickly enable LEM rules important to you with only a few clicks
  • Fixes and Improvements:
    • Improved IIS logging support - now more flexible and reliable on first configuration
    • Support for Windows 8.1 with workstation edition - no longer consuming Universal licenses
    • Support for SQL 2014 with SQL Auditor - audit SQL 2014 databases along with your 2012, 2008, and even 2005/2000 databases
  • New and updated connectors, as always!

 

Here's a handy video about adding rules with the new Rules configuration wizard:

 

For even more details, be sure to check out the LEM Release Candidate blog post: Check Out Quicker & Easier Config with the Log & Event Manager v6.1 Release Candidate - Now Available!

 

And, as always, the Release Notes with all the juicy details can be found here.

 

Download Information for LEM v6.1

 

Customers on active maintenance can download LEM v6.1 from their Customer Portal.

 

New to LEM and want to see what it's all about? Check out our product page and download a free (fully functional) trial right here on solarwinds.com.

The Log & Event Manager (LEM) team has been hard at work on a release intended to make your lives easier. We know you're swamped and decided to take a little time to make it faster to get LEM up and running and configure rules related to problems you're interested in solving without looking through a big list and clicking, clicking, clicking.

 

Getting Started with LEM: New & Improved!

GSWidgetOpsCenter.PNG

 

We had a handy dandy getting started widget in LEM before, but we've taken it one step further and glued together those steps into one location rather than sending you on a bit of a wild goose chase. Now, from the Ops Center Getting Started widget you can:

  • Quickly configure Basic Settings needed for LEM to be up and running - email server settings and directory service server (for groups and/or authentication) configuration
    • As you click "Next" to move through the wizard, each step will be tested and verified. If you see a pause, that's what's going on. If there's a problem, LEM will let you know.

BasicSettingsWizard.PNGBasicSettingsFailureSummary.PNG

  • Quickly access the Add Node Wizard including links to the agent installers and the full syslog scan to configure connectors automatically

GSNodeWidget.PNG

  • Use the NEW Add Rules Wizard to add rules for different areas of interest (more on that later)
  • And, view all of the quick training videos from within the LEM console!

GSAdvancedTools.PNG

 

New Feature: Add Rules Quickly by Category

 

Our next big addition is the new Add Rules wizard. From Build > Rules or from the Ops Center Getting Started widget/wizard flow, you can launch the fancy new wizard. This wizard will configure for you ALL rules that match a given category that can be configured easily - no active responses, just email and "infer alert" or "create incident" actions (we'll look at improving to add more active response choices in the future). This should be MUCH MUCH faster than all that cloning you had to do before. As a part of this, we've also created a new "General Best Practices" subcategory in each parent category - if you're not sure where to get started, these categories will get you a wide swath of the most common rules enabled.

AddRulesWizard.PNG

          Select each category of interest

AddRulesWizardSubcategory.PNG

          View and select subcategories, including the new General Best Practice subcategory

AddRulesWizardEmail.PNG

          Specify email server settings (if not already configured), and email recipients (even add contacts from within the wizard if they don't exist)

AddRulesWizardFinish.PNG

          Clicking "Finish" will clone, select the right users, and enable the selected rules all in one step!

 

We've also revamped our quick rules training video to include how to use the new wizard and a fast example of building a rule by hand:

 

 

 

Fixes, Fixes, Fixes

 

As always, we've fixed a bunch of customer issues and included notable minor improvements. Included:

  • Improvements to our IIS coverage - if you've struggled with which fields to configure, or IIS not working right after you configure it, this is for you!
  • Support for Windows 8.1 with Workstation Edition - if your 8.1 workstations are coming up as servers and pulling Universal instead of Workstation nodes, this will help
  • Lots of new connectors! As always you can download connectors out of band to releases, but you'll get them automatically with the upgrade, too.

 

Sounds Great - Where do I Download? Where do I Ask Questions?

 

Easy: from the Customer Portal!

 

If you've got any questions, head on over to the Log & Event Manager Release Candidate thwack forum and let us know.

 

As always, feel free to post here and/or contact me directly with any feedback.

Last week, we released version 6.0.1 of Log & Event Manager. Normally we don't make too much noise about service releases (minor dot releases) 'round these parts, but this time we decided to make an exception. We packed a lot of security enhancements and customer requests into this release that you should definitely be aware of.

 

Enhanced Security Features

  • Removal of several flagged "vulnerabilities" on the LEM appliance: We continuously monitor security scans and while rarely (dare I say, almost never?) has there been an issue without a mitigation (like ShellShockfor example), we do still try to make sure we reduce and ideally eliminate any vulnerability scans from flagging the LEM appliance in any way. With this release, we've cleaned up all known security scan flags except the visibility of the Tomcat error page which we're looking into for a future release, and a couple of certificate triggers which are expected and would be resolved by using a CA-signed cert.
  • Better support for using signed certs: We had some customers use the ability to sign and re-upload certs for the LEM console, but there were some cases where it didn't work quite right. We shored up our support for certs and things should be much improved.
  • Improved enforcement for password storage for connectors: some connectors require storing username/password credentials (connectors that use a database to retrieve log data, for example) so we've beefed up encryption and version enforcement for storage of those passwords. Once you upgrade your 6.0.1 appliance and console, you'll need to also upgrade any agents that might have one of these connectors configured (or where you'll configure one going forward).

And the big one...

  • Named user access and TLS support for reports: our database has been encrypted since version 5.6 (and even before that has always been limited access), but using JDBC access and a fixed username was cause for concern for some folks. We've migrated to using LEM users (including AD users) for reports instead, and optionally allowing you to enable TLS connectivity.
    • There's a new "reports" role in the LEM console that you can assign if you have users that shouldn't have access to real-time data, but do need access to historical data. In addition, admin and auditor roles also have reporting access (but not monitor users).
    • When you install v6.0.1, be sure to install v6.0.1 reports, launch it, and specify your access credentials, especially if you have scheduled reports. Your reports won't run until you do.
    • If you're interested in using TLS, use the CMC's "enabletls" command to toggle TLS support for reports (you'll have to export the cert using "exportcert" and then import into the reports console as well).

 

FIM (File Integrity Monitoring) Updates

  • Fix for the "NT AUTHORITY\SYSTEM" username when accessed by a fileshare: we put this out in a hot fix but now it's incorporated in the agent install and agent automatic upgrade. When someone accesses a file remotely, the username should be shown instead of the stock NT AUTHORITY\SYSTEM user.
  • Fixes for several configuration issues with FIM: we've had a few issues reported with FIM from customers - directories not displaying, for example - that we've resolved.

 

But Wait, There's More!

  • Support for SQL Auditor with SQL 2012: we're working on SQL 2014, too, but for now we officially support SQL 2012 with SQL Auditor, along with the previous support for earlier versions (2008, 2005, 2000).
  • Better support for large memory configurations: customers with high throughput have assigned extra RAM and CPU to the LEM appliance, but support often had to remote in and tweak some settings. We've improved our auto-tuning on startup to detect and support these configurations.
  • Several additional utilities and smaller fixes, including: improvements to our internal logging, a utility to rebuild indexes more easily, and as always more officially supported connectors (remember, you can download these at any time - see SolarWinds Knowledge Base :: How to apply a LEM connector update package)

 

Customers on maintenance can download the LEM v6.0.1 upgrade on the Customer Portal immediately. For everyone else, the download on the LEM Product Page is now v6.0.1, too.

 

Be sure to check out What We're Working on - Log & Event Manager Edition for some ideas on where we're going next. If you've got any questions about v6.0.1 or all things LEM, post them here or over in the Log & Event Manager forum.

We've been getting an increasing number of questions about the ShellShock vulnerability that was announced, this post will collect the status across different products into one place to make it easy for you to determine if your product is affected or not.

 

What is ShellShock? How does it work?

 

ShellShock is a vulnerability in a command shell commonly used on Linux (and some other Unix flavors) - basically EVERY Linux system out there (before yesterday that hasn't been patched today) is vulnerable in some fashion. The vulnerability allows someone with local access to log in to a Linux system OR remotely run unchecked commands to a linux system (via the web, for example) to elevate their privileges such that they may even have root-level access (at least, they'll have the context of the process they exploited - the service or user account). At that point, changes could be made to the system, additional services could be run (to do things like serve exploit or phishing sites), and further exploits could be attempted to get root-level access and full control.

 

There is not YET a massive scale exploit of this vulnerability, but it's entirely possible that before the day is done, one will be in the wild (we're already seeing smaller scale exploits winding up). With so many web applications that control and access Linux systems (doing things as simple as image manipulation or as complex as system management control panels, for example) and the common usage of Linux for web servers and application platforms in general, it's probably not going to be very long before something is written and scaled to take advantage of this exploit and create a ton of zombie systems out there.

 

For more reading, there's a great summary post here: Troy Hunt: Everything you need to know about the Shellshock Bash bug

 

What SolarWinds Products are Affected?

 

First, anything that exists exclusively on Windows is not affected. This is the majority of SolarWinds products - including NPM, SAM, NCM, Patch Manager, and more.

 

Products installed on a Linux OS or used to manage a Linux OS are not vulnerable, but their underlying system may be. Storage Manager or Serv-U on Linux isn't affected, but if your Linux OS is, you should consider that system at-risk. Similarly, if you are using LEM agents or monitoring Linux systems, those software bits are okay, but the underlying OS probably isn't.

 

The only affected products are our virtual appliance-based products, which run limited versions of Linux.

 

Below is a chart of all products, which are affected, and mitigation or resolution steps you can take if necessary.

ProductAffected?Notes & Next Steps
Alert CentralPartially, See Notes

Alert Central is running a vulnerable version of bash, however at this point none of the exploit vectors apply:

  • Access to the virtual appliance shell requires authentication and the exploit does not elevate privileges.
  • It is not possible to exploit the vulnerability remotely.

 

To mitigate the threat, limit access to the virtual appliance management console and VAMI configuration interfaces where commands can be ran and instantiated. Ensure your appliance "admin" password used for VAMI access is set and secure.


To be safe, we will include the updated bash software in an upcoming Alert Central release. If it becomes necessary to issue a patch, we will do so. More info will be posted here.

Log & Event Manager (LEM)Partially, See Notes

LEM is running a vulnerable version of bash, however at this point none of the exploit vectors apply:

  • It is not possible to exploit the vulnerability interactively (customers do not have access to an authenticated bash prompt).
  • No LEM management commands allow setting environment variables or are used in a vulnerable way.

If you are still concerned, you should limit access to the virtual appliance management console and restrict SSH access to LEM using the LEM advanced configuration console (CMC).

 

To be safe, we will include the updated bash software in an upcoming LEM release. If it becomes necessary to issue a patch, we will do so. More info will be posted here.

Virtualization ManagerPartially, See Notes

Virtualization manager is running a vulnerable version of bash, however at this point none of the exploit vectors apply:

  • Access to the virtual appliance shell requires authentication and the exploit does not elevate privileges.
  • It is not possible to exploit the vulnerability remotely.

 

To mitigate the threat, limit access to the virtual appliance management console and VAMI configuration interfaces where commands can be ran and instantiated. Ensure your appliance "admin" password used for VAMI access is set and secure.


To be safe, we will include the updated bash software in an upcoming Virtualization Manager release. If it becomes necessary to issue a patch, we will do so. More info will be posted here.

Web Helpdesk (WHD)Partially, See Notes

Web Helpdesk is running a vulnerable version of bash, however at this point none of the exploit vectors apply:

  • Access to the virtual appliance shell requires authentication and the exploit does not elevate privileges.
  • It is not possible to exploit the vulnerability remotely.

 

To mitigate the threat, limit access to the virtual appliance management console and VAMI configuration interfaces where commands can be ran and instantiated. Ensure your appliance "admin" password used for VAMI access is set and secure. Please also note this KB article for WHD Virtual Appliance patch SolarWinds Knowledge Base :: Bash Code Injection Vulnerability - Shellshock.


To be safe, we will include the updated bash software in an upcoming WHD release. If it becomes necessary to issue a patch, we will do so. More info will be posted here.

Patch Manager No
DameWareNo
Firewall Security Manager (FSM)No
Storage ManagerNoWhen the patch is installed on the same system, Storage Manager will continue to function normally.
Serv-U ProductsNoServ-U does not run any shell scripts except during install time and it sets no environment variables. The only other use of sub process spawning is direct shell commands to manipulate files, and no environment variables are set.
Network Configuration Manager (NCM)No
Kiwi ProductsNo
Enterprise Operations Console (EOC)No
Web Performance Monitor (WPM)No
Server & Application Monitor (SAM)No
Network Performance Monitor (NPM)No
User Device Tracker (UDT)No
Network Topology Mapper (NTM)No
Netflow Traffic Analyzer (NTA)No
Failover Engine (FoE)No
Mobile AdminNo
ipMonitorNo
IP Address Manager (IPAM)No
VoIP and Network Quality Manager (VNQM)No
Free ToolsNo
Database Performance Analyzer (DPA)No
Engineers ToolsetNo

 

Can any SolarWinds products help determine if I have other systems affected?

If you've got Server & Application Monitor, user mcam posted a template you can use here in our Content Exchange: Bash Vulnerability Test. You can use this to check and change the status of a monitored Linux node if it comes up vulnerable.

 

How do I fix them or prevent them from being attacked?

 

The fix is pretty straightforward - check your Linux distribution maintainer for an update to bash, or as Troy Hunt suggested in his article, compile and deploy your own.

 

You can prioritize what to fix based on how the attack works (requires a shell to be ran to instantiate the attack):

  1. Systems with web or remote control applications that run local commands on the appliance after taking input from users
    1. If you can identify a known vulnerable application (e.g. cPanel), patch the application AND the system - there may be future attacks that the application will now also protect you from
  2. Systems with accounts where you allow people to log in and run commands arbitrarily
  3. Systems with sensitive data or access to sensitive networks
  4. Anything else!

 

If you can't fix something right away, here are some suggested mitigation steps:

  1. Disable or limit access to web or remote control applications that run local commands - ESPECIALLY from the public-facing internet.
    1. NOTE: If you're using SSH, it can't be exploited EXCEPT by an authenticated user, so you don't necessarily need to limit visibility entirely (though it may still be a good idea!)
  2. Disable or limit access (local or remote) from any unnecessary accounts.
    1. For accounts that run services, prevent them from logging in and spawning a shell.
      1. NOTE: Make sure any services you're running, especially accessible from the internet, use service accounts, not root.
    2. For accounts that only need access to something like FTP, prevent them from logging in and spawning a shell.
    3. Audit for dead accounts - users that may not exist any longer.
  1. Consider disabling login access to systems that have access to sensitive data or networks to only critical users while you deploy the fix.
  2. Monitor for common post-attack signs:
    1. Usage of the root account
    2. Services restarting (could be a sign of configuration changes)
    3. Accounts being created and/or sudo access being granted
    4. Monitoring systems being disabled/shut off

 

We'll update this post if anything changes or as more information becomes available on the updates for our virtual appliances.

To receive updates on the LEM roadmap, JOIN thwack and BOOKMARK this page

 

(Updated March 2, 2015)

Following our 6.1 release of LEM (more info: Announcing General Availability of Log & Event Manager 6.1 - Better Config, and More!), we're back to work on more features and improvements. Some of these features will be included in our 6.2 release of LEM (more info: Log & Event Manager 6.2 and a Threat Intelligence Feed).

 

Obligatory disclaimer: Comments given in this forum should not be interpreted as a commitment that SolarWinds will deliver any specific feature in any particular time frame. All discussions of future plans or product roadmaps are base on the product teams intentions, but those plans can change at any time.

 

Here's what's on the top of our radar:

 

Also, the following ongoing/longer-term items:

  • Ongoing performance investments in our core data processing
    • Customers are sending more and more data to LEM, naturally, so we're investing some time in staying ahead of the curve.
  • Ongoing investments in new connector development
  • Continued customer feedback-driven fixes and updates
  • Better integration with LEM and the Orion platform products (Integration: Log & Event Manager and Orion Platform)
  • Connector building, generic connectors, and general data integration (Build Your Own Syslog Connectors, among others)

 

Be sure to let us know here, and in the Log & Event Feature Requests forum, if there are features you're really keen on. This list doesn't enumerate a lot of the features we're looking into for long term development and further releases, but we continually use Thwack as our biggest source of feedback.

The time has come for yet another Log & Event Manager (LEM) Release Candidate! The RC is already available on the Customer Portal for all LEM customers under maintenance. As a Release Candidate you can deploy it in production and work with our awesome support team if you need any assistance. Here's what you'll see in the RC...

 

Automate Searching and Augment Reporting with Scheduled nDepth Searches

 

Reporting is useful when you want static content with graphs and charts with pages of content, but it's hard to slice and dice the data and it can be tough to get and edit your report criteria just right. Our search interface, lovingly called nDepth, has the ability for you to do more flexible searching, using components like User-Defined Groups and Directory Service Groups, and to piggy back on existing filter criteria to get a jump-start. With this release, you'll be able to take any Saved Search in nDepth (in the normalized data store or the original log message store) and both generate an event from it and/or have the results sent to you in email.

 

nDepthSavedSearch2-MenuSmall.pngnDepthSavedSearch3-Schedule.png

 

Let's say I've got a saved search (or am using a default saved search!) for Logon Failure activity for the last week. With reports, I can schedule, filter, and export to different formats, but I might also want to create my own charts or pass the data off to another team for investigation, which are harder to do with reports. nDepth has a new option in the gear menu on the left side, "Schedule," which will open up a dialog that lets me schedule any saved search on whatever repeating interval I like. By specifying an "End Date," I can also decide how long I want the scheduled search to run, in case there's a short-term issue that doesn't need to be ran indefinitely. If I choose to email the results, up to 10MB (millions and millions of records) will be included in an attached zipped CSV file with all of the original data, similar to a manual export from the Console, except MANY more results.

 

Support Flexible Workstation Environments by Recycling Agent Licenses Automatically


VDI and other flexible temporary workstation initiatives are becoming much more commonplace, but even temporary workstations need to be monitored the same as their semi-permanent counterparts. With LEM Workstation Edition, we've made licensing affordable for workstations, and with this release we've made it possible to automatically recycle licenses from nodes that haven't sent any data in a while.

 

You'll find the license recycling feature (off by default!) in Manage>Appliances>License toward the bottom. With this feature you can:

  • Specify the age of last event before the license is eligible to be recycled (e.g. must have been offline for more than an hour, in case someone is rebooting or temporarily shut down): default 1 hour
  • Specify the schedule frequency to recycle licenses (e.g. every day at 5am, check for old licenses to recycle): default every day at 4am, and
  • Specify the matching parameters for what systems to recycle so that unexpected systems don't get deleted (e.g. only nodes with hostnames or IP addresses that match your VDI network): default all nodes

 

AgentLicenseRecycle-small.pngAgentLicenseRecycle2-UDG-small.png

...But Wait, There's More!

 

Import User-Defined Groups from CSV Files

A commonly requested feature is the ability to import CSV files to automatically populate groups, rather than having to edit data elements by hand, which we've implemented in this RC. From Build>Groups, go to (top right) Gear>Import, change to "All File Types" and choose your CSV file. The format of the file is basically what you see in Build>Groups:

UDG, UDG Name, UDG Description

Element Name, Element Data, Element Description

Element 2 Name, Element 2 Data, Element 2 Description

...


Performance and Platform Improvements

We're investing time in improving things under the hood, too. With this release, we've done some heavy lifting in the correlation engine, updated our agent and appliance Java Runtime Environments, updated Tomcat, and a lot of other somewhat invisible changes. For those of you who want to prevent an agent update from automatically being pushed out after upgrading, make sure to go to Manage>Nodes or Manage>Appliances and turn off Automatic Updates for specific nodes or globally.

 

We've also improved small areas like the performance of nDepth CSV export from the Console (be sure to check out scheduled searches if you still need to export more than 250,000 records), adding more info to our troubleshooting logs to help our support team help you faster, and a ton of other things.


New Connectors and Device Support

We'll provide a more complete list with the release notes, but the most notable addition is that we've included out of the box support for NetApp File Auditing. Most new connectors are released regularly with the connector download, but for NetApp auditing you'll need to upgrade your appliance and agent to the new release first.


Questions, Issues, Comments - Send 'em Our Way


Feel free to use the Log & Event Manager Release Candidate Thwack forum to report and comment on any issues, questions, or comments you have about this release. Our product management, development, and QA teams are keeping an eye out for any possible issues.


If you have a question about whether a case you've filed was resolved in this release or a certain feature request implemented, feel free to ping back on this post or in the RC forum and let me know - I'll be sure to look into it.


Happy Logging!

Hey everyone, the latest release of Alert Central is now available for download!

 

Here's what we've been working on for this release:

 

  • Notification of Alert Central upgrades in your admin AC console and admin weekly report (and support for proxy servers, so your upgrade feed doesn't get interrupted)
  • A slew of improvements to user validation to help those of you with cell phones and improve your expected workflow, including:
    • The option to not automatically send the self-validation email on import
    • The ability for an admin to see whether a user's address is self-validated from the users grid, not just from alerts themselves
    • The ability for an admin to re-send the self-validation email
    • The ability for an admin to manually validate an email address directly
    • Issues with self-validating SMS from Android
  • Support for Exchange Web Services to receive email from Exchange directly, not just via IMAP/POP
  • A bunch of bug fixes for issues reported here on Thwack, including:
    • Buttons appearing wonky when viewing alerts in gmail
    • Notes being truncated to 60 characters and some general misbehavior with longer text
    • Increased session timeout from 10 to 60 minutes
    • Some issues with creating recurring calendar entries

 

Whew!

 

You can download the Alert Central upgrade directly from right here. There's more info about what's new and links to the release notes over in the Alert Central Upgrade Info document here on Thwack.

 

If you are new to Alert Central, you can download it from the Alert Central website.

 

Alert Central is powered by YOUR feedback here on Thwack, so if you've got anything to say, or have any questions, let us know! Check out the Alert Central forum, respond to this post, or contact me (colby) directly.

FreeDNS.png

Today we made it official - our library of handy DNS and email troubleshooting tools on DNSstuff.com are now free! You can use the tools a few times before being prompted to create an account, with an account you can create your own favorites and arrange the tools as you see fit. The DNSstuff Toolbox ("Professional Toolset") has dozens of tools for testing/retrieving DNS records and troubleshooting configuration issues that might affect network connectivity, routing, DNS and email; the Mail Server Test Center consolidates a few critical tests that might affect email into one place.

 

Questions DNSstuff Tools can Help you Answer

 

Is it down for everyone or just me? Can people reach my site? Is my site slower from different parts of the internet?

DNSstuff is a great third party external resource that lets you troubleshoot remote connectivity issues without calling all your friends to ask them to try accessing your site and tell you how slow it is. In addition to the usual traceroute, our vector trace tool lets you drop in an IP and we'll connect to it from 2 different locations to see whether there's any issues with your site that might differ based on someone's location. While you're there, dig into the details about any hop to find out more about who owns it and where it is.

 

DNSvector1.pngDNSvector2.png

 

 

Are my DNS records good? Has my DNS change propagated through the internet yet? I am setting up a new site or working with a new customer and how can I tell their DNS is configured correctly? Is there a problem with one of my listed DNS servers not responding?

 

Nothing's more fun than making changes or making a new site live and having someone ask you every 5 minutes why the traffic isn't coming yet, or why people are still seeing the old site. Okay, maybe more fun than that is when you have random reports of lookup failures or communication errors to your site and have no idea why - but all of the network connectivity itself seems good. You can use DNSstuff to do tests against your DNS records and their listed DNS servers to make sure all of the listings are correct AND help you identify common problems or pitfalls with DNS that might lead to these types of failures.

 

Our DNSreport tool will run a number of tests against your domain and subsequent listed DNS servers to tell you if there's a problem anywhere down the line. You can also use the ISP cached DNS lookup tool to see if ISPs have cached your records or are showing the new ones.

 

DNSreport.pngDNScachedISP.png

 

Where is this IP address physically located? How does traffic route through different geographic areas to get to one of my sites? Could a wonky route that sends traffic overseas be making my site particularly slow?

DNSstuff has a few IP location and visual traceroute tools that help you identify where a certain IP is located and what routes traffic might be taking to get to an IP address. The vector trace (sample above) is handy to see how different locations might be reaching you. Our standard traceroute performs the exact same lookup but from a single host. IP information will look up all of the info related to an IP (including location across several location tools) and there's always standard IPWhois to do a simple reverse ownership lookup.

DNSipinfo.pngDNSipwhois.png


Am I listed on any spam blacklists?

There's a bunch of spam blacklists out there and getting listed means email from you might not reach someone using a blacklist to prevent unwanted email. DNSstuff has a blacklist lookup tool that checks your IP address against those lists.  You can also check a domain against a URI blacklist tool in case your domain has appeared in spam and that's causing you to be blocked, too. If you are blocked, we've lined up the contact info so you can react accordingly.

 

DNSspambl.pngDNSuribl.png

 

Who owns this site and which whois service should I use to find out if I need to contact them? I changed my whois record info, but how do I know it's done?

There are several online whois tools, but usually in order to use them, you have to at least figure out which whois service to go to. We've hooked them all up to one lookup tool so you can drop in an IP or domain name and see who it belongs to (and all the other useful whois info). We also have snapshots so you can see when whois information last changed and what changed.


DNSwhois1.pngDNSwhois2.png

 

People are getting bounce messages from our email, but not all of the time, and I can't figure out what's going on - help?

If you've got multiple servers configured as your MX, mail can find its way to all of them - but only sometimes, which makes problems pretty hard to track down unless you check every server. Mail Server Test Center runs a bunch of tests that check basic connectivity, records, SPF, and other info against ALL of your MX records.

 

DNSmstc.png

 

And More!!

There's a ton of other tools on DNSstuff - everything from a Speed Test to DKIM Key/SPF record checking to a CIDR calculator and even social media search. DNSstuff is also here on Thwack, over in the DNSstuff space where our product management and sometimes even dev team checks in to see how we can help.

 

NOTE: All paying DNSstuff customers continue to have access to the same tools and information that they did before, and access to create an online case with our support team up to the expiration date of your account. As your account expires, your access to any tools won't change and you'll experience no interruption in service.

Log & Event Manager's latest Release Candidate (v5.6) is now available on everyone's customer portal. As always, this release candidate is supported in production, so if you have any questions or issues post them here on our Log & Event Manager Release Candidate forum or contact our awesome support team. Here's the two big items that will make it worth your while to upgrade.

 

 

Let's take all those rules... and move them to categories!

http://i.qkme.me/3uhjia.jpg

LEM ships with a lot of rules out of the box... a lot. The problem we've had is that they are hard to find - if you're looking for "rules that help with PCI Compliance" you have to cross reference a separate list. Well, no longer! We have categorized and tagged everything into different areas that are oriented much more toward how you're actually using rules.

LEM-Rules-v56.png

 

  • To use a rule template, click the "Gear" to the right of the rule you want to use, then choose "Clone". (This isn't new, but where it's located is!)
  • Your rules will appear in the "Custom" category (underneath "Compliance" in the screenshot above).
  • We've hidden some of the advanced refinements (searching by date, user who modified the rule last, etc) to the "Advanced Search" area on the top left.
  • There are some rules that monitor common traffic patterns that are enabled by default. They will show up as Created By "Unknown" and be enabled (easy to spot if you click on All Rules).

 

When editing a rule...

LEM-Rules-v56-Edit.png

  • Click the link next to "Tags" to the top left to add categories and tags. The "Tags" dialog will open where you can add your own or check off existing tags.
  • To delete a category/tag from the list, remove the category/tag from any rules that are using that category/tag.
  • You can create your own categories/tags on the fly, too, just add them when you're editing the rule, and check them off. You can always go back and add those tags to new rules.

 

Some cool things you can do with categories and tags:

  • Tag rules you're using for compliance so that they don't get inadvertently disabled.
  • Categorize rules used for production, lab, and other environments so that you know how rules are used.
  • Tag "in progress" or "testing" rules so that you can find rules that you're working on developing.
  • Categorize rules for different departments or teams (sort of like how we have Security and IT Operations) so that each team can find their relevant rules quickly.

 

 

 

Next up....

Improvements to Database Storage Infrastructure, Archiving, and nDepth

We've done some revamping of our database storage backend in order to satisfy some internal and external requirements. What does this mean to you?

  • Your data will be migrated to a new format during upgrade. You'll want to take a snapshot (upgrade will remind you) or archive before starting the migration since it's a one-time operation.
  • You can resize your appliance beyond the previous 1TB limit (the next most common barrier is 2.2TB, based on virtual infrastructure capabilities to address a single disk).
  • Database archiving is not a full archive each time anymore, only what's new. The first time after you upgrade when your archive runs (use "archiveconfig" in the CMC to check) it will effectively be a "full" archive, though, so be prepared.

 

What you'll see during migration:

  • The console will show the progress of your migration, along with an estimate for completion. For most people, it should be a few days to a week. People with larger databases could experience longer migrations (and if you've got a full 1TB database it could be a couple weeks).
  • While the migration is taking place, you'll be able to search new data and migrated data, but not older data. Most recent data is prioritized, as is processing real-time data vs. migrating.
  • The Database Maintenance Report will also show you the status and historical info (so you can tell how far back you're migrated in days, rather than percentage/numbers).

 

With nDepth search, you'll see a new cool feature that draws the charts dynamically as your data is returned, rather than waiting until the end. You'll also see that you can now sort results by oldest to newest or newest to oldest, rather than always having them in the same order.

 

How to Upgrade

  1. Go to the customer portal. Scroll to the "Release Candidates" area. Click on the LEM v5.6 RC. Download the "Upgrade" bit (zip file).  If it's not in the Release Candidates section, you can go to License Management, then under your LEM license you'll see a "Release Candidates" area.
  2. Extract the zip file contents. Put the "TriGeo" and "Upgrade" folders in a windows file share that the appliance can access. (It's big, so you probably don't want to pull over the WAN or anything)
  3. Log in to the CMC via SSH, your hardware appliance, or your virtual appliance console's "Advanced Configuration".
  4. Run the "upgrade" command
  5. Answer the prompts.
  6. That's it! (It'll take about 5 minutes to run through everything, then migration starts in the background)

 

Get help, ask questions, tell us what we missed!

Come to the Log & Event Manager Release Candidate forum and tell us what you think. There's also an additional thread over there with some more technical details: LEM 5.6 Release Candidate Notes & Info (RC2 - Available Now).

Filter Blog

By date: By tag: