100,00 event limit not large enough...

I am getting close to 2k logs per second = 120k per minute or more.  The Historical search only allows a 100k limit over 1 minute...  Does this mean I will never be able to search all logs?  So far, that has been my experience.  Disappointing.

  • I guess, building queries would be the only way....?

  • I am in the same boat. I have spoken to two different support techs and neither had a solution. I am fine with bigger searches taking longer. Limiting it to 100k is infuriating. And we aren't even as big as you -- we get about 100k per 10 minutes. But what if I want to search for a day?

    One tech recommended using the report program, but it isn't really built for you to plug in a keyword or timeframe and go hunting. It's more for specific reports that you want to run consistently.

  • So, I was curious when you said that and went looking...this might be the solution: https://documentation.solarwinds.com/en/success_center/sem/content/admin_guide/2020_2/sem-create-a-search-query.htm

    If you do EventInfo = "*keyword*" as your search, it appears to get around the 100k limit.

    I'm glad you made this suggestion. I spoke to two support techs who couldn't tell me how to get around the limit. I was about to start shopping for other programs.

    [Edit] EventInfo will only search that box. If you just straight up put "keyword" (with the ") in the top box, you can search beyond the 100k limit. Or more accurately, the 100k limit will be applied to the number of logs with keyword in it and not the last 100k logs.

  • What type of information your are looking to retrieve?  I offset some daily routines by setup of reports to gather the data I need but it depend on the type of data you searching for.  I have used many programs each have  there own unique problems.  

    Prebuilt query are effective if there are very granular because 100K of event is not user friendly at all to a analysis.  

    As I mention I receive about 1-2K of events per minute put have no issues search week worth of logs for the information I need. The cap is there because 10GB memory for 100K of event means search a entire day will be over 100GB of memory which is not feasible.

  • This is an interesting topic.  Due to STIG's and 800-53 security controls I get a LOT of events per minute.  I'm interested in hearing more about dealing with many events and best ways to use SEM search and reporting features.  I worked with UX team on replacement for Crystal Reports.  What do you modify in your SEM to assist with audits?  Are there some tricks to queries you've found?

    Bill

  • The first way  I found to find reports that contain the dataset you want for example failed login attempts.

    Select the time range for the report

    with keywords from Crystal reports by finding a report that has the information you require. Run the report please note I have these reports scheduled to run every day for certain requirements.  I would select the short date range at first as I customize the report later with keywords. I recommend a computer with 32 GB ram and for running the report and good network connection for better performance.

    After the report is run you then Customize the report with keywords under the view tab select expert add variable keyword you want to search by.

    Here is a URL for custom Reports

    documentation.solarwinds.com/.../sem-create-custom-filtered-report.htm

    The second way using the HTML 5 interface is to layer filters for the required information.

    I start using a live filter for information considering this is constant piping in data this gives you a quasi-real-time look at rules without memory usage too much. 

    After narrowing down the filter rules you can covert the filter into a rule and search the still same 100K of data but filter to the data you're looking for.  

    Then you can search for the rule firing without having to type a lengthy search command. 

    Realistic 100K of events is more than needed if know what you're looking for.  You should know the location of the information you require some limited to the source via destination IP or tool alias.  If it is the main firewall for example is this broken down by ports, tcp/udp, possible url or IP address.  

    Even large enterprise program like Cisco Firepower has a limitation on searching  I have found.  I use SolarWinds SEM a summary of events compared to detailed events as which I will retrieve review the original log source that is still available for more information.

    If Storage is not proper I also recommend enabling at the connection level a copy of RAW and Parsed data for more details if required.