So if I wanted to show the full path specifically in EventInfo "created". Would I then have to edit the connector source to get this done?
The actual IIS FTP log will show created /FTP/FTPtest.zip which is what I want.
It's easier for us to track that way, I can put the issue in as well but by dropping it in the support queue it follows a more 'normal' procedure. Either way, a log sample and what you'd like to see there (screenshot or text and which fields/data you're interested in) will make it pretty straightforward on our end.
FYI support was kind enough to send me the new revision #5 for the iisftp connector and revision #6 for the globalscape connector. However I still wasn't seeing the path/filename in nDepth. A closer look at the .xml file revealed the issue.
We need cs-uri-stem which reveals the path/filename. This is verified in the Globalscape log. According to the allFiledsLocations (which I'm assuming is a typo) is number 6. However the FastFormatter is giving us $5 which is cs-method. A quick edit and re-upload of the connector and it now shows in nDepth correctly. I have not looked at the iisftp connector in depth to see if it exhibits the same issue.
The iisftp connector was the same. I edited the FastFormatter to $8 and the filename is now showing correctly in nDepth. I also added the cs-username ($4) to see the user doing the uploading.
"Closing data connection for "$3". "$8" action successful"
So, you all knew this was coming, but I'll say it anyway... building your own connectors is not currently supported and can cause problems with both existing connectors and your new connectors (and your agent performance, and....). If you do this, please do not reuse readerNames and IDs that are in use from existing connectors or there is NO WAY to guarantee which one will actually be ran and you could find your work erased on a connector update AND your agents running the wrong connectors.
We have in our (disclaimer of no timeline guaranteed) roadmap to provide you with a tool you can use to properly build connectors the same way (or similar) we do internally, so you could properly build and distribute your own.
We've also considered providing somewhat generic connectors in the current product framework, but haven't quite figured out a very good way to do this. Either all of your data would go to a single generic event and be very taxing on the correlation engine to alert from (because of all the additional pattern matching), or you'd need to be able to provide some context to get them to the right event (which kind of ends up meaning a connector as above), or you'd need to ensure your files were written in some kind of format that matched our taxonomy to get them mapped.
Another possibility that builds on one or both of those is to provide a path between the original log storage and the normalized store, so you could have the log store collect 100% but then choose some log lines to raise as events that need real-time correlation. (Otherwise, you can use a scheduled search, for example, to get some historical data as emails if it occurs.)
Thanks nicole pauls for the insight into what you guys have been thinking of, it's always nice to hear even if it isn't fully baked yet!
I personally wouldn't want to go with any option that would have any significant negative impact on the performance of LEM.
The crux of the problem and the issue I was trying to help solve with my feature request HERE is that there are a lot of logs out there as you have suggested. The SolarWinds team doesn't have nearly the man power to create connectors for all of the things that people are going to want to have. As the product becomes more popular (which I hope it does) there will only be more requests to have new connectors created and of course more connectors means maintenance for those connectors as things change and this could easily be a scalability problem.
By having some way that the community can help build and support these connectors you gain a huge extension of the SolarWinds work force already doing this.
Thanks for sharing the info! Its good to know there are things being discussed and thought of. Most of us will be excited to see them in some release notes soon!
I know there have been some requests for a generic syslog connector. That may also lead to a generic plaintext log file connector, but I don't know when or if either of those might happen.
As for more immediate advice...
First, I'm presuming you have a LEM agent on the device that has the text file?
DISCLAIMER: Solarwinds will almost certainly NOT support connectors you create yourself.
That being said, we've had some success with custom connectors that we've created in house, and if you're feeling adventurous and have a little time on your hands, it might be worthwhile to give it a try.
All of the connectors are essentially just XML definitions of what LEM should look at and for. You can see them all in one of the connector upgrade packages. Just unzip it and navigate down to the folder with all the XML files. From there you can learn a whole lot about how they work.
I would suggest that maybe the best place to start would be with the DefaultReaderConfiguration line in the IIS.xml connector. That should give you a sense of how LEM connects to a plaintext log file. From there, any connector that collects syslog data should be a decent example of how to parse the log file. My first was a customization of the McAfee email gateway, and that one was fairly straightforward (as long as you're comfortable with regular expressions, and if you're not, here's a great chance to learn!).
When you're ready to load your new connector, use the same folder structure and process as the LEM connector upgrade, but make sure that the only connector in the bottom folder is your new one (or updated one). That way LEM will only update that connector and not all of them.
Hopefully this may be of some help. Good luck!
Hmm run across this one? I got my new connector installed but doesn't want to start. Not sure if you have seen this in creating yours or not. Not sure if its something in my FastPattern causing this?
Weird. A reboot and reupload of my custom connector and its working now. Now I just have to tweak the FastFields..etc. Very cool indeed. Thanks for the info.
So I'm assuming the DefaultReaderConfiguration is where I plug in all the information for the connector itself and the FastToolID is where all the regular expressions go. Or the events from the log file I want to show up in LEM?
Yeah, it's worth submitting it and seeing what happens.
Not all requests end up being fulfilled, probably for a couple reasons. First, there are a lot of different kinds of logs out there, and not enough time to make connectors for them all. They have to prioritize the ones that seem likely to get the most use (read: the most requested) so some more obscure ones may not make the cut. But if enough people want it, you may see it released.
Also, they presumably have a lot stricter standards for putting out good connectors than when I'm just trying to hack something together. If you look at the connectors they've developed, they're designed to parse and properly account for pretty much any event that might appear in a log source, not just the most common ones. It takes a lot of work--not to mention good documentation that isn't always available from the vendors--to make a connector that can reliably recognize, parse, and normalize all those events. Whereas if I'm making a connector, I can prioritize the events I'm most likely to see and not be as worried if something slips through the cracks.
Disclaimer the second: I don't work for Solarwinds or have any real insight into how their connector development process works. I just know that they've made a whole lot of connectors that work extremely well over a huge range of products and devices, and that has to have been a huge amount of work.
Not all requests end up being fulfilled, probably for a couple reasons. First, there are a lot of different kinds of logs out there, and not enough time to make connectors for them all. They have to prioritize the ones that seem likely to get the most use (read: the most requested) so some more obscure ones may not make the cut.
It's for exactly this reason that I think my feature request is important. By providing functionality within the product that allows customers to create their own connectors (via some type of dev studio) and share them with the community it will help get more connectors into circulation faster.
By providing functionality within the product that allows customers to create their own connectors (via some type of dev studio) and share them with the community it will help get more connectors into circulation faster.
Now that would be awesome!
Exactly! That's what the feature request I submitted is for. Having SolarWinds maintain and support connectors is good but why not get the community in on the action also in a way that is supported within the product. It's much like application monitoring templates are already handled for SolarWinds SAM.
You have the ability to set LEM to store RAW log data and this includes Syslog data so if your systems in question can send the data via Syslog to LEM then you can capture and store that data for searching through later.
Hope this helps!
SolarWinds solutions are rooted in our deep connection to our user base in the THWACK® online community. More than 150,000 members are here to solve problems, share technology and best practices, and directly contribute to our product development process.