This discussion has been locked. The information referenced herein may be inaccurate due to age, software updates, or external references.
You can no longer post new replies to this discussion. If you have a similar question you can start a new discussion in this forum.

JSch (and other software) can't connect to Serv-U 15.3.2

Since upgrading to Serv-U 15.3.2, I have many users who cannot connect anymore.

In my case, the similarity with all the cases is that they are using an application that uses the very popular JSch SFTP library within it to connect to external SFTP servers to upload/download files.

This has worked fine for over 10+ years but none of these users can now connect to Serv-U at all, which is causing major problems.

I originally discussed the problem with in a separate thread as he was having issues with some users and we thought it was key related initially, but it is not. I have created this specific thread for the issue as many users of Serv-U 15.3.2 are affected by this issue and will probably be Googling for it.

Solarwinds have released an FAQ and acknoledge this issue in 15.3.2 which can be seen here. This also affects Maverick Legacy Client and Cisco Unified Backup, as well as some older OpenSSH clients.

In summary, the cause is that some client software passes its "name" and version number to Serv-U in a format that isn't straictly compliant with the SFTP RFC, mainly because these libraries do not pass the invisible CR (carriage return) symbol to the end of their name and version number. From what I have observed, this makes Serv-U just continually wait at the point the connection is opened and then the connection times out. Therefore, zero connections can now be made from these clients or any clients/software that uses libraries such as JSch.

Whilst I understand that the RFC compliance is useful, in this case it literally stops software that has worked for 10+ years from making any connections, ever.

In my opinion, because Serv-U has alloed these connections (like most other SFTP servers) since it was created (decades ago), it needs to have backward compatability for the systems that integrate with it.

I would like to respond to each suggestion in the Solarwinds KB to demonstrate why there needs to be a long term solution..


Responses to KB suggested solutions

Suggestion 1: Reach out to your application team to add a CR symbol in your Java-Based client code and ensure that the program is RFC compliant.

Response 1: In 99% of cases this is not possible. Automation software and long established applications use the latest JSch library and it cannot be changed as it is an integrated part of the application.


Suggestion 2. Use a different application that is RFC compliant

Response 2: For the same reasons as Respose 1, most of the time these libraries are integrated into software and have been for 10+ years


Suggestion 3. Rollback to the previous version of Serv-U either by reverting to your Serv-U server snapshot backup, or by following this article.

Response 3: This may be possible as a temporary fix but 1) it is messy due to the Server Identity changes in 15.3.2 and 2) It is not a long term solution, the servers will eventually need to be upgraded. If you are stuck and urgently need to roll back, the article is here but I am a little skeptical it will work due to the Server Identity changes made in 15.3.2 which are detailed toward the end of that article.


Impact

I've already seen others having this issue and can observe how hundreds of users using all different systems will not be able to connect or use Serv-U on 15.3.2. These systems have connected to Serv-U for 10+ years and they cannot just stop - many are automated processes and custom software that users cannot control.

, , have resported on Thwack that their own users are having the same issue, feel free to share here if you have any other observations or thoughts.


The long term solution

There needs to be a perminent backward compatability released as a hotfix for 15.3.2 and then rolled into future versions to allow any clients that use this old name/version formatting to continue to work with Serv-U, the same as it has for 10+ years.



  • Unfortunately, we will never be able to upgrade to the new version as a significant amount of our 5000+ trading partners use JSch and other legacy clients unless a fix is in place. 

  • Just for completeness and fairness, you have to say, that only JSCH 0.1.53 and 0.1.54 are affected. Newer clients are fixed.

  • SW support told me the devs are working on it. However, reading that article they released sounds like I have to roll back and never update my MFT instances, unless I can get the client to update - which is not feasible in a lot of cases. Quite a predicament, but I am leaning towards exploring other options/products. 

  • Step 1. Documentation - 34 Days after the Release
    SFTP connection not established for legacy Java clients (solarwinds.com)

    Sorry, too much joy citing this...

    RFC 4253              SSH Transport Layer Protocol          January 2006

    4.2 ... Implementers who wish to maintain

    compatibility with older, undocumented versions of this protocol may

    want to process the identification string without expecting the

    presence of the carriage return character for reasons described in

    Section 5 of this document.

    ...
    5. Compatibility With Old SSH Versions

    As stated earlier, the 'protoversion' specified for this protocol is

    "2.0".  Earlier versions of this protocol have not been formally

    documented, but it is widely known that they use 'protoversion' of

    "1.x" (e.g., "1.5" or "1.3").  At the time of this writing, many

    implementations of SSH are utilizing protocol version 2.0, but it is

    known that there are still devices using the previous versions.

    During the transition period, it is important to be able to work in a

    way that is compatible with the installed SSH clients and servers

    that use the older version of the protocol.

  • the problme(s) is that the JSCH lib is likely part of larger software packages that manage business transactions adding FTP client functionality, and for some, will not be trivial to up the larger software packages. This is further reinforced by the cited RFC addressing backward compatibility; and perhaps advanced notice of the change (my personal biggest peeve)

  • 0.1.54 is the final version of JSch, later versions are a non-official fork of the project. This is why so many pieces of business software have 0.1.54 bundled inside, in a lot of cases it cannot even be replaced so the software will just break.

    From the server end, you cannot tell users to change their software after it has worked forever - we no control over user software and sometimes even the end user doesnt.

    I understand what you are saying, but practically, it will just break thousands of business processes if a permenant option isn't added to Serv-U to allow this (even just at a domain level), just like it has forever.

  • Solarwinds have released a further KB article on this: SFTP connection not established for legacy Java clients

    This mentions that they are working on a "Buddy Drop" for 15.3.2 to allow the old clients to work so you will need to request this from Support.

    However, this is for the "transition period" for people to "upgrade their infrastructure". There is a VERY important point Solarwinds need to understand here. Serv-U customers cannot control the end user client software. Whilst I appreciate it is good for it to be RFC compliant, Serv-U has allowed these clients forever and stopping them now that they are integrated into systems and software will break so many systems and processes.

    The only option is to have a setting in Serv-U on a Domain level in "Limits & Settings" to allow "Legecy Java Clients" or "Allow non-RFC Version info". This way Serv-U can be RFC compliant but the old sofrware can be allowed on a per-domain basis if it causes problems with processes that have been established since Serv-U was created.

    There cannot have a situation where Solarwinds customers cannot upgrade Serv-U in the future because of this issue because future updates could fix vulnerabilities and MUST be installed.

    please can you confirm the longer term solution?

  • Friday, Buddy Drop Hot Fix appears to have worked, a few clients have taken their personal time to Test and/or Follow-up over the weekend. I was able to test one client immediately after the HF and heard that our SAP HR reports were again flowing.

    I take objection to SW saying the following
    "... For the transition period, SolarWinds provides a buddy drop for Serv-U 15.3.2 that ignores RFC compliance. ..."

    The above statement to me totally ignores the RFC which clearly states it's the developer's choice to maintain backward compatibility. I get the phrase in the RFC, "...transition period..." But this was written in 2006 and if there was a transition period, it's years past, and at some point due diligence by the general public along with the RFC must make observable actions that indicate there actually is a transition period in situ.

    AND, more ire, and pain can be directed to SW Devs for not including this in the Release Notes... I mean come-on; if the line termination LF vs CRLF is a "thing" that needs to be corrected to conservatively follow the RFC, then a minimum of Documentation sorely missing.

    p.s. ++Gratitude  & a few other for their hours of working on "my" and SolarWinds' issue


    Regards,
    JeffP...

  • you are correct, this should be noted in release notes. I will request the change. Thank you

  • Yes, just tried the Buddy Drop and it does fix the issue.

    Also, had another differernt user not able to connect prior to the fix and they were using Sun_SSH_2.4 client, just connecting from their Sun server natively with the sftp command line. The Buddy Drop fixed it for them too.

    This is a much bigger issue than Solarwinds realise so it needs a long term fix, not a stop-gap or temporary fix. It must be in all future versions, either as a global change, or a per-domain setting that can be turned on to allow for backward compatability.