JSch (and other software) can't connect to Serv-U 15.3.2

Since upgrading to Serv-U 15.3.2, I have many users who cannot connect anymore.

In my case, the similarity with all the cases is that they are using an application that uses the very popular JSch SFTP library within it to connect to external SFTP servers to upload/download files.

This has worked fine for over 10+ years but none of these users can now connect to Serv-U at all, which is causing major problems.

I originally discussed the problem with in a separate thread as he was having issues with some users and we thought it was key related initially, but it is not. I have created this specific thread for the issue as many users of Serv-U 15.3.2 are affected by this issue and will probably be Googling for it.

Solarwinds have released an FAQ and acknoledge this issue in 15.3.2 which can be seen here. This also affects Maverick Legacy Client and Cisco Unified Backup, as well as some older OpenSSH clients.

In summary, the cause is that some client software passes its "name" and version number to Serv-U in a format that isn't straictly compliant with the SFTP RFC, mainly because these libraries do not pass the invisible CR (carriage return) symbol to the end of their name and version number. From what I have observed, this makes Serv-U just continually wait at the point the connection is opened and then the connection times out. Therefore, zero connections can now be made from these clients or any clients/software that uses libraries such as JSch.

Whilst I understand that the RFC compliance is useful, in this case it literally stops software that has worked for 10+ years from making any connections, ever.

In my opinion, because Serv-U has alloed these connections (like most other SFTP servers) since it was created (decades ago), it needs to have backward compatability for the systems that integrate with it.

I would like to respond to each suggestion in the Solarwinds KB to demonstrate why there needs to be a long term solution..


Responses to KB suggested solutions

Suggestion 1: Reach out to your application team to add a CR symbol in your Java-Based client code and ensure that the program is RFC compliant.

Response 1: In 99% of cases this is not possible. Automation software and long established applications use the latest JSch library and it cannot be changed as it is an integrated part of the application.


Suggestion 2. Use a different application that is RFC compliant

Response 2: For the same reasons as Respose 1, most of the time these libraries are integrated into software and have been for 10+ years


Suggestion 3. Rollback to the previous version of Serv-U either by reverting to your Serv-U server snapshot backup, or by following this article.

Response 3: This may be possible as a temporary fix but 1) it is messy due to the Server Identity changes in 15.3.2 and 2) It is not a long term solution, the servers will eventually need to be upgraded. If you are stuck and urgently need to roll back, the article is here but I am a little skeptical it will work due to the Server Identity changes made in 15.3.2 which are detailed toward the end of that article.


Impact

I've already seen others having this issue and can observe how hundreds of users using all different systems will not be able to connect or use Serv-U on 15.3.2. These systems have connected to Serv-U for 10+ years and they cannot just stop - many are automated processes and custom software that users cannot control.

, , have resported on Thwack that their own users are having the same issue, feel free to share here if you have any other observations or thoughts.


The long term solution

There needs to be a perminent backward compatability released as a hotfix for 15.3.2 and then rolled into future versions to allow any clients that use this old name/version formatting to continue to work with Serv-U, the same as it has for 10+ years.



Parents
  • Solarwinds have released a further KB article on this: SFTP connection not established for legacy Java clients

    This mentions that they are working on a "Buddy Drop" for 15.3.2 to allow the old clients to work so you will need to request this from Support.

    However, this is for the "transition period" for people to "upgrade their infrastructure". There is a VERY important point Solarwinds need to understand here. Serv-U customers cannot control the end user client software. Whilst I appreciate it is good for it to be RFC compliant, Serv-U has allowed these clients forever and stopping them now that they are integrated into systems and software will break so many systems and processes.

    The only option is to have a setting in Serv-U on a Domain level in "Limits & Settings" to allow "Legecy Java Clients" or "Allow non-RFC Version info". This way Serv-U can be RFC compliant but the old sofrware can be allowed on a per-domain basis if it causes problems with processes that have been established since Serv-U was created.

    There cannot have a situation where Solarwinds customers cannot upgrade Serv-U in the future because of this issue because future updates could fix vulnerabilities and MUST be installed.

    please can you confirm the longer term solution?

Reply
  • Solarwinds have released a further KB article on this: SFTP connection not established for legacy Java clients

    This mentions that they are working on a "Buddy Drop" for 15.3.2 to allow the old clients to work so you will need to request this from Support.

    However, this is for the "transition period" for people to "upgrade their infrastructure". There is a VERY important point Solarwinds need to understand here. Serv-U customers cannot control the end user client software. Whilst I appreciate it is good for it to be RFC compliant, Serv-U has allowed these clients forever and stopping them now that they are integrated into systems and software will break so many systems and processes.

    The only option is to have a setting in Serv-U on a Domain level in "Limits & Settings" to allow "Legecy Java Clients" or "Allow non-RFC Version info". This way Serv-U can be RFC compliant but the old sofrware can be allowed on a per-domain basis if it causes problems with processes that have been established since Serv-U was created.

    There cannot have a situation where Solarwinds customers cannot upgrade Serv-U in the future because of this issue because future updates could fix vulnerabilities and MUST be installed.

    please can you confirm the longer term solution?

Children
  • Friday, Buddy Drop Hot Fix appears to have worked, a few clients have taken their personal time to Test and/or Follow-up over the weekend. I was able to test one client immediately after the HF and heard that our SAP HR reports were again flowing.

    I take objection to SW saying the following
    "... For the transition period, SolarWinds provides a buddy drop for Serv-U 15.3.2 that ignores RFC compliance. ..."

    The above statement to me totally ignores the RFC which clearly states it's the developer's choice to maintain backward compatibility. I get the phrase in the RFC, "...transition period..." But this was written in 2006 and if there was a transition period, it's years past, and at some point due diligence by the general public along with the RFC must make observable actions that indicate there actually is a transition period in situ.

    AND, more ire, and pain can be directed to SW Devs for not including this in the Release Notes... I mean come-on; if the line termination LF vs CRLF is a "thing" that needs to be corrected to conservatively follow the RFC, then a minimum of Documentation sorely missing.

    p.s. ++Gratitude  & a few other for their hours of working on "my" and SolarWinds' issue


    Regards,
    JeffP...

  • you are correct, this should be noted in release notes. I will request the change. Thank you

  • Yes, just tried the Buddy Drop and it does fix the issue.

    Also, had another differernt user not able to connect prior to the fix and they were using Sun_SSH_2.4 client, just connecting from their Sun server natively with the sftp command line. The Buddy Drop fixed it for them too.

    This is a much bigger issue than Solarwinds realise so it needs a long term fix, not a stop-gap or temporary fix. It must be in all future versions, either as a global change, or a per-domain setting that can be turned on to allow for backward compatability.

  • To request the fix / "Buddy Drop" please contact support and reference this link.

    When you do, please make support aware that this is a major problem needing a permanent fix, not a stop-gap/temporary fix, otherwise in future versions this fix will NOT be included.

    , , , ,

  • Thank you  I agree with your statement and will be entering the support case very soon.

  • major issue here too.  For example boomi ESB can't connect anymore after the upgrade to 15.3.2.  Is used by many of our customers to connect and we use it too internally for all our integrations.  Gone contact boomi trough our integrator but we gone have a lot of these issues with customers who use older SFTP library's.

    And we can't ignore new serv-u updates forever, especially with al security fixes in every version.

  • Thanks for the details , the fix should for Boomi ESB too. Please let us know on the thread as I imagine people will probably Google Boomi and this issue.

    When you contact them for the Buddy Drop fix, please reiterate with support that this fix needs to be in future versions so we dont all end up here again with the next update.

  • FormerMember
    0 FormerMember in reply to PieterGr

    Thanks for this thread and seconding the Boomi problem. I have a ticket open with Boomi support about this issue and it's good to see some useful information.

  • All, just a reminder that it appears, any client based on Unix type environment where strings are self-trimming and only need a char(10) \n new line, Will be impacted. The RFC indicates CR+LF (chars 13+10) characters, but in same paragraph describes how at the discretion of the developer may support legacy clients and accept the single char(10) character.  Ideally if SW wants to make this change, they should offer a way for us to set a per user rule/option/switch/parameter to enable the single char(10) character.

    SW Developers have taken a strict interpretation of the 2006 RFC saying that that the support for the single character is only for the transition period; and IMHO 2006 to 2023 without any movement on this topic by the greater community and SW, means that there simply isn't a functional transition period; well not one with community feedback and impact analysis...

    ++Thanks (can't do this enough) to Calc2014 for their efforts