JSch (and other software) can't connect to Serv-U 15.3.2

Since upgrading to Serv-U 15.3.2, I have many users who cannot connect anymore.

In my case, the similarity with all the cases is that they are using an application that uses the very popular JSch SFTP library within it to connect to external SFTP servers to upload/download files.

This has worked fine for over 10+ years but none of these users can now connect to Serv-U at all, which is causing major problems.

I originally discussed the problem with in a separate thread as he was having issues with some users and we thought it was key related initially, but it is not. I have created this specific thread for the issue as many users of Serv-U 15.3.2 are affected by this issue and will probably be Googling for it.

Solarwinds have released an FAQ and acknoledge this issue in 15.3.2 which can be seen here. This also affects Maverick Legacy Client and Cisco Unified Backup, as well as some older OpenSSH clients.

In summary, the cause is that some client software passes its "name" and version number to Serv-U in a format that isn't straictly compliant with the SFTP RFC, mainly because these libraries do not pass the invisible CR (carriage return) symbol to the end of their name and version number. From what I have observed, this makes Serv-U just continually wait at the point the connection is opened and then the connection times out. Therefore, zero connections can now be made from these clients or any clients/software that uses libraries such as JSch.

Whilst I understand that the RFC compliance is useful, in this case it literally stops software that has worked for 10+ years from making any connections, ever.

In my opinion, because Serv-U has alloed these connections (like most other SFTP servers) since it was created (decades ago), it needs to have backward compatability for the systems that integrate with it.

I would like to respond to each suggestion in the Solarwinds KB to demonstrate why there needs to be a long term solution..


Responses to KB suggested solutions

Suggestion 1: Reach out to your application team to add a CR symbol in your Java-Based client code and ensure that the program is RFC compliant.

Response 1: In 99% of cases this is not possible. Automation software and long established applications use the latest JSch library and it cannot be changed as it is an integrated part of the application.


Suggestion 2. Use a different application that is RFC compliant

Response 2: For the same reasons as Respose 1, most of the time these libraries are integrated into software and have been for 10+ years


Suggestion 3. Rollback to the previous version of Serv-U either by reverting to your Serv-U server snapshot backup, or by following this article.

Response 3: This may be possible as a temporary fix but 1) it is messy due to the Server Identity changes in 15.3.2 and 2) It is not a long term solution, the servers will eventually need to be upgraded. If you are stuck and urgently need to roll back, the article is here but I am a little skeptical it will work due to the Server Identity changes made in 15.3.2 which are detailed toward the end of that article.


Impact

I've already seen others having this issue and can observe how hundreds of users using all different systems will not be able to connect or use Serv-U on 15.3.2. These systems have connected to Serv-U for 10+ years and they cannot just stop - many are automated processes and custom software that users cannot control.

, , have resported on Thwack that their own users are having the same issue, feel free to share here if you have any other observations or thoughts.


The long term solution

There needs to be a perminent backward compatability released as a hotfix for 15.3.2 and then rolled into future versions to allow any clients that use this old name/version formatting to continue to work with Serv-U, the same as it has for 10+ years.



Parents
  • Serv-U 15.3.2 Hotfix is available and your work has been rewarded:

    ========================================
    SolarWinds® Serv-U® 15.3.2 Hotfix 1
    ========================================


    This SolarWinds hotfix addresses the following issues:
    • OpenSSL upgraded to 3.0.8
    • New Serv-U settings allowing non-RFC compliant SSH protocol version exchange
    • Security enhancement in SSH key pair authentication

    This hotfix is available in the customer portal for Serv-U Customers.

  • Thank you @maeh!

    On initial testing it seemed to work well. However, after installing on a couple of servers, we then had other systems not being able to connect via SFTP. These were not JSch users either, those worked ok it seemed.

    It looks like maybe a new bug has been introduced for some SFTP connections as they cannot connect and a blank log line is added with no description when these users connect..

    [02] Mon 20Feb23 11:23:02 - (012442) Connected to 123.123.123.123 (local address 192.168.1.123, port 22)
    [02] Mon 20Feb23 11:23:02 - (012442)
    [02] Mon 20Feb23 11:23:02 - (012442) Closed session
    [02] Mon 20Feb23 11:23:10 - (012444) Connected to 123.123.123.123 (local address 192.168.1.123, port 22)
    [02] Mon 20Feb23 11:23:10 - (012444)
    [02] Mon 20Feb23 11:23:10 - (012444) Closed session
    [02] Mon 20Feb23 11:23:12 - (012446) Connected to 123.123.123.123 (local address 192.168.1.123, port 22)
    [02] Mon 20Feb23 11:23:12 - (012446)
    [02] Mon 20Feb23 11:23:12 - (012446) Closed session

    If you enable detailed SFTP commands/replies in the logs you can see the user connecting and attempting to login. The first login failure is normal with WinSCP connecting as it is seeing what auth methods are available.

    As soon as the correct password is used to login, the empty log line is written and the connection is closed by Serv-U..

    [02] Mon 20Feb23 11:24:56 - (012442) Connected to 123.123.123.123 (local address 192.168.1.123, port 22)
    [31] Mon 20Feb23 11:24:56 - (012442) SSH2_MSG_KEXINIT
    [30] Mon 20Feb23 11:24:56 - (012442) SSH2_MSG_KEXINIT
    [30] Mon 20Feb23 11:24:56 - (012442) SSH2_MSG_KEXDH_INIT
    [31] Mon 20Feb23 11:24:56 - (012442) SSH2_MSG_NEWKEYS
    [30] Mon 20Feb23 11:24:56 - (012442) SSH2_MSG_NEWKEYS
    [30] Mon 20Feb23 11:24:56 - (012442) SSH2_MSG_USERAUTH_REQUEST: user: TestUsername; service: ssh-connection; type: none
    [31] Mon 20Feb23 11:24:56 - (012442) SSH2_MSG_USERAUTH_FAILURE: login failed
    [30] Mon 20Feb23 11:24:56 - (012442) SSH2_MSG_USERAUTH_REQUEST: user: TestUsername; service: ssh-connection; type: password
    [02] Mon 20Feb23 11:24:56 - (012442)
    [02] Mon 20Feb23 11:24:56 - (012442) Closed session

    I've had to roll back the hotfix and it instantly resolves this above issue (just copied back old files I backed up before the hotfix).

    @maeh have you had similar issues with the hotfix?

    Trying to replicate this has been difficult so far but I can see our users having the problem, I just need to understand the circumstances that trigger the bug. Will update here when I have more info.

    Reverting the hotfix to the Buddy Drop version fixes the issue.

    Please let me know is anyone else experiences this and is able to replicate it themsevles. Thanks!

  • Hello,

    I had to revert to 15.3.1 long time ago because of another problem (we have to implement Argon2-hashing for our Database-Users first), so I can't test the hotfix.
    Can you tell, if the fixed file from the BuddyDrop is replaced by the Hotfix or not? If not, it could be possible, that you have to revert to 15.3.2 without BuddyDrop first and after that install the Hotfix.

    But this is just a big suppostion;)

    best regards,
    Markus

  • Hi Markus, yes the hotfix replaces those files added by the buddy drop, so I reverted to the files from the Buddy Drop and it seems to be working ok.

    It seems that the Hotfix may actually cause more problems but its going to take a lot of testing to work out why.

  • I've disovered the bug.

    In Serv-U 15.3.2 HF1, the username for SFTP connections crash if the case of the username does not match exactly, Serv-U closes the connection and writes a blank line to the log with no details.

    This is significant problem if you have lots of users as many may have put a capital letter in the username and those connections instantly break.

    Reverting to 15.3.2 or 15.3.2 with the Buddy Drop fixes the issue.

    It is therefore a bug in the new 15.3.2 HF1 update.

  • Thank you for the information and troubleshooting once again @Calc2014. I was going to apply this hot fix this morning but after reading your messages I believe ill hold off for now. 

  • Good plan, I will let you know here when I have an update from support on this.

  • Hi Slight smile

    Any news on this topic? Any answers from the support?

    thanks and best regards,

    Markus

  • Prior to upgrading to 15.3.2.HF1 is there an easy way to identify connections where the case of the connecting username doesn't match the case of Serv-U user definition? 

  • Hi @maeh I expect this will be fixed in the next update and I'll post when released and I've tested.

Reply Children