Project

General

Profile

Bug #25383

Slow scp transfer speeds using ssh cipher chacha20-poly1305@openssh.com

Added by Lucas Burson 3 months ago. Updated 2 months ago.

Status:
RC: Needs Status Update
Priority:
No priority
Category:
Services
Target version:
Start date:
07/31/2017
Due date:
% Done:

0%

Seen in:
Hardware Configuration:

Board: Supermicro X10SDV-4C+-TLN4F-O Mini-ITX Intel Xeon D-1518 DDR4 Motherboard and CPU
Memory: 2x Crucial - 16GB (1 x 16GB) DDR4-2133 Memory
Network controller: Intel i350-AM2 Gigabit Ethernet

Blanket Approval:
No
ChangeLog Required:
No
Needs QA:
Yes
QA Status:
Not Tested

Description

On a 1Gbps network I see 30MB/s scp transfers when using cipher , which is extremely slow. Using a different cipher results in the expected ~112MB/s.
I'm not sure if this is a FreeNAS issue or BSD's openssh.
I see this as a significant issue since, for many users, ssh transfers will default to and users will migrate to "faster" NAS software.

See this forum post for my original investigation:
https://forums.freenas.org/index.php?threads/very-slow-scp-transfer-speeds-to-baremetal-freenas-but-fast-to-vm-in-freenas-bhyve.56390/

Workaround:
  • Remove from the openssh cipher list.
  • Removing that cipher increases transfer rate to 50MB/s. Use these systctls to get the FULL 1Gbps rates:
    sysctl kern.ipc.maxsockbuf=33554432
    sysctl net.inet.tcp.recvbuf_auto=1
    sysctl net.inet.tcp.sendbuf_auto=1
    sysctl net.inet.tcp.sendbuf_max=33554432
    sysctl net.inet.tcp.recvbuf_max=33554432
    sysctl net.inet.tcp.recvspace=4194304
    sysctl net.inet.tcp.sendspace=4194304
    
    

History

#1 Updated by Dru Lavigne 3 months ago

  • Assignee changed from Release Council to Alexander Motin

#2 Updated by Alexander Motin 3 months ago

  • Status changed from Unscreened to Screened
  • Target version set to 11.1

Have you tried to look on scp CPU usage in `top` while using different ciphers? I suspect that on your CPU chacha20 is slower then AES because it is not accelerated by hardware (AES accelerated by AES-NI CPU instructions), while CPU frequency is not very high to do it in software.

#3 Updated by Alexander Motin 3 months ago

  • Status changed from Screened to Waiting For Feedback

#4 Updated by Lucas Burson 3 months ago

Alexander Motin wrote:

Have you tried to look on scp CPU usage in `top` while using different ciphers? I suspect that on your CPU chacha20 is slower then AES because it is not accelerated by hardware (AES accelerated by AES-NI CPU instructions), while CPU frequency is not very high to do it in software.

CPU usage of chacha20 is 15/0 (usr/sys), and AES is 9/15. The kernel is doing more of the work with AES, is this an indication of hardware acceleration?

#5 Updated by Alexander Motin 3 months ago

scp is single-threaded application. I asked about percent of CPU time used by it specifically to identify the bottleneck. Higher total CPU usage can be just a result of higher throughput.

#6 Updated by Lucas Burson 3 months ago

Okay. Let me know if you'd like other metrics, I'm happy to help. I have a Fedora26 VM running on the FreeNAS node (in bhyve) which performs at 112MB/s but I didn't think to check the cipher. I'll verify the cipher used report back. That should help indicate if the cipher is slow from hardware or compiled bits.

#7 Updated by Alexander Motin 3 months ago

As I have told, I do like to see CPU time usage by the scp/ssh thread. Check for used cipher would also be useful. If it ends up in confirmation that chacha20 is just slower by itself, then I tend to close this with "Behaves correctly".

#8 Updated by Lucas Burson 3 months ago

Copying the file to FreeNAS using causes sshd to use 100% CPU, limiting the transfer to 30MB/s.
Copying the file to Fedora 26 VM (running on FreeNAS), again using , is at 112MB/s.

#9 Updated by Lucas Burson 3 months ago

Metrics from FreeBSD 11.1 VM.

Cipher     | top sshd | usr%/sys% | rate
chacha20   |    100   | 15/23     |  84MB/s
sha128-crt |     94   |  6/30     | 100MB/s

#10 Updated by Lucas Burson 3 months ago

Hi, do you need more feedback on this?

#11 Updated by Dru Lavigne 2 months ago

  • Status changed from Waiting For Feedback to RC: Needs Status Update

Sasha: please indicate if more info is needed.

Also available in: Atom PDF