Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

data packet unmarshal binary: remove uneeded copy #338

Merged
merged 2 commits into from
Mar 1, 2020
Merged

Conversation

drakkan
Copy link
Collaborator

@drakkan drakkan commented Feb 15, 2020

No description provided.

@drakkan
Copy link
Collaborator Author

drakkan commented Feb 18, 2020

some interesting numbers here:

drakkan/sftpgo#69 (comment)

this patch give a 20% performance improvement for the user who posted the results. In my tests, on my laptop, the improvement is smaller

Copy link
Collaborator

@puellanivis puellanivis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. The []byte is made in `recvPacket' and then unused except for this copy.

@drakkan
Copy link
Collaborator Author

drakkan commented Feb 23, 2020

@puellanivis Thanks!

I wonder if we can improve this code too:

       bb, err := m.MarshalBinary()
       ...
        // Slide packet down 4 bytes to make room for length header.
        packet := append(bb, make([]byte, 4)...) // optimistically assume bb has capacity
        copy(packet[4:], bb)

I need to find some time to benchmark the different ways to prepend packet (fixed 4 bytes) to bb that could be bigger.

For what I can undestand reading pkg/sftp code there are no other obvious optimizations

@puellanivis
Copy link
Collaborator

prepending is known to be an expensive operation, which is why Go does not have a builtin primitive for it, that would parallel append.

Although, looking at it, it might be better to just allocate the 4 bytes at the head right from the start, and just skip over them. Then later we can inject the length into it.

@drakkan drakkan mentioned this pull request Feb 25, 2020
@drakkan
Copy link
Collaborator Author

drakkan commented Feb 26, 2020

the latest patch, applied on the top of my allocator branch, makes pkg/sftp upload performance very similar to my scp implementation.

I don't know why statvfs test fails on MacOS, it use sshFxpStatvfsPacket, my change doesn't modify that packet

@eikenb
Copy link
Member

eikenb commented Mar 1, 2020

Travis has made it impossible to trigger re-tests so I'll pull this down and check locally. The integration tests can be a bit overly fragile.

[edit] Wait... I dont' have a Mac, so I can't check it locally. :P Guess I'll push up a test branch to trigger Travis. Seriously thinking about switching to CircleCI.

@eikenb
Copy link
Member

eikenb commented Mar 1, 2020

I triggered 2 travis runs and they both came back fine. So just going to ignore the failure and merge this.

@eikenb
Copy link
Member

eikenb commented Mar 1, 2020

@drakkan Thanks for the PR!
@puellanivis Thanks for the review!

@eikenb eikenb merged commit 8f77623 into pkg:master Mar 1, 2020
@eikenb eikenb added this to the v1.11.1 milestone Mar 7, 2020
@eikenb eikenb modified the milestones: v1.11.1, v1.12.0 Jul 19, 2020
@drakkan drakkan deleted the copy branch November 26, 2020 08:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants