Code

Add FullyShardedDataParallel (FSDP)#413

Code

Merged

myleott merged 51 commits intomasterfacebookresearch/fairscale:masterfrom fsdpfacebookresearch/fairscale:fsdpCopy head branch name to clipboardFeb 23, 2021

+2,399-30

Commits

Commits on Feb 1, 2021

• Add fairscale.utils.testing.DeviceAndTypeCheckModule

  myleott

  committed
  a7be0d2

• Add fairscale.utils.containers

  myleott

  committed
  e2ad716

• Add ShardParamsDataParallel

  myleott

  committed
  4d6a5c9

• [test]: skip autocast-needed tests on torch < 1.6 (#34)

  

  min-xu-ai

  authored and

  myleott

  committed
  dd57e30

• [mypy]: fairscale/utils/containers.py (#33)

  

  min-xu-ai

  authored and

  myleott

  committed
  80678fc

• [mypy]: fixed fairscale/utils/testing.py (#32)

  

  min-xu-ai

  authored and

  myleott

  committed
  5fb36d1

• More docs (#35)

  myleott

  committed
  774e130

• [mypy]: fixed all the mypy errors (#37)

  Show description for b35a28a
  

  min-xu-ai

  and

  anj-s

  authored
  b35a28a

• Sharded DDP: test cpu_offload arg (#40)

  Show description for 92c550b
  sshleifer

  authored
  92c550b

• Misc comments from @anj-s (#43)

  myleott

  authored
  5bb212f

Commits on Feb 2, 2021

• Only sync fp32_to_fp16 stream for the top-most (root) ShardParams wrapper (#42)

  Show description for cbd243e
  myleott

  authored
  cbd243e

• Fix streams test (#45)

  myleott

  authored
  8db0cf6

• move_grads_to_cpu defaults to same value as cpu_offload (#44)

  sshleifer

  authored
  bc7e337

• formatting change (#46)

  min-xu-ai

  authored
  a1b3924

Commits on Feb 3, 2021

• Test that backward hooks are registered (#49)

  Show description for e10df73
  sshleifer

  authored
  e10df73

• [test] add test for apply_to_tensors (#50)

  Show description for e139857
  min-xu-ai

  authored
  e139857

Commits on Feb 4, 2021

• Test save/load state_dict V2 (#51)

  sshleifer

  authored
  6f153b0

• Replace x.view(-1) with torch.flatten(x) (#59)

  myleott

  authored
  4114772

• Add more comments + docstrings (#58)

  myleott

  authored
  36b2d39

• Rearrange dtype and device change in post-backward hook (#61)

  myleott

  authored
  bc5190b

• Do reduce-scatter in a separate CUDA stream (#62)

  Show description for 8a5f81c
  myleott

  authored
  8a5f81c

Commits on Feb 5, 2021

• tests use spawn_for_all_world_sizes (#63)

  sshleifer

  authored
  72c1f63

Commits on Feb 6, 2021

• Fix state_dict bugs (#60)

  sshleifer

  authored
  f481877

Commits on Feb 7, 2021

• update comments to reflect where we are in stack (#69)

  sshleifer

  authored
  515411b

• [CI] use parameterized.expand to make each test faster (#68)

  sshleifer

  authored
  ec4e75e

• Fix delayed reduce_scatter test (#74)

  Show description for b1460d3
  myleott

  authored
  b1460d3

• add unit test pack/unpack kwargs (#65)

  Show description for 014ad05
  

  min-xu-ai

  and

  sshleifer

  authored
  014ad05

• Refactor param init and streams logic (#73)

  Show description for 0ca378b
  myleott

  authored
  0ca378b

Commits on Feb 8, 2021

• Add test for NoGrad mode

  myleott

  committed
  bebe7fd

• Add all_gather stream and disable reshard_after_forward on root instance (#75)

  myleott

  authored
  74b0223

Commits on Feb 9, 2021

• Leave buffers on self.compute_device (#67)

  sshleifer

  authored
  6797964

• Pad parameters inside: right before gather, scatter (#76)

  sshleifer

  authored
  0937594

Commits on Feb 10, 2021

• Add no_sync() context manager (#77)

  myleott

  authored
  93670cb

• rename

  sshleifer

  committed
  1d0bf73

Commits on Feb 11, 2021

• Slightly faster execution when world_size == 1 (#81)

  myleott

  authored
  5fc1f12

• merge new base (which is public/master) (#82)

  Show description for 3681242
  

  8 peopleauthored
  3681242

• Merge branch 'shard_params_ddp_base' into shard_params_ddp

  myleott

  committed
  dfada29

• Merge branch 'shard_params_ddp_base' into shard_params_ddp

  myleott

  committed
  9fb974b

Commits on Feb 12, 2021

• two small changes (#83)

  Show description for 7bd82d1
  min-xu-ai

  authored
  7bd82d1

Commits on Feb 14, 2021

• fixing uneven shard support and add uneven shard unit test (#80)

  Show description for d8f3349
  min-xu-ai

  authored
  d8f3349

Commits on Feb 15, 2021

• rename test file (#86)

  Show description for 366c38e
  min-xu-ai

  authored
  366c38e

Commits on Feb 19, 2021

• adding TrainingState enum for asserting purpose (#87)

  Show description for 87d28c9
  min-xu-ai

  authored
  87d28c9

• Misc (#88)

  Show description for 72b5967
  myleott

  authored
  72b5967

Commits on Feb 20, 2021

• Bugfix for backward stream (#91)

  myleott

  authored
  a0fe832

Commits on Feb 22, 2021

• Small fix to activation checkpointing + FSDP (#92)

  Show description for 2abf21f
  myleott

  authored
  2abf21f

• Bugfix (+ add test) for no_sync before first forward (#93)

  myleott

  authored
  977f459

• FSDP.clip_grad_norm (#89)

  sshleifer

  authored
  43d1f73

• Add ReduceScatterBucketer (#94)

  Show description for 77dc364
  myleott

  authored
  77dc364

• Merge branch 'oss-master' into shard_params_ddp

  myleott

  committed
  bf30e59

• Fix merge conflict

  myleott

  committed
  ec8bf9f

• Update docstring slightly

  myleott

  committed
  3f2b7f1