Summary:
When saving models, it can appear as available for loading before saving has finished.  This can cause race conditions, particularly on network file systems.

This changes saving code to write to a temporary filename in the same directory and then atomically rename to the target filename.  This is a common technique that relies on the atomicity of `os.rename`.

Reviewed By: ashwinb, samialsheikh

Differential Revision: D14495989

fbshipit-source-id: 2e44dd8e747331b319ef46132ddeff604618a5dd

Summary:
It is convenient to run with a set of precomputed proposals for a
larger set (e.g., coco_trainval2014) and then do experiments using annotations
for a smaller subset of the data.

It is possible to do this filtering only if a config flag is set, but it seems
harmless enough to do it always.

Reviewed By: rbgirshick

Differential Revision: D14106953

fbshipit-source-id: f726425ee5b6b22fcb711e3ec479b302ea09e837

Reviewed By: rbgirshick

Differential Revision: D13546589

fbshipit-source-id: eb0a442a4620fa4d82a57671cabb58cdb5ed02df

Summary:
Per t34270005 we're changing all publicly hosted detectron assets to a new low-cost CDN-backed S3 bucket. The migration requires rewriting all instances of `https://s3-us-west-2.amazonaws.com/detectron` to `https://dl.fbaipublicfiles.com/detectron`.

This giant codemod does it.

```
cd fbcode

rg -l s3-us-west-2.amazonaws.com/detectron > ~/filelist.txt

for F in $(cat ~/filelist.txt);
  sed -i '' -e 's/s3-us-west-2\.amazonaws\.com\/detectron/dl.fbaipublicfiles.com\/detectron/g' $F
done
```

Reviewed By: jma127

Differential Revision: D13538734

fbshipit-source-id: 4fe5c797770b0d9f675cee53dbcb6b52857fa45d

Summary:
Codemod generated with clangr shard mode, 50 files per diff,
clangr code(ndim()->dim()): diffusion/FBS/browse/master/fbcode/caffe2/caffe2/fb/codemods/TensorMethodRename.cpp

Reviewed By: ezyang

Differential Revision: D12935918

fbshipit-source-id: 4ce24a3debcd471b3649fa71f890f7b24bb8678a

Reviewed By: rbgirshick

Differential Revision: D10857403

fbshipit-source-id: 5ea53ece3418e8f6dcb20df3d3521d661a7380a3

Summary:
Codemod generated with clangr shard mode, 50 files per diff,
clangr code(size->numel): diffusion/FBS/browse/master/fbcode/caffe2/caffe2/fb/codemods/TensorMethodRename.cpp

Reviewed By: ezyang

Differential Revision: D12833487

fbshipit-source-id: 5dec5a64f91912620f3a5e48a2cc571be675c5fd

Reviewed By: ashwinb

Differential Revision: D10496498

fbshipit-source-id: eb12fe573ec3270172e27c2fdb39a70fc92d8d99

Reviewed By: rbgirshick

Differential Revision: D10508713

fbshipit-source-id: c410d88cc64864087dacd04a73639952cb063df0

Reviewed By: cdelahousse

Differential Revision: D10452557

fbshipit-source-id: d64c10e7b0c6ecce2db50c6470ae282ac76db9a0

Reviewed By: rbgirshick

Differential Revision: D10452054

fbshipit-source-id: ee5c909844c034dc2e4a1ee995f3c86e1f353694

Reviewed By: rbgirshick

Differential Revision: D10433948

fbshipit-source-id: 0d4dbc7f532e13c559da65c403aa06238f612e25

Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/12456

codemod with 'Yes to all'
codemod -d . --extensions h,cc,cpp,cu,py,proto,pbtxt,pb.txt,config cuda_gpu_id device_id

Overload TextFormat::ParseFromString to do string replace when parsing from protobuf format

Reviewed By: Yangqing

Differential Revision: D10240535

fbshipit-source-id: 5e6992bec961214be8dbe26f16f5794154a22b25

Reviewed By: ashwinb

Differential Revision: D10223705

fbshipit-source-id: bb46ffea8185abb7abf6df7273c2941c6d95f8b1

Reviewed By: ashwinb

Differential Revision: D10223289

fbshipit-source-id: 1e6fb55f77e01a2f653fd614d13d1d4a7ceeccea

Reviewed By: ashwinb

Differential Revision: D10218305

fbshipit-source-id: d9d285959b1e4f5c0d38f080d6ce8c260b1959e0

Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/12232

Original commit changeset: fca91fea58b7

This adds proper modifications to the DeviceType <->DeviceOption conversion code added in D10033396

Reviewed By: jerryzh168

Differential Revision: D10132473

fbshipit-source-id: 801ef777e2950982cb47b48051b1471a0a91e64b

Reviewed By: rbgirshick

Differential Revision: D10131443

fbshipit-source-id: ee6df3147b94711eecbe558bcaf30d021976b593

Differential Revision:
D10123245

Original commit changeset: d83da8e00a12

fbshipit-source-id: fca91fea58b7df208edc2e218a1d514f9821ec7b

Reviewed By: rbgirshick

Differential Revision: D9691247

fbshipit-source-id: 7a9c1a33698be6907dede589965f8f4f4d337f93

Summary:
A couple small issues pointed out by Ross.

 - Treating bytes as strings with a subprocess' output :/
 - Controling precision of logged floats using a workaround instead of
   FLOAT_REPR which doesn't work with newer versions of Python3.

Reviewed By: rbgirshick

Differential Revision: D9724292

fbshipit-source-id: a6aa1730f25df5d165291dc30b9350a9fff6fca6

Summary:
There were two "bugs" associated with loading pickle files:

The first is easy: these files should have been `open()`ed as binary, but
weren't.

The second is slightly nuanced. The default encoding used wwhile unpickling
is 7-bit (ASCII.) However, the blobs are arbitrary 8-bit bytes which don't
agree. The absolute correct way to do this is to use `encoding="bytes"` and
then interpret the blob names either as ASCII, or better, as unicode utf-8. A
reasonable fix, however, is to treat it the encoding as 8-bit latin1 (which
agrees with the first 256 characters of Unicode anyway.)

As part of this, I also centralized all pickling operations into
`detectron.utils.io`.

This /still/ does not change the build to Python3, but I believe it is ready
now.

Reviewed By: rbgirshick

Differential Revision: D9689294

fbshipit-source-id: add1f2d784fe196df27b20e65e35922536d11a3c

Summary:
This is a first step towards python3 compatibility. Specifically, it
tackles:

 - `cPickle`: uses six.moves
 - `Queue`: uses six.moves
 - `urllib2`: uses six.moves and changes `urllib2.urlopen` to `urllib.request.urlopen`

Also, fundamentally it changes the types of all config "byte" types to
"string" types. Those configurations aren't un-encoded byte streams but very
specifically ascii (or unicode encoded) strings which are specified and
consumed by human eyes.

Reviewed By: rbgirshick

Differential Revision: D9662024

fbshipit-source-id: b8372f685b57ec4260ae881a2f8bb7967f337b10

Summary:
Original commit changeset: f5614a5d2607

D9986213 is causing Multifeed Aggregator a [huge performance different](https://our.intern.facebook.com/intern/ads/analyze_canary/412951953278781781/) and is blocking aggregator push since last Friday night: https://fburl.com/feedtools/b6izvwjz
We need to land this revert ASAP to unblock aggregator push.

Reviewed By: orionr

Differential Revision: D10123245

fbshipit-source-id: d83da8e00a1250f5d09811a0a587c127e377aab2

Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/12022

codemod -d . --extensions h,cc,cpp,cu,py,proto,pbtxt,pb.txt,config cuda_gpu_id device_id

codemod with 'Yes to all'

Reviewed By: orionr

Differential Revision: D9986213

fbshipit-source-id: f5614a5d26078817aee8caf79a494abfd6a95ff1

Summary:
Soft-NMS is present in test.py, but it is missing in test_retinanet.py.
Pull Request resolved: https://github.com/facebookresearch/Detectron/pull/670

Reviewed By: rbgirshick

Differential Revision: D10035825

Pulled By: ir413

fbshipit-source-id: c49ac7e595b32d2ac757815df4add1c644a34027

Summary:
Fixing error:
TypeError: Layout of the output array img is incompatible with cv::Mat (step[ndims-1] != elemsize or step[1] != elemsize*nchannels)

in vis_class(..) and vis_bbox(..) methods when calling vis_one_image_opencv(..).
Pull Request resolved: https://github.com/facebookresearch/Detectron/pull/555

Reviewed By: rbgirshick

Differential Revision: D10027949

Pulled By: ir413

fbshipit-source-id: e34b8b881ffe231984f270910f778c5abe79e489

Summary:
Detectron code is inconsistent about when it uses (1, 1, 1, 1) as the
regression weights vs. `cfg.MODEL.BBOX_REG_WEIGHTS` (which has a default
value of (10, 10, 5, 5)). This change allows the caller to make the call
(heh!) about what to use.

Reviewed By: rbgirshick

Differential Revision: D9981875

fbshipit-source-id: 8798408bd2592486eb1a2e40ccb70b9bedfdbf46

Reviewed By: rbgirshick

Differential Revision: D9940697

fbshipit-source-id: 60a7d76a6ac2ef89a58f6c3dca8adce27ffd74f7

Reviewed By: rbgirshick

Differential Revision: D9818825

fbshipit-source-id: a1f2182a7760568987100aa7402aca9327dee952

Summary:
Negative labels were never set if `len(bg_inds) <= num_bg`. This
condition happens when there aren't as many negative anchor samples compared to the
typical batch size (256). Since detection scales are typically ~800, there are
plenty of negative samples and this bug is likely never hit. In any case, it
is certainly a bug which should be fixed!

Reviewed By: rbgirshick

Differential Revision: D9647703

fbshipit-source-id: fada9f4f2e0a0b5c1619940328c505e73f9a1c7e

Summary:
Fix for #620 & #293.
Pull Request resolved: https://github.com/facebookresearch/Detectron/pull/624

Reviewed By: ir413

Differential Revision: D9574052

Pulled By: rbgirshick

fbshipit-source-id: f44cea71e5cf6e613d319322db62a60ecb814a75

Summary: Renamed should_stop to has_stopped in roi_loader.

Reviewed By: rbgirshick

Differential Revision: D9596732

fbshipit-source-id: b34a5b20d215fa7423d96c9a8a9cf271b500b530

Summary:
Pull Request resolved: https://github.com/facebookresearch/Detectron/pull/638

Potential fix for training stuck caused by data loader failure.

Reviewed By: rbgirshick

Differential Revision: D9513621

fbshipit-source-id: 123974eac83f40ef2f582a90fedea790fdc442d1

Summary: Make data loader threads daemonic so the main process can exit when an exception is raised.

Reviewed By: ashwinb, newstzpz

Differential Revision: D9478059

fbshipit-source-id: 00bfaab51295b63c2f95ac1be99ae56a0ccf9d94

Summary:
In some cases, eg training from scratch with a high LR or w/o activation normalization, we saw a "Negative Areas Found" exception. The RPN proposal generation code filters proposals that have a height or width that are too small. By default we use a `min_size` of 0. In a distant other part of the code, it computes the area of the proposals in order to determine FPN scale assignment. An issue arises because the `+ 1` in the `x2 - x1 + 1` width (and height) computation is not scale invariant. It turns out that the filter based on `min_size` was performed in the scaled input coordinates while the FPN scale assignment areas are computed using the unscaled, original image size. Thus a `min_size` of 0 can result in proposals with a negative area as measured wrt the original image. Uuuugh :P.

This diff addresses the issue by filtering based on the area in the unscaled input image and also using a generous (though still small) margin of 1 pixel for `min_size`.

Reviewed By: KaimingHe

Differential Revision: D8773216

fbshipit-source-id: b4ffbc5b6a831176b2656810edea3d3d4e52d687

Reviewed By: ir413

Differential Revision: D9360044

fbshipit-source-id: c44c392d065a2cb53db3bbd1d0f83b8ecd80a430

Summary:
Based on orionr's work

 - [x] Solve [the problem about GenerateProposals](https://github.com/facebookresearch/Detectron/pull/334#issuecomment-378764468)
 - [x] Use the existing [ResizeNearest](https://github.com/caffe2/caffe2/blob/master/caffe2/operators/resize_op.cc#L57) layer instead of UpsampleNearest. ResizeNearest has cpu implementation and neon optimization
 - [x] Make it work (with https://github.com/pytorch/pytorch/pull/7091)

With this PR, FPN is supported in cooperation with https://github.com/pytorch/pytorch/pull/7091. I have verified that it works on `e2e_faster_rcnn_R-50-FPN_1x.yaml`
Pull Request resolved: https://github.com/facebookresearch/Detectron/pull/372

Reviewed By: newstzpz

Differential Revision: D9213242

Pulled By: rbgirshick

fbshipit-source-id: 8fc7b77e6cbf08adaafd760505dd760df59bfd79

Reviewed By: rbgirshick

Differential Revision: D9133758

fbshipit-source-id: a698a637b294b7f2aef4c181cb47505187be3935

Summary:
AFAIK, this is what the rest of the codebase does. I don't want to
see logger.debug() messages show up during normal runs.

Differential Revision: D9071531

fbshipit-source-id: 4ac5ef57ee47bf958e35d723c379f695554f0ef0