In this tutorial, we will create a Swin transformer model and use it for visual recognition .
from fastai.vision.all import *
from transcv.swin import *
path = untar_data(URLs.PETS)/'images'
dls = ImageDataLoaders.from_name_func(
path, get_image_files(path), valid_pct=0.2,
label_func=lambda x: x[0].isupper(), item_tfms=Resize(224), bs = 32)
100.00% [811712512/811706944 00:22<00:00]
dls.show_batch()
swint_ob = SwinT('swin_base_patch4_window7_224', pretrained = True, num_classes = dls.c)
swin_model = swint_ob.get_model()
Downloading: "https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window7_224_22kto1k.pth" to /root/.cache/torch/hub/checkpoints/swin_base_patch4_window7_224_22kto1k.pth
learn = Learner(dls, swin_model, loss_func = CrossEntropyLossFlat(), metrics = accuracy).to_fp16()
learn.summary()
Sequential (Input shape: 32)
============================================================================
Layer (type) Output Shape Param # Trainable
============================================================================
32 x 128 x 56 x 56
Conv2d 6272 True
LayerNorm 256 True
Dropout
LayerNorm 256 True
____________________________________________________________________________
32 x 49 x 384
Linear 49536 True
Dropout
Linear 16512 True
Dropout
Softmax
Identity
LayerNorm 256 True
____________________________________________________________________________
32 x 3136 x 512
Linear 66048 True
GELU
____________________________________________________________________________
32 x 3136 x 128
Linear 65664 True
Dropout
LayerNorm 256 True
____________________________________________________________________________
32 x 49 x 384
Linear 49536 True
Dropout
Linear 16512 True
Dropout
Softmax
DropPath
LayerNorm 256 True
____________________________________________________________________________
32 x 3136 x 512
Linear 66048 True
GELU
____________________________________________________________________________
32 x 3136 x 128
Linear 65664 True
Dropout
____________________________________________________________________________
32 x 784 x 256
Linear 131072 True
LayerNorm 1024 True
LayerNorm 512 True
____________________________________________________________________________
32 x 49 x 768
Linear 197376 True
Dropout
Linear 65792 True
Dropout
Softmax
DropPath
LayerNorm 512 True
____________________________________________________________________________
32 x 784 x 1024
Linear 263168 True
GELU
____________________________________________________________________________
32 x 784 x 256
Linear 262400 True
Dropout
LayerNorm 512 True
____________________________________________________________________________
32 x 49 x 768
Linear 197376 True
Dropout
Linear 65792 True
Dropout
Softmax
DropPath
LayerNorm 512 True
____________________________________________________________________________
32 x 784 x 1024
Linear 263168 True
GELU
____________________________________________________________________________
32 x 784 x 256
Linear 262400 True
Dropout
____________________________________________________________________________
32 x 196 x 512
Linear 524288 True
LayerNorm 2048 True
LayerNorm 1024 True
____________________________________________________________________________
32 x 49 x 1536
Linear 787968 True
Dropout
Linear 262656 True
Dropout
Softmax
DropPath
LayerNorm 1024 True
____________________________________________________________________________
32 x 196 x 2048
Linear 1050624 True
GELU
____________________________________________________________________________
32 x 196 x 512
Linear 1049088 True
Dropout
LayerNorm 1024 True
____________________________________________________________________________
32 x 49 x 1536
Linear 787968 True
Dropout
Linear 262656 True
Dropout
Softmax
DropPath
LayerNorm 1024 True
____________________________________________________________________________
32 x 196 x 2048
Linear 1050624 True
GELU
____________________________________________________________________________
32 x 196 x 512
Linear 1049088 True
Dropout
LayerNorm 1024 True
____________________________________________________________________________
32 x 49 x 1536
Linear 787968 True
Dropout
Linear 262656 True
Dropout
Softmax
DropPath
LayerNorm 1024 True
____________________________________________________________________________
32 x 196 x 2048
Linear 1050624 True
GELU
____________________________________________________________________________
32 x 196 x 512
Linear 1049088 True
Dropout
LayerNorm 1024 True
____________________________________________________________________________
32 x 49 x 1536
Linear 787968 True
Dropout
Linear 262656 True
Dropout
Softmax
DropPath
LayerNorm 1024 True
____________________________________________________________________________
32 x 196 x 2048
Linear 1050624 True
GELU
____________________________________________________________________________
32 x 196 x 512
Linear 1049088 True
Dropout
LayerNorm 1024 True
____________________________________________________________________________
32 x 49 x 1536
Linear 787968 True
Dropout
Linear 262656 True
Dropout
Softmax
DropPath
LayerNorm 1024 True
____________________________________________________________________________
32 x 196 x 2048
Linear 1050624 True
GELU
____________________________________________________________________________
32 x 196 x 512
Linear 1049088 True
Dropout
LayerNorm 1024 True
____________________________________________________________________________
32 x 49 x 1536
Linear 787968 True
Dropout
Linear 262656 True
Dropout
Softmax
DropPath
LayerNorm 1024 True
____________________________________________________________________________
32 x 196 x 2048
Linear 1050624 True
GELU
____________________________________________________________________________
32 x 196 x 512
Linear 1049088 True
Dropout
LayerNorm 1024 True
____________________________________________________________________________
32 x 49 x 1536
Linear 787968 True
Dropout
Linear 262656 True
Dropout
Softmax
DropPath
LayerNorm 1024 True
____________________________________________________________________________
32 x 196 x 2048
Linear 1050624 True
GELU
____________________________________________________________________________
32 x 196 x 512
Linear 1049088 True
Dropout
LayerNorm 1024 True
____________________________________________________________________________
32 x 49 x 1536
Linear 787968 True
Dropout
Linear 262656 True
Dropout
Softmax
DropPath
LayerNorm 1024 True
____________________________________________________________________________
32 x 196 x 2048
Linear 1050624 True
GELU
____________________________________________________________________________
32 x 196 x 512
Linear 1049088 True
Dropout
LayerNorm 1024 True
____________________________________________________________________________
32 x 49 x 1536
Linear 787968 True
Dropout
Linear 262656 True
Dropout
Softmax
DropPath
LayerNorm 1024 True
____________________________________________________________________________
32 x 196 x 2048
Linear 1050624 True
GELU
____________________________________________________________________________
32 x 196 x 512
Linear 1049088 True
Dropout
LayerNorm 1024 True
____________________________________________________________________________
32 x 49 x 1536
Linear 787968 True
Dropout
Linear 262656 True
Dropout
Softmax
DropPath
LayerNorm 1024 True
____________________________________________________________________________
32 x 196 x 2048
Linear 1050624 True
GELU
____________________________________________________________________________
32 x 196 x 512
Linear 1049088 True
Dropout
LayerNorm 1024 True
____________________________________________________________________________
32 x 49 x 1536
Linear 787968 True
Dropout
Linear 262656 True
Dropout
Softmax
DropPath
LayerNorm 1024 True
____________________________________________________________________________
32 x 196 x 2048
Linear 1050624 True
GELU
____________________________________________________________________________
32 x 196 x 512
Linear 1049088 True
Dropout
LayerNorm 1024 True
____________________________________________________________________________
32 x 49 x 1536
Linear 787968 True
Dropout
Linear 262656 True
Dropout
Softmax
DropPath
LayerNorm 1024 True
____________________________________________________________________________
32 x 196 x 2048
Linear 1050624 True
GELU
____________________________________________________________________________
32 x 196 x 512
Linear 1049088 True
Dropout
LayerNorm 1024 True
____________________________________________________________________________
32 x 49 x 1536
Linear 787968 True
Dropout
Linear 262656 True
Dropout
Softmax
DropPath
LayerNorm 1024 True
____________________________________________________________________________
32 x 196 x 2048
Linear 1050624 True
GELU
____________________________________________________________________________
32 x 196 x 512
Linear 1049088 True
Dropout
LayerNorm 1024 True
____________________________________________________________________________
32 x 49 x 1536
Linear 787968 True
Dropout
Linear 262656 True
Dropout
Softmax
DropPath
LayerNorm 1024 True
____________________________________________________________________________
32 x 196 x 2048
Linear 1050624 True
GELU
____________________________________________________________________________
32 x 196 x 512
Linear 1049088 True
Dropout
LayerNorm 1024 True
____________________________________________________________________________
32 x 49 x 1536
Linear 787968 True
Dropout
Linear 262656 True
Dropout
Softmax
DropPath
LayerNorm 1024 True
____________________________________________________________________________
32 x 196 x 2048
Linear 1050624 True
GELU
____________________________________________________________________________
32 x 196 x 512
Linear 1049088 True
Dropout
LayerNorm 1024 True
____________________________________________________________________________
32 x 49 x 1536
Linear 787968 True
Dropout
Linear 262656 True
Dropout
Softmax
DropPath
LayerNorm 1024 True
____________________________________________________________________________
32 x 196 x 2048
Linear 1050624 True
GELU
____________________________________________________________________________
32 x 196 x 512
Linear 1049088 True
Dropout
LayerNorm 1024 True
____________________________________________________________________________
32 x 49 x 1536
Linear 787968 True
Dropout
Linear 262656 True
Dropout
Softmax
DropPath
LayerNorm 1024 True
____________________________________________________________________________
32 x 196 x 2048
Linear 1050624 True
GELU
____________________________________________________________________________
32 x 196 x 512
Linear 1049088 True
Dropout
LayerNorm 1024 True
____________________________________________________________________________
32 x 49 x 1536
Linear 787968 True
Dropout
Linear 262656 True
Dropout
Softmax
DropPath
LayerNorm 1024 True
____________________________________________________________________________
32 x 196 x 2048
Linear 1050624 True
GELU
____________________________________________________________________________
32 x 196 x 512
Linear 1049088 True
Dropout
____________________________________________________________________________
32 x 49 x 1024
Linear 2097152 True
LayerNorm 4096 True
LayerNorm 2048 True
____________________________________________________________________________
32 x 49 x 3072
Linear 3148800 True
Dropout
Linear 1049600 True
Dropout
Softmax
DropPath
LayerNorm 2048 True
____________________________________________________________________________
32 x 49 x 4096
Linear 4198400 True
GELU
____________________________________________________________________________
32 x 49 x 1024
Linear 4195328 True
Dropout
LayerNorm 2048 True
____________________________________________________________________________
32 x 49 x 3072
Linear 3148800 True
Dropout
Linear 1049600 True
Dropout
Softmax
DropPath
LayerNorm 2048 True
____________________________________________________________________________
32 x 49 x 4096
Linear 4198400 True
GELU
____________________________________________________________________________
32 x 49 x 1024
Linear 4195328 True
Dropout
LayerNorm 2048 True
____________________________________________________________________________
[]
AdaptiveAvgPool1d
____________________________________________________________________________
32 x 2
Linear 2050 True
____________________________________________________________________________
Total params: 86,681,730
Total trainable params: 86,681,730
Total non-trainable params: 0
Optimizer used: <function Adam at 0x7ffb3a23edd0>
Loss function: FlattenedLoss of CrossEntropyLoss()
Callbacks:
- TrainEvalCallback
- MixedPrecision
- Recorder
- ProgressCallback
learn.lr_find()
SuggestedLRs(valley=1.2022644114040304e-05)
learn.fit_one_cycle(1, 1e-4)
| epoch | train_loss | valid_loss | accuracy | time |
|---|---|---|---|---|
| 0 | 0.022827 | 0.003018 | 0.998647 | 37:26 |