Upstage AI Lab 2기 [Day058] 수강생 TODO #02 - Pytorch Data Loading

Upstage AI Lab 2기

2024년 3월 6일 (수) Day_058

수강생 TODO

 

목표 :

- Pytorch를 이용하여 다양한 딥러닝 모델 구현에 필요한 tensor 연산을 자유자재로 할 수 있다.

- 기본적인 딥러닝 모델들을 직접 구현하여 학습 및 추론할 수 있다.

- 다양한 라이브러리에 구현되어 있는 딥러닝 모델을 손쉽게 사용할 수 있고, 구현 과정에서 생기는 다양한 에러들을 해결할 수 있다.

 

수강생 미션 : 직접 데이터를 선정해서 학습이 가능한 형태로 변경함으로써 Pytorch의 data loading에 대해 이해한다.

 

수강생 세부 todo

1. UCI repository에서 학습하고 싶은 tabular 데이터를 선택. (Adult, Heart disease 등)

2. 해당 tabular data를 학습이 가능한 형태로 변경

• 2-1 preprocessing.ipynb 파일 생성
• 2-2 UCI repository에서 다운로드 받은 csv파일을 pandas를 이용해 로딩
• 2-3 categorical data가 있는 경우 label encoder 및 one-hot encoding으로 변경
• 2-4 학습 데이터 (X)와 정답 (Y)를 분리
• 2-5 train-test 비율을 6:4로 분리
• 2-6 test set에서 5:5로 validation-test로 분리
• 2-7 training, validation, test set의 X,y 데이터를 npy 또는 npz 파일로 저장

3. Dataset

• 3-1 앞서 진행했던 MNIST 코드를 참고하여 training 파일 새로 생성
• 3-2 저장된 npy 또는 npz 파일을 로딩 후, Pytorch를 이용해 CustomDataset 클래스를 생성
• 3-3 이후 MNIST코드와 같이 DataLoader를 통해 학습 진행

 

4. Training

• 4-1 기존 코드와 같이 training을 진행하되, 데이터가 변경됨에 따라 생긴 error들을 해결

-참고자료

UCI repository: https://archive.ics.uci.edu/
one-hot encoder (preprocessing): https://pandas.pydata.org/docs/reference/api/pandas.get_dummies.html
label-encoder: https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html

train-test split: https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html

Custom Dataset: https://pytorch.org/tutorials/beginner/basics/data_tutorial.html

 

 

---

선택한 학습 데이터 

The CIFAR-10 dataset

https://www.cs.toronto.edu/~kriz/cifar.html

 

    self.layer = nn.Sequential(
        # input shape (batch, 3, 32, 32)
        nn.Conv2d(in_channels=3, out_channels = 16, kernel_size = 5),
        # output shape (batch, 16, 28, 28)
        nn.ReLU(),

        nn.Conv2d(in_channels = 16, out_channels = 32, kernel_size = 4, stride = 2),
        # output shape (batch, 32, 13, 13)
        nn.ReLU(),

        nn.Flatten(),
        nn.Linear(32*13*13, self.num_classes),
        nn.LogSoftmax(dim=1)
    )

Valid max accuracy :  0.6172

 

 

    self.layer = nn.Sequential(
        # input shape (batch, 3, 32, 32)
        nn.Conv2d(in_channels=3, out_channels = 16, kernel_size = 5),
        # output shape (batch, 16, 28, 28)
        nn.LeakyReLU(),

        nn.Conv2d(in_channels = 16, out_channels = 32, kernel_size = 5),
        # output shape (batch, 32, 24, 24)
        nn.LeakyReLU(),

        nn.Conv2d(in_channels = 32, out_channels = 64, kernel_size = 5),
        # output shape (batch, 64, 20, 20)
        nn.LeakyReLU(),

        nn.MaxPool2d(kernel_size = 2),
        # output shape (batch, 64, 10, 10)

        nn.Flatten(),
        nn.Linear(64*10*10, 4*10*10),
        nn.Linear(4*10*10, self.num_classes),
        nn.LogSoftmax(dim=1)
    )

Valid max accuracy :  0.6531

 

    self.layer = nn.Sequential(
        # input shape (batch, 3, 32, 32)
        nn.Conv2d(in_channels=3, out_channels = 16, kernel_size = 5),
        # output shape (batch, 16, 28, 28)
        nn.LeakyReLU(),

        nn.Conv2d(in_channels = 16, out_channels = 32, kernel_size = 5),
        # output shape (batch, 32, 24, 24)
        nn.LeakyReLU(),

        nn.MaxPool2d(kernel_size = 2),
        # output shape (batch, 32, 12, 12)
        
        nn.Conv2d(in_channels = 32, out_channels = 64, kernel_size = 4, stride= 2),
        # output shape (batch, 64, 5, 5)
        nn.LeakyReLU(),
        
        nn.Flatten(),
        nn.Linear(64*5*5, self.num_classes),
        nn.LogSoftmax(dim=1)
    )

Valid max accuracy :  0.6793

 

 

 

 

    self.layer = nn.Sequential(
        # input shape (batch, 3, 32, 32)
        nn.Conv2d(in_channels=3, out_channels = 16, kernel_size = 5),
        # output shape (batch, 16, 28, 28)
        nn.LeakyReLU(),

        nn.Conv2d(in_channels = 16, out_channels = 32, kernel_size = 5),
        # output shape (batch, 32, 24, 24)
        nn.LeakyReLU(),

        nn.AvgPool2d(kernel_size = 2),
        # output shape (batch, 32, 12, 12)
        
        nn.Conv2d(in_channels = 32, out_channels = 64, kernel_size = 4, stride= 2),
        # output shape (batch, 64, 5, 5)
        nn.LeakyReLU(),
        
        nn.Flatten(),
        nn.Linear(64*5*5, self.num_classes),
        nn.LogSoftmax(dim=1)
    )

 

Valid max accuracy :  0.6653

 

 

batch_size = 128 -> batch_size = 256

 

Valid max accuracy :  0.6566

 

 

batch_size = 256 -> batch_size = 512

Valid max accuracy :  0.6463

 

 

 

    self.layer = nn.Sequential(
        # input shape (batch, 3, 32, 32)
        nn.Conv2d(in_channels=3, out_channels = 16, kernel_size = 3),
        # output shape (batch, 16, 30, 30)
        nn.LeakyReLU(),

        nn.Conv2d(in_channels = 16, out_channels = 32, kernel_size = 5),
        # output shape (batch, 32, 26, 26)
        nn.LeakyReLU(),

        nn.AvgPool2d(kernel_size = 2),
        # output shape (batch, 32, 13, 13)
        
        nn.Conv2d(in_channels = 32, out_channels = 64, kernel_size = 3, stride= 2),
        # output shape (batch, 64, 6, 6)
        nn.LeakyReLU(),
        
        nn.Flatten(),
        nn.Linear(64*6*6, self.num_classes),
        nn.LogSoftmax(dim=1)
    )

Valid max accuracy :  0.6415

 

 

    self.layer = nn.Sequential(
        # input shape (batch, 3, 32, 32)
        nn.Conv2d(in_channels=3, out_channels = 8, kernel_size = 3, padding=1),
        # output shape (batch, 8, 32, 32)
        nn.LeakyReLU(),

        nn.Conv2d(in_channels = 8, out_channels = 16, kernel_size = 3),
        # output shape (batch, 16, 30, 30)
        nn.LeakyReLU(),

        nn.Conv2d(in_channels = 16, out_channels = 32, kernel_size = 3),
        # output shape (batch, 32, 28, 28)
        nn.LeakyReLU(),
        
        nn.Conv2d(in_channels = 32, out_channels = 64, kernel_size = 3),
        # output shape (batch, 64, 26, 26)
        nn.LeakyReLU(),
        

        nn.Conv2d(in_channels = 64, out_channels = 128, kernel_size = 3),
        # output shape (batch, 128, 24, 24)
        nn.LeakyReLU(),

        nn.AvgPool2d(kernel_size = 2),
        # output shape (batch, 128, 12, 12)

        
        nn.Conv2d(in_channels = 128, out_channels = 256, kernel_size = 3),
        # output shape (batch, 256, 10, 10)
        nn.LeakyReLU(),

        nn.Flatten(),
        nn.Linear(256*10*10, self.num_classes),
        nn.LogSoftmax(dim=1)
    )

Valid max accuracy :  0.6503

 

 

    self.layer = nn.Sequential(
        # input shape (batch, 3, 32, 32)
        nn.Conv2d(in_channels=3, out_channels = 8, kernel_size = 3, padding=1),
        # output shape (batch, 8, 32, 32)
        nn.LeakyReLU(),

        nn.Conv2d(in_channels = 8, out_channels = 16, kernel_size = 3),
        # output shape (batch, 16, 30, 30)
        nn.LeakyReLU(),

        nn.Conv2d(in_channels = 16, out_channels = 32, kernel_size = 3),
        # output shape (batch, 32, 28, 28)
        nn.LeakyReLU(),
        
        nn.Conv2d(in_channels = 32, out_channels = 64, kernel_size = 3),
        # output shape (batch, 64, 26, 26)
        nn.LeakyReLU(),
        
        nn.Conv2d(in_channels = 64, out_channels = 128, kernel_size = 3),
        # output shape (batch, 128, 24, 24)
        nn.LeakyReLU(),
        
        nn.Conv2d(in_channels = 128, out_channels = 256, kernel_size = 3),
        # output shape (batch, 256, 22, 22)
        nn.LeakyReLU(),
        
        nn.Conv2d(in_channels = 256, out_channels = 512, kernel_size = 3),
        # output shape (batch, 512, 20, 20)
        nn.LeakyReLU(),

        nn.AvgPool2d(kernel_size = 2),
        # output shape (batch, 512, 10, 10)
        
        nn.Flatten(),
        nn.Linear(512*10*10, self.num_classes),
        nn.LogSoftmax(dim=1)
    )

 

Valid max accuracy :  0.672

 

dropout 없이 했더니 완전히 오버피팅 됨

 

 

 

dropout 추가

dropout_ratio = 0.5

    self.layer = nn.Sequential(
        # input shape (batch, 3, 32, 32)
        nn.Conv2d(in_channels=3, out_channels = 8, kernel_size = 3, padding=1),
        # output shape (batch, 8, 32, 32)
        nn.LeakyReLU(),

        nn.Conv2d(in_channels = 8, out_channels = 16, kernel_size = 3),
        # output shape (batch, 16, 30, 30)
        nn.LeakyReLU(),

        nn.Conv2d(in_channels = 16, out_channels = 32, kernel_size = 3),
        # output shape (batch, 32, 28, 28)
        nn.LeakyReLU(),

        nn.Dropout(dropout_ratio),
        
        nn.Conv2d(in_channels = 32, out_channels = 64, kernel_size = 3),
        # output shape (batch, 64, 26, 26)
        nn.LeakyReLU(),
        
        nn.Conv2d(in_channels = 64, out_channels = 128, kernel_size = 3),
        # output shape (batch, 128, 24, 24)
        nn.LeakyReLU(),
        
        nn.Dropout(dropout_ratio),

        nn.Conv2d(in_channels = 128, out_channels = 256, kernel_size = 3),
        # output shape (batch, 256, 22, 22)
        nn.LeakyReLU(),
        
        nn.Conv2d(in_channels = 256, out_channels = 512, kernel_size = 3),
        # output shape (batch, 512, 20, 20)
        nn.LeakyReLU(),

        nn.AvgPool2d(kernel_size = 2),
        # output shape (batch, 512, 10, 10)
        
        nn.Flatten(),
        nn.Linear(512*10*10, self.num_classes),
        nn.LogSoftmax(dim=1)
    )

Valid max accuracy :  0.6835

 

 

 

dropout_ratio = 0.2

Valid max accuracy :  0.6664

 

dropout_ratio = 0.3

Valid max accuracy :  0.6865

 

 

---