Ahmed Taha – Medium

Pinned

L2-CAF: A Neural Network Debugger

Every software engineer has used a debugger to debug his code. Yet, a neural network debugger… That’s news! This paper [1] proposes a debugger to debug and visualize attention in convolutional neural networks (CNNs). Before describing the CNN debugger, I want to highlight few attributes of program debuggers (e.g., gdb)…

Machine Learning

5 min read

May 16

Understanding the Effective Receptive Field in Deep Convolutional Neural Networks

In deep networks, a receptive field — or field of view — is the region in the input space that affects the features of a particular layer as shown in Fig.1. The receptive field is important for understanding and diagnosing a network’s performance. …

Machine Learning

5 min read

Understanding the Effective Receptive Field in Deep Convolutional Neural Networks

Apr 4

Understanding Transfer Learning for Medical Imaging

Transfer learning (a.k.a. ImageNet pre-training) is a common practice in deep learning where a pre-trained network is fine-tuned on a new dataset/task. This practice is implicitly justified by feature-reuse where features learned from ImageNet are beneficial to other datasets/tasks. This paper [1] evaluates this justification on medical images datasets. The…

Deep Learning

6 min read

Understanding Transfer Learning for Medical Imaging

Jan 24

Sharpness-Aware Minimization for Efficiently Improving Generalization

For training a deep network, picking the right optimizer has become an important design choice. Standard optimizers (e.g., SGD, Adam, etc.) seek a minimum on the loss curve. This minimum is sought without regard for the curvature, i.e., the 2nd degree derivative of the loss curve. A curvature denotes the…

Optimization

4 min read

Sharpness-Aware Minimization for Efficiently Improving Generalization

Dec 27, 2021

Feature Embedding Regularizers: SVMax & VICReg

What is more important, a deep network weights or its activations? Obviously, we can derive the network’s activation from its weights. Yet, deep networks are non-linear embedding functions; we want this non-linear embedding only. On top of this embedding, we either slap a linear classifier in a classification network or…

Deep Learning

7 min read

Feature Embedding Regularizers: SVMax & VICReg

Nov 3, 2021

IIRC: Incremental Implicitly-Refined Classification

While training a deep network on multiple tasks jointly is easy, training on multiple tasks sequentially is challenging. This challenge is addressed in various literature: Lifelong Learning, Incremental Learning, Continual Learning, and Never-Ending Learning. Within these forms, the common problem is catastrophic forgetting, i.e., the network forgets older tasks. To…

Computer Vision

6 min read

IIRC: Incremental Implicitly-Refined Classification

Aug 10, 2021

Knowledge Evolution in Neural Networks

Deep learning stands on two pillars: GPUs and large datasets. Thus, deep networks suffer when trained from scratch on small datasets. This phenomenon is known as overfitting. This paper [1] proposes knowledge evolution to reduces both overfitting and the burden for data collection. In the paper, knowledge evolution is supported…

Knowledge Distillation

4 min read

Jul 5, 2021

Cross-Iteration Batch Normalization

This paper [1] leverages two simple ideas to solve an important problem. The paper solves the problem of batch normalization when the batch size b is small, e.g., b=2. …

Deep Learning

6 min read

May 10, 2021

Hello

Kubra Tas

Thanks, Kubra, for sharing your thoughts.

Thanks, Kubra, for sharing your thoughts. I never used Deep image prior. It is a cool idea, yet, its training cost is a barrier. As you mentioned, the total time spent on training is optimized; that's correct. Yet, this training time is incurred for every image, i.e., every image is a training sample, there is no inference. Furthermore, I had a colleague who used deep prior for RGB-D images [1]. The training cost becomes more severe as the number of dimensions increases.

1 min read

Furthermore, I had a colleague who used deep prior for RGB-D images [1]. The training cost becomes more severe as the number of dimensions increases.

So, Yes, for 2D images the cost might be manageable, but I would proceed with caution for higher dimensions (e.g., videos). Thanks again for sharing your input :)

[1] Depth Completion Using a View-constrained Deep Prior

Apr 20, 2021

Deep Metric Learning Beyond Binary Supervision

Metric learning literature assumes binary labels where samples belong to either the same or different classes. While this binary perspective has motivated fundamental ranking losses (e.g., Contrastive and Triplet loss), this binary perspective has reached a stagnant point [2]. Thus, one novel direction for metric learning is continuous (non-binary) similarity…

Metric Learning

4 min read