Researchers from Tsinghua University present a new machine learning algorithm under the meta-learning paradigm

Researchers from Tsinghua University present a new machine learning algorithm under the meta-learning paradigm

Recent achievements in supervised deep learning activities can be attributed to the availability of large amounts of labeled training data. Yet it takes a lot of effort and money to collect accurate labels. In many practical contexts, only a small fraction of the training data has labels attached. Semi-supervised learning (SSL) aims to improve model performance using labeled and unlabeled inputs. Many effective SSL approaches, when applied to deep learning, engage in unsupervised consistency regularization to use unlabeled data.

State-of-the-art consistency-based algorithms typically introduce several configurable hyperparameters, even if they achieve excellent performance. For optimal performance of the algorithm, it is common practice to adjust these hyperparameters to optimal values. Unfortunately, hyperparameter lookup is often unreliable in many real-world SSL scenarios, such as medical image processing, hyperspectral image classification, network traffic recognition, and document recognition. This is because annotated data is sparse, leading to high variance when cross-validation is employed. Having algorithm performance sensitive to hyperparameter values ​​makes this problem even more pressing. Furthermore, the computational cost can become unmanageable for cutting-edge deep learning algorithms as the search space grows exponentially in the number of hyperparameters.

Researchers at Tsinghua University have introduced a meta-learning-based SSL algorithm called Meta-Semi to make more use of labeled data. Meta-Semi achieves outstanding performance in many scenarios by adjusting just one other hyperparameter.

Build high-quality training datasets with Kili Technology and solve NLP machine learning challenges to develop powerful ML applications

The team was inspired by the realization that the network can be successfully trained using the properly ‘pseudo-labeled’ unannotated examples. Specifically, during the online training phase, they produce pseudo-soft labels for unlabeled data based on network predictions. Next, they remove samples with unreliable or incorrect pseudo-labels and use the remaining data to train the model. This work shows that the distribution of properly “pseudo-labeled” data should be comparable to that of labeled data. If the network is trained with the former, the final loss on the latter should also be minimized.

They defined the meta-reweighting goal to minimize the ultimate loss on labeled data by selecting the most appropriate weights (weights throughout the paper always refer to the coefficients used to reweight each unlabelled sample rather than refer to neural network parameters). Researchers have encountered computational difficulties in tackling this problem using optimization algorithms.

For this reason they suggest an approximation formulation from which to derive a closed-form solution. Theoretically, they demonstrate that each training iteration only needs a single meta gradient pass to obtain the approximate solutions.

In conclusion, they suggest a dynamic weighting approach to reweight previously pseudo-labeled samples with 0-1 weights. The results show that this approach eventually reaches the stationary point of the supervised loss function. In popular image classification benchmarks (CIFAR-10, CIFAR-100, SVHN and STL-10), the proposed technique has been shown to perform better than state-of-the-art deep networks. For demanding CIFAR-100 and STL-10 SSL tasks, Meta-Semi performs much better than state-of-the-art SSL algorithms such as ICT and MixMatch, and performs slightly better than them on CIFAR-10. Furthermore, Meta-Semi is a useful addition to consistency-based approaches; incorporating consistency regularization into the algorithm further increases performance.

According to the researchers, Meta-Semi takes a little longer to train is a disadvantage. They plan to look into this issue in the future.

Check out thePaperANDReference article.All the credit for this research goes to the researchers of this project. Also, don’t forget to subscribeour 26k+ ML SubReddit,Discord channel,ANDEmail newsletterwhere we share the latest news on AI research, cool AI projects and more.


Dhanshree Shenwai is a software engineer and has good experience in FinTech companies covering Finance, Cards & Payments and Banking domain with keen interest in AI applications. He is enthusiastic about exploring new technologies and advancements in today’s changing world, making everyone’s life easier.


Gain a competitive edge with data – actionable market insights for global brands, retailers, analysts and investors. (Sponsored)

#Researchers #Tsinghua #University #present #machine #learning #algorithm #metalearning #paradigm
Image Source : www.marktechpost.com

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *