We released a free tool (fastdup) where you can use to cluster image using DINOv2 embeddings.

fastdup is a tool that let you gain insights from a large image/video collection.

It lets you identify image duplicates, video duplicates, wrong labels, outliers, corrupted data, and image clusters.

fastdup is -

Unsupervised: fits any visual dataset. Scalable: handles 400M images on a single machine. Efficient: works on CPU (even on Google Colab with only 2 CPU cores!). Low Cost: can process 12M images on a $1 cloud machine budget.

The authors at fastdup ran an analysis on LAION 400M and Imagenet21K. Here's what they found.

LAION 400M

> 60M duplicates. > 962K broken images. > Various label discrepancies.

ImageNet21K

> 1.2M duplicate images. > 104K train/val leak.

fastdup GitHub repo - https://github.com/visual-layer/fastdup

Even pros have dark, blurry & duplicate shots. But disorganization can make it hard to find those special memories. Let's fix that.

Update - After some code optimization, I got the inference time down to below 100ms. The lowest I got on my Pixel 3 XL is 37ms!

https://dicksonneoh.com/portfolio/pytorch_at_the_edge_timm_t...

With flutter you can also build web apps.

https://flutter.dev/multi-platform/web

Thank you for the feedback! Let me know if you have questions :)

Thank you!

Have you had a chance to play with Pytorch 2.0's export yet?

Note yet! But I heard things are going to be a lot faster in 2.0. Have you tried?

Thank you!!!

Thanks a lot there! I was hesitating to write this piece actually, thinking it's not going to be valuable. I'm glad you find value in them!

Thnx for writing! In academia we're getting the next step operational: training on Android. Any advise for us what to watch out for?

Obviously you need a bit of patience and lots of volunteer devices. With unsupervised continuous learning this is solved, in emulation. See "G-Rank: Unsupervised Continuous Learn-to-Rank for Edge Devices in a P2P Network" [1]. Optimal learning rate is left as an exercise for the developer. (disclaimer: our own work, I run a lab with "systems for peer-to-peer machine learning")

[1] https://arxiv.org/abs/2301.12530

You're welcome! I'm not sure if I'm the right person to advise on this. But this idea is also known as federated learning right?