Metadata-Version: 2.1
Name: clip_by_openai
Version: 0.1.1.3
Summary: CLIP by OpenAI
Home-page: UNKNOWN
Author: OpenAI (Packaged by Vector AI)
Author-email: dev@vctr.ai
License: MIT
Description: # CLIP
        
        [[Blog]](https://openai.com/blog/clip/) [[Paper]](https://cdn.openai.com/papers/Learning_Transferable_Visual_Models_From_Natural_Language_Supervision.pdf) [[Model Card]](model-card.md) [[Colab]](https://colab.research.google.com/github/openai/clip/blob/master/Interacting_with_CLIP.ipynb)
        
        CLIP (Contrastive Language-Image Pre-Training) is a neural network trained on a variety of (image, text) pairs. It can be instructed in natural language to predict the most relevant text snippet, given an image, without directly optimizing for the task, similarly to the zero-shot capabilities of GPT-2 and 3. We found CLIP matches the performance of the original ResNet50 on ImageNet “zero-shot” without using any of the original 1.28M labeled examples, overcoming several major challenges in computer vision.
        
        
        
        ## Approach
        
        ![CLIP](CLIP.png)
        
        
        
        ## Usage
        
        ```bash
        $ pip install clip-by-openai 
        ```
        Or if you want cuda installation:
        ```bash
        $ pip install clip-by-openai[cuda]
        ```
        
        ```python
        import torch
        import clip
        from PIL import Image
        
        device = "cuda" if torch.cuda.is_available() else "cpu"
        model, preprocess = clip.load("ViT-B/32", device=device)
        
        image = preprocess(Image.open("CLIP.png")).unsqueeze(0).to(device)
        text = clip.tokenize(["a diagram", "a dog", "a cat"]).to(device)
        
        with torch.no_grad():
            image_features = model.encode_image(image)
            text_features = model.encode_text(text)
            
            logits_per_image, logits_per_text = model(image, text)
            probs = logits_per_image.softmax(dim=-1).cpu().numpy()
        
        print("Label probs:", probs)  # prints: [[0.9927937  0.00421068 0.00299572]]
        ```
        
        
        ## API
        
        The CLIP module `clip` provides the following methods:
        
        #### `clip.available_models()`
        
        Returns the name(s) of the available CLIP models.
        
        #### `clip.load(name, device=..., jit=True)`
        
        Returns the model and the TorchVision transform needed by the model, specified by the model name returned by `clip.available_models()`. It will download the model as necessary. The device to run the model can be optionally specified, and the default is to use the first CUDA device if there is any, otherwise the CPU.
        
        When `jit` is `False`, a non-JIT version of the model will be loaded.
        
        #### `clip.tokenize(text: Union[str, List[str]], context_length=77)`
        
        Returns a LongTensor containing tokenized sequences of given text input(s). This can be used as the input to the model
        
        ---
        
        The model returned by `clip.load()` supports the following methods:
        
        #### `model.encode_image(image: Tensor)`
        
        Given a batch of images, returns the image features encoded by the vision portion of the CLIP model.
        
        #### `model.encode_text(text: Tensor)`
        
        Given a batch of text tokens, returns the text features encoded by the language portion of the CLIP model.
        
        #### `model(image: Tensor, text: Tensor)`
        
        Given a batch of images and a batch of text tokens, returns two Tensors, containing the logit scores corresponding to each image and text input. The values are cosine similarities between the corresponding image and text features, times 100.
        
        
        
        ## More Examples
        
        ### Zero-Shot Prediction
        
        The code below performs zero-shot prediction using CLIP, as shown in Appendix B in the paper. This example takes an image from the [CIFAR-100 dataset](https://www.cs.toronto.edu/~kriz/cifar.html), and predicts the most likely labels among the 100 textual labels from the dataset.
        
        ```python
        import os
        import clip
        import torch
        from torchvision.datasets import CIFAR100
        
        # Load the model
        device = "cuda" if torch.cuda.is_available() else "cpu"
        model, preprocess = clip.load('ViT-B/32', device)
        
        # Download the dataset
        cifar100 = CIFAR100(root=os.path.expanduser("~/.cache"), download=True, train=False)
        
        # Prepare the inputs
        image, class_id = cifar100[3637]
        image_input = preprocess(image).unsqueeze(0).to(device)
        text_inputs = torch.cat([clip.tokenize(f"a photo of a {c}") for c in cifar100.classes]).to(device)
        
        # Calculate features
        with torch.no_grad():
            image_features = model.encode_image(image_input)
            text_features = model.encode_text(text_inputs)
        
        # Pick the top 5 most similar labels for the image
        image_features /= image_features.norm(dim=-1, keepdim=True)
        text_features /= text_features.norm(dim=-1, keepdim=True)
        similarity = (100.0 * image_features @ text_features.T).softmax(dim=-1)
        values, indices = similarity[0].topk(5)
        
        # Print the result
        print("\nTop predictions:\n")
        for value, index in zip(values, indices):
            print(f"{cifar100.classes[index]:>16s}: {100 * value.item():.2f}%")
        ```
        
        The output will look like the following (the exact numbers may be slightly different depending on the compute device):
        
        ```
        Top predictions:
        
                   snake: 65.31%
                  turtle: 12.29%
            sweet_pepper: 3.83%
                  lizard: 1.88%
               crocodile: 1.75%
        ```
        
        Note that this example uses the `encode_image()` and `encode_text()` methods that return the encoded features of given inputs.
        
        
        ### Linear-probe evaluation
        
        The example below uses [scikit-learn](https://scikit-learn.org/) to perform logistic regression on image features.
        
        ```python
        import os
        import clip
        import torch
        
        import numpy as np
        from sklearn.linear_model import LogisticRegression
        from torch.utils.data import DataLoader
        from torchvision.datasets import CIFAR100
        from tqdm import tqdm
        
        # Load the model
        device = "cuda" if torch.cuda.is_available() else "cpu"
        model, preprocess = clip.load('ViT-B/32', device)
        
        # Load the dataset
        root = os.path.expanduser("~/.cache")
        train = CIFAR100(root, download=True, train=True, transform=preprocess)
        test = CIFAR100(root, download=True, train=False, transform=preprocess)
        
        
        def get_features(dataset):
            all_features = []
            all_labels = []
            
            with torch.no_grad():
                for images, labels in tqdm(DataLoader(dataset, batch_size=100)):
                    features = model.encode_image(images.to(device))
        
                    all_features.append(features)
                    all_labels.append(labels)
        
            return torch.cat(all_features).cpu().numpy(), torch.cat(all_labels).cpu().numpy()
        
        # Calculate the image features
        train_features, train_labels = get_features(train)
        test_features, test_labels = get_features(test)
        
        # Perform logistic regression
        classifier = LogisticRegression(random_state=0, C=0.316, max_iter=1000, verbose=1)
        classifier.fit(train_features, train_labels)
        
        # Evaluate using the logistic regression classifier
        predictions = classifier.predict(test_features)
        accuracy = np.mean((test_labels == predictions).astype(np.float)) * 100.
        print(f"Accuracy = {accuracy:.3f}")
        ```
        
        Note that the `C` value should be determined via a hyperparameter sweep using a validation split.
        
Keywords: vector,embeddings,machinelearning,ai,artificialintelligence,nlp,pytorch,nearestneighbors,search,analytics,clustering,dimensionalityreduction
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Education
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Financial and Insurance Industry
Classifier: Intended Audience :: Healthcare Industry
Classifier: Intended Audience :: Manufacturing
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.4
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: Implementation :: PyPy
Classifier: Topic :: Database
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Multimedia :: Sound/Audio :: Conversion
Classifier: Topic :: Multimedia :: Video :: Conversion
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Image Recognition
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Scientific/Engineering :: Visualization
Classifier: Topic :: Software Development :: Libraries :: Application Frameworks
Requires-Python: >=3
Description-Content-Type: text/markdown
Provides-Extra: cuda
Provides-Extra: dev
