Moondream: A Tiny Vision-Language Model for Image Q&A, Captioning, Detection, and Pointing

Hey folks, in this tutorial I want to introduce you to one of the most fun models I have played with recently, and it is called Moondream. It is a vision-language model (VLM) that is really small compared to those giant models we usually hear about. But do not underestimate it, because even though it is tiny, Moondream can handle a lot of cool tasks ranging from writing image captions, answering questions about image content, detecting objects, all the way to pointing at the location of objects inside an image.

What I love about Moondream is that it is designed to run cheaply and locally. You can run it on a regular laptop CPU, on a GPU if you have one, or through a cloud API if you do not want to deal with infrastructure. So for those of you learning computer vision, building edge device projects, or needing to tag thousands of images in batch, Moondream is really worth trying.

In this tutorial I will explain everything from scratch: what a small VLM is and why it is useful, how to install it, how to load the model (both through the official package and HuggingFace transformers), how to use its four main skills (caption, query, detect, point), a comparison of running locally versus in the cloud, and some best practices to get the most out of it. Let us get started.

Introduction

Before diving into the code, let me talk about the concept first so we are on the same page.

What is a Vision-Language Model?

A vision-language model or VLM is a model that can "see" images while also "understanding" language. So the input is not just text like a regular LLM, but a combination of images plus text. You give it an image and ask "how many people are in this photo?", and it will answer in natural language. Essentially a VLM is a bridge between the visual world and the text world.

Large VLM models usually have billions of parameters, need expensive GPUs, and are also expensive to call through an API. This is where Moondream comes in. Moondream is a small VLM, only around 2 billion parameters (there is even a smaller variant), so it can run on much more affordable hardware.

Why are small VLMs useful?

You might be thinking, if it is small then the quality must be bad, right? Not always. For many practical tasks, a small VLM like Moondream is more than enough. Here are a few reasons why I think small VLMs matter:

• Cheap and efficient: No need to rent expensive GPUs by the hour. A regular CPU works, although a bit slow.
• Local and private: Your image data does not need to leave your own machine. This is crucial for sensitive data like medical records or internal documents.
• Great for edge: It can be deployed on small devices like a Raspberry Pi, smart cameras, or robots, where internet connectivity is not always available.
• Low latency: Because the model is small, processing is fast, perfect for applications that need quick responses.
• Batch processing: If you have millions of images that need tagging or captioning, a small model makes the total cost reasonable.

Moondream's four main capabilities

Moondream has four core skills that we will discuss one by one:

Caption: Automatically generate a text description of an image.

Query: Visual question answering, meaning you ask anything about the image and it answers.

Detect: Detect objects and return their bounding box coordinates.

Point: Point at the location of objects with point coordinates (x, y).

The combination of these four skills makes Moondream super flexible. You can use it for accessibility (image descriptions for the visually impaired), content moderation, tagging automation, robotics, all the way to document analysis. Okay, now let us jump straight into practice.

Installation

In this section I will show you a few ways to install Moondream. There are two main paths: using the official moondream package, or through HuggingFace transformers. I will cover both so you can pick according to your needs.

Method 1: Install the official Moondream package

This is the easiest way. The official package is lightweight and prepared to be plug-and-play. Open your terminal and run:

pip install moondream

I always recommend using a virtual environment so your dependencies stay tidy and do not clash with other projects. Here is how:

python -m venv venv
source venv/bin/activate   # on Windows: venv\Scripts\activate
pip install --upgrade pip
pip install moondream pillow

I also installed pillow (PIL) there because we will need this library to open and manipulate images. Almost all examples in this tutorial will use PIL.

Method 2: Install through HuggingFace transformers

If you are already familiar with the HuggingFace ecosystem and want full control over the model (for example you want to quantize, manually set the device, or integrate into an existing pipeline), the transformers path is a better fit. Install the dependencies like this:

pip install transformers torch pillow einops

Some Moondream versions need einops for certain tensor operations, so I included it. If you have an NVIDIA GPU, make sure to install the CUDA version of PyTorch so you can take advantage of the GPU. Check the official PyTorch website first for the install command that matches your CUDA version.

Verify the installation

After installing, it is a good idea to check whether everything is installed correctly. Try running this small script:

import moondream as md
import PIL.Image

print("Moondream imported successfully")
print("PIL version:", PIL.version)

If there are no errors, then we are ready to continue. If there is an error about a module not being found, double check whether your virtual environment is active, or reinstall the missing package.

Setting up the API key for the cloud (optional)

If you want to use the cloud version (I will explain the difference later), you need an API key from the Moondream website. Once you have it, I recommend storing the key in an environment variable for safety, do not write it directly in the code:

export MOONDREAMAPIKEY="your-key-here"

On Windows PowerShell, the way is:

$env:MOONDREAMAPIKEY="your-key-here"

Later in the code we grab it using os.environ. Okay, installation done, now let us move on to basic usage.

Basic Usage

In this section we will try out Moondream's four main skills one by one. I use a simple example first so you understand the flow, then later in the advanced section we will build something more complex.

Loading the model locally

First, we load the model. If you use the official package, you usually need a model file (typically a .mf format or similar) that can be downloaded from the official Moondream repo. Once you have the file, load it like this:

import moondream as md
from PIL import Image

Load a local model from file (runs on CPU/GPU depending on setup)
model = md.vl(model="moondream-2b-int8.mf")

Open the image we want to analyze
image = Image.open("example.jpg")

print("Model and image ready to use")

Now we have a model object and an image. From here we can call the four skills.

Skill 1: Caption (generate an image description)

Caption is for automatically generating a text description of an image. Super useful for accessibility, image SEO, or creating metadata. Moondream usually provides a few caption lengths, for example "short" and "normal".

from PIL import Image
import moondream as md

model = md.vl(model="moondream-2b-int8.mf")
image = Image.open("example.jpg")

Short caption, good for brief alt text
shortresult = model.caption(image, length="short")

print("Short caption:", shortresult["caption"])

Normal caption, more detailed
normalresult = model.caption(image, length="normal")

print("Normal caption:", normalresult["caption"])

The output could be something like "A dog sitting on a wooden floor" for the short one, or a longer and more detailed description for the normal one. Pretty easy, right?

Skill 2: Query (visual question answering)

Now this is my favorite. Query is visual question answering, meaning you can ask anything about the image and Moondream will answer in natural language.

from PIL import Image
import moondream as md

model = md.vl(model="moondream-2b-int8.mf")
image = Image.open("example.jpg")

Ask about the number of objects
answer = model.query(image, "How many people are in this image?")
print("Answer:", answer["answer"])

Ask about color
answer2 = model.query(image, "What color is the shirt of the person in the middle?")
print("Answer:", answer2["answer"])

Ask about more abstract context
answer3 = model.query(image, "Does the mood of this photo look happy?")
print("Answer:", answer3["answer"])

You can ask about counts, colors, positions, activities, all the way to abstract things like mood. Keep in mind, because this is a small model, answers to very complex questions might not always be perfect, but for everyday questions the results are pretty good.

Skill 3: Detect (object detection with bounding boxes)

Detect is for finding specific objects in an image while also getting their bounding box coordinates. You just tell it what object you are looking for.

from PIL import Image
import moondream as md

model = md.vl(model="moondream-2b-int8.mf")
image = Image.open("street.jpg")

Detect all objects of type "car"
result = model.detect(image, "car")

print("Number of cars detected:", len(result["objects"]))
for i, obj in enumerate(result["objects"]):
    print(f"Car {i+1}: xmin={obj['xmin']:.3f}, ymin={obj['ymin']:.3f}, "
          f"xmax={obj['xmax']:.3f}, ymax={obj['ymax']:.3f}")

Keep in mind, the returned coordinates are usually in normalized form (values between 0 and 1), not absolute pixels. So if you want to draw the bounding box on the original image, you have to multiply first by the image width and height. I will show an example in the advanced section.

Skill 4: Point (pointing at object locations)

Point is similar to detect, but the difference is it gives a point coordinate (x, y) pointing to the center of the object, not a box. This is really useful for robotics (for example "grab the object at this point") or for a UI that wants to highlight a location.

from PIL import Image
import moondream as md

model = md.vl(model="moondream-2b-int8.mf")
image = Image.open("table.jpg")

Point at all "cups" in the image
result = model.point(image, "cup")

print("Number of points:", len(result["points"]))
for i, point in enumerate(result["points"]):
    print(f"Cup {i+1}: x={point['x']:.3f}, y={point['y']:.3f}")

Just like detect, the coordinates are normalized. So if x=0.5 and y=0.5, that means the object is right in the center of the image.

That covers Moondream's four basic skills. Now let us level up to more serious usage.

Advanced Usage

In this section I will discuss more practical things: how to load via transformers, how to run locally versus in the cloud, how to visualize detection results, all the way to batch processing for thousands of images.

Loading Moondream via HuggingFace transformers

If you want more control, the transformers path is a great option. Here we can set the device explicitly and take advantage of the GPU if there is one.

from transformers import AutoModelForCausalLM, AutoTokenizer
from PIL import Image
import torch

modelid = "vikhyatk/moondream2"

revision = "2024-08-26"  # pin the revision so results stay consistent

model = AutoModelForCausalLM.frompretrained(
    modelid,

    trustremotecode=True,

    revision=revision,
    torchdtype=torch.float16,
    devicemap={"": "cuda"} if torch.cuda.isavailable() else None,
)
tokenizer = AutoTokenizer.frompretrained(modelid, revision=revision)

image = Image.open("example.jpg")

Encode the image once, then reuse it many times
encimage = model.encodeimage(image)

Query using the encoded image
answer = model.answerquestion(encimage, "What is in this image?", tokenizer)
print("Answer:", answer)

An important trick here: encodeimage processes the image into an internal representation. If you want to ask many questions about the same image, encode it once and reuse the result. This saves a lot of computation because the heaviest part (the visual processing) is not repeated.

Running locally on CPU vs GPU

Moondream can run on either CPU or GPU. The difference is just speed. Here is an example of how we consciously set the device and measure the time:

import torch
import time
from transformers import AutoModelForCausalLM, AutoTokenizer
from PIL import Image

modelid = "vikhyatk/moondream2"

Pick the device automatically
device = "cuda" if torch.cuda.isavailable() else "cpu"

print(f"Using device: {device}")

dtype = torch.float16 if device == "cuda" else torch.float32

model = AutoModelForCausalLM.frompretrained(
    modelid, trustremotecode=True, torchdtype=dtype
).to(device)
tokenizer = AutoTokenizer.frompretrained(modelid)

image = Image.open("example.jpg")

start = time.time()
enc = model.encodeimage(image)

answer = model.answerquestion(enc, "Describe this image briefly.", tokenizer)
duration = time.time() - start

print("Answer:", answer)
print(f"Processing time: {duration:.2f} seconds")

From my experience, on a regular laptop CPU one query can take several seconds to a dozen seconds depending on the specs. On a GPU it can drop to under one second. So if you need high throughput, a GPU is clearly nicer. But for prototyping or small volumes, a CPU is enough.

Using the cloud endpoint with an API key

If you do not want to think about hardware, or need to scale big without hassle, Moondream provides a cloud API. The difference from local: the processing runs on their server, you just send the image and receive the result. The nice part, no need to download the model and it does not eat your machine's resources. The downside, your image data leaves your machine and there is a cost per call.

import os
import moondream as md
from PIL import Image

Grab the API key from an environment variable (do not hardcode!)
apikey = os.environ["MOONDREAMAPIKEY"]


Initialize the cloud client
model = md.vl(apikey=apikey)


image = Image.open("example.jpg")

The calls are exactly the same as the local version
caption = model.caption(image, length="normal")
print("Caption:", caption["caption"])

answer = model.query(image, "What objects are here?")
print("Answer:", answer["answer"])

What I like is that the API is consistent. The code for local and cloud is almost identical, only differing in how the model is initialized. So you can develop using the cloud first, then move to local later without changing much code, or vice versa.

When to pick local, when to pick cloud?

To make it easy, here is a short guide from me:

• Pick local if: your data is sensitive and cannot leave, you want to save costs long term, you run on an edge device, or there is no internet.
• Pick cloud if: you do not want infrastructure hassle, you need to scale quickly, your volume is unpredictable, or your machine is not powerful enough.

Many also use a hybrid approach: use the cloud for traffic spikes, and local for the baseline. Super flexible.

Visualizing detection results

Raw coordinates are sometimes hard to imagine. Let us draw the bounding boxes directly on the image using PIL. Remember, the coordinates are normalized, so we multiply first by the image dimensions.

from PIL import Image, ImageDraw
import moondream as md

model = md.vl(model="moondream-2b-int8.mf")
image = Image.open("street.jpg")
width, height = image.size

result = model.detect(image, "car")

Create a draw object to sketch on
draw = ImageDraw.Draw(image)

for obj in result["objects"]:
    # Convert normalized coordinates to absolute pixels
    x0 = obj["xmin"]  width

    y0 = obj["ymin"]  height
    x1 = obj["xmax"]  width

    y1 = obj["ymax"]  height
    draw.rectangle([x0, y0, x1, y1], outline="red", width=3)

image.save("detectionresult.jpg")
print("Image with bounding boxes saved to detectionresult.jpg")

Now you can open detectionresult.jpg and see the red boxes around the detected cars. The same way can be used for point, just replace rectangle with a small ellipse around the point.

Batch processing: tagging thousands of images

This is one of the most powerful use cases of a small VLM: automatically tagging or captioning many images at once. Imagine you have a folder with thousands of product photos that do not have descriptions yet. With Moondream, we can automate it.

import os
import json
import moondream as md
from PIL import Image

model = md.vl(model="moondream-2b-int8.mf")

folder = "imagedataset"

allresults = []

for filename in os.listdir(folder):
    if not filename.lower().endswith((".jpg", ".jpeg", ".png")):
        continue

    path = os.path.join(folder, filename)
    try:
        image = Image.open(path).convert("RGB")

        caption = model.caption(image, length="short")["caption"]
        category = model.query(image, "What product category is this? Answer in one word.")["answer"]

        allresults.append({

            "file": filename,
            "caption": caption,
            "category": category.strip(),
        })
        print(f"Done: {filename} -> {caption}")
    except Exception as e:
        print(f"Failed to process {filename}: {e}")

Save all results to JSON
with open("imagetags.json", "w", encoding="utf-8") as f:
    json.dump(allresults, f, ensureascii=False, indent=2)

print(f"Total {len(allresults)} images successfully tagged")

This script will loop through all the images in the folder, create captions and categories, then save the results to a JSON file. Notice I wrap the processing of each image with try/except so that one broken image does not fail the whole batch. This is really important for long-running processes.

Edge use case example: smart camera

For those of you working in IoT or robotics, Moondream can be deployed on an edge device. Imagine a smart camera that checks whether there is a package at the front door:

import moondream as md
from PIL import Image
import time

model = md.vl(model="moondream-0.5b-int8.mf")  # the smallest variant for edge

def checkpackage(imagepath):

    image = Image.open(imagepath)
    answer = model.query(image, "Is there a package or cardboard box in this image? Answer yes or no.")
    return "yes" in answer["answer"].lower()

Simulate a checking loop every 30 seconds
while True:
    if checkpackage("camerasnapshot.jpg"):
        print("Package detected! Sending notification...")
    else:
        print("No package yet.")
    time.sleep(30)

Here I use the smallest model variant (0.5b) so it is light for constrained devices like a Raspberry Pi. The trade-off is that its accuracy is slightly below the larger variant, but for simple tasks like detecting present/absent, this is more than enough.

Best Practices

After playing with Moondream for a while, I have a few tips to make your results better and your pipeline more reliable.

1. Encode the image once if you ask multiple times

Like I mentioned earlier, if you want to ask many questions about the same image, encode the image just once using encodeimage (in the transformers path). This saves a lot of time because the visual encoding part is the heaviest.

enc = model.encodeimage(image)
q1 = model.answerquestion(enc, "How many people are there?", tokenizer)

q2 = model.answerquestion(enc, "What is the dominant color?", tokenizer)
q3 = model.answer_question(enc, "Is it indoors or outdoors?", tokenizer)

2. Write specific and focused questions

A small model is best when the question is clear and focused. Instead of asking "tell me everything about this image", it is better to break it into specific questions. If you need an answer that can be processed by a program, ask for a certain format, for example "Answer with a number only" or "Answer yes or no".

3. Always normalize the image format

Get into the habit of converting images to RGB before processing, because some images (for example PNGs with an alpha channel or grayscale) can cause problems. Just one line:

image = Image.open(path).convert("RGB")

4. Handle errors cleanly in large batches

When processing thousands of images, there will always be some that are corrupt or have a weird format. Always wrap per-image processing with try/except and record which ones failed, so the batch keeps running and you can recheck the errors later.

5. Choose the model variant according to your needs

Moondream has several size and quantization variants (for example int8, the 0.5b version, the 2b version). If you need accuracy, pick a larger one. If you need speed and low memory usage (especially on edge), pick a small and quantized one. Try a few and measure which fits your case best.

6. Remember the normalized coordinate constraint

This often confuses beginners. Coordinates from detect and point are between 0 and 1, not pixels. Always multiply by the image width/height before using them to draw or crop. A small mistake here and your bounding box will be way off.

7. Cache results for the same image

If you often process the same image repeatedly (for example in a web app), store the caption/query result in a cache or database. No need to recompute it each time, saving time and cost (especially if using the cloud).

8. Verify results for important decisions

Moondream is good, but it is still a small model and can be wrong. For high-impact decisions (for example content moderation or medical), do not just rely on the raw output. Add a verification layer, whether that is a human or another model, for critical cases.

9. Secure your API key

If you use the cloud, never write the API key directly in the code or commit it to Git. Always use an environment variable or secret manager. I often see people accidentally leak their key because they pushed it to a public repo, and that can cause the bill to balloon.

10. Limit image resolution if needed

Super high resolution images do not always produce better results, and instead make processing slower and consume more memory. If memory is limited, resize the image to a reasonable size first before processing:

image.thumbnail((1024, 1024))  # resize while keeping the aspect ratio

Conclusion

Okay folks, we have reached the end of the tutorial. Hopefully by now you have a clear picture of Moondream and how to use it.

A quick recap: Moondream is a tiny vision-language model that can handle four main things, namely caption (image description), query (visual question answering), detect (object detection with bounding boxes), and point (pointing at object locations). What makes it special is that it can run cheaply and locally on CPU or GPU, or through the cloud API if you want convenience. The API is consistent between local and cloud, so it is very easy to switch back and forth.

For those of you looking for a computer vision solution that is cheap, private, and can run anywhere, I think Moondream is a very worthwhile option to try. It is perfect for accessibility, batch image tagging, content moderation, robotics, all the way to edge applications on small devices.

My advice, start small first. Try installing it, load the model, then play around with one image using all four skills. Once you are comfortable, then scale up to batch processing or integrate it into your application. Do not forget to apply the best practices I mentioned earlier, especially around error handling and API key security.

Computer vision is getting more democratic, and small models like Moondream are proof that you do not need a giant GPU or a big budget to build cool things. Happy tinkering, and see you in the next tutorial. Keep up the spirit of learning, folks!