Create a dataset

This creates a binary dataset. Binary datasets don’t support sequences. The CLI can’t create sequence datasets.

viam dataset create --org-id=YOUR-ORG-ID --name=my-inspection-dataset

The command returns the dataset ID, which you need for subsequent CLI and SDK operations.

This creates a binary dataset and a sequence dataset. Python supports both types.

import asyncio
from viam.rpc.dial import DialOptions
from viam.app.viam_client import ViamClient
from viam.proto.app.dataset import DatasetType


API_KEY = "YOUR-API-KEY"
API_KEY_ID = "YOUR-API-KEY-ID"
ORG_ID = "YOUR-ORGANIZATION-ID"


async def connect() -> ViamClient:
    dial_options = DialOptions.with_api_key(API_KEY, API_KEY_ID)
    return await ViamClient.create_from_dial_options(dial_options)


async def main():
    viam_client = await connect()
    data_client = viam_client.data_client

    # Binary dataset (the default type). Holds binary files such as images, no sequences.
    dataset_id = await data_client.create_dataset(
        name="my-inspection-dataset",
        organization_id=ORG_ID,
    )
    print(f"Created dataset: {dataset_id}")

    # Sequence dataset. Holds sequences only, no individual images.
    sequence_dataset_id = await data_client.create_dataset(
        name="my-sequence-dataset",
        organization_id=ORG_ID,
        type=DatasetType.DATASET_TYPE_SEQUENCE_DATA,
    )
    print(f"Created dataset: {sequence_dataset_id}")

    viam_client.close()


if __name__ == "__main__":
    asyncio.run(main())

This creates a binary dataset. Binary datasets don’t support sequences. The Go SDK can’t create sequence datasets.

package main

import (
    "context"
    "fmt"

    "go.viam.com/rdk/app"
    "go.viam.com/rdk/logging"
)

func main() {
    apiKey := "YOUR-API-KEY"
    apiKeyID := "YOUR-API-KEY-ID"
    orgID := "YOUR-ORGANIZATION-ID"

    ctx := context.Background()
    logger := logging.NewDebugLogger("create-dataset")

    viamClient, err := app.CreateViamClientWithAPIKey(
        ctx, app.Options{}, apiKey, apiKeyID, logger)
    if err != nil {
        logger.Fatal(err)
    }
    defer viamClient.Close()

    dataClient := viamClient.DataClient()

    // Binary dataset (the default type). Holds binary files such as images, no sequences.
    datasetID, err := dataClient.CreateDataset(
        ctx, "my-inspection-dataset", orgID)
    if err != nil {
        logger.Fatal(err)
    }
    fmt.Printf("Created dataset: %s\n", datasetID)
}

The CLI can add images to a binary dataset. It has no sequence commands, so use the Viam app or an SDK for sequence datasets.

Add images to a dataset using filter criteria:

viam dataset data add filter \
  --dataset-id=YOUR-DATASET-ID \
  --location-ids=YOUR-LOCATION-ID \
  --tags=label1,label2

This adds all images matching the filter to the dataset. You can filter by location, machine, component, tags, or time range. To add images by ID instead, see Datasets and training.

To remove images, use viam dataset data remove.

The Python SDK can add data to both dataset types.

Binary dataset:

async def main():
    viam_client = await connect()
    data_client = viam_client.data_client

    await data_client.add_binary_data_to_dataset_by_ids(
        binary_ids=["binary-data-id-1", "binary-data-id-2"],
        dataset_id="YOUR-DATASET-ID",
    )
    print("Images added to dataset.")

    viam_client.close()

You can get binary data IDs by querying for images first using the data client’s binary_data_by_filter method, which returns objects that include the binary ID. To remove images, call remove_binary_data_from_dataset_by_ids.

Sequence dataset:

await data_client.add_sequences_to_dataset(
    dataset_id="YOUR-DATASET-ID",
    sequence_ids=["YOUR-SEQUENCE-ID"],
)

To remove sequences, call remove_sequences_from_dataset.

The TypeScript SDK can add data to both dataset types.

Binary dataset:

await dataClient.addBinaryDataToDatasetByIds(
  ["binary-data-id-1", "binary-data-id-2"],
  "YOUR-DATASET-ID",
);

To remove images, call removeBinaryDataFromDatasetByIds.

Sequence dataset:

await dataClient.addSequencesToDataset(["YOUR-SEQUENCE-ID"], "YOUR-DATASET-ID");

To remove sequences, call removeSequencesFromDataset.

The Go SDK can add data to both dataset types.

Binary dataset:

err = dataClient.AddBinaryDataToDatasetByIDs(
    ctx,
    []string{"binary-data-id-1", "binary-data-id-2"},
    "YOUR-DATASET-ID",
)
if err != nil {
    logger.Fatal(err)
}
fmt.Println("Images added to dataset.")

To remove images, call RemoveBinaryDataFromDatasetByIDs.

Sequence dataset:

err = dataClient.AddSequencesToDataset(ctx, "YOUR-DATASET-ID", []string{"YOUR-SEQUENCE-ID"})

To remove sequences, call RemoveSequencesFromDataset.

The Viam app can download a binary dataset’s files, without annotations. It can’t export a sequence dataset.

1. Go to the DATA tab and click the DATASETS subtab.
2. Click your dataset to open it.
3. Click the … (dataset actions) menu next to the dataset name and click Download. The option appears only when the dataset contains files.
4. Click Download files. Your browser saves a <dataset-name>.tar.gz archive of the dataset’s files, including non-image files such as video or audio.

The browser download includes the files only, not tags, bounding boxes, or other annotations. The browser builds the whole archive in memory, so it refuses a dataset larger than 500 MB. Files that fail to download are left out of the archive, and a warning says how many. For a large dataset, or when you need a complete copy, use the CLI export instead.

The SDKs can export a sequence dataset. To export a binary dataset with its annotations, use the CLI.

Call start_sequence_dataset_export (Python) or startSequenceDatasetExport (TypeScript) to get a job ID, poll get_sequence_dataset_export or getSequenceDatasetExport until its status is COMPLETED, then download the zip from download_url. The URL expires after one hour, so download it right away.

The Go SDK has no dataset export. Use the CLI.