# API Compatibility Policy
Source: https://api-reference.scale.com/docs/api-compatibility-policy
API Compatibility Policy The Scale REST API is versioned via a prefix in the URL. Currently, we only support v1 of the API via https://api.scale.com/v1/. Within an API version, we strive to only make backward-compatible
# API Compatibility Policy
The Scale REST API is versioned via a prefix in the URL. Currently, we only support v1 of the API via `https://api.scale.com/v1/`. Within an API version, we strive to only make backward-compatible changes to the API. This means that a client that integrates with `v1` of our REST API will continue to work when querying `v1` in the future.
### **Backward-compatible changes**
Backward-compatible changes can be made within an API version, and include:
* Adding new optional request parameters or HTTP request headers.
* Making a required request parameter optional.
* Rejecting a request which contains invalid parameter values, as a result of improved parsing and validation. Improperly formed requests that were previously accepted may be later rejected.
* Adding new fields to a response or callback object.
* Adding new values for existing parameters that have enumerated sets of values.
* Undocumented functionality may be removed or changed at any time.
### **Backward-incompatible changes**
Backward-incompatible changes will only happen in a new version of the API, and include:
* Changing the behavior of existing request parameters.
* Changing the names of elements used in API POST or PUT requests.
* Adding a new required request parameter.
* Changing the meaning of parameters that have enumerated values, or the meanings of those parameter values.
* Changing the meaning or type of existing response or callback fields.
* Changing the meaning of an API route.
### **Deprecation policy**
At times, we will deprecate and remove parts of our API that can no longer be supported. In this case, we will make a best effort to identify and notify affected users, and will continue to support the deprecated API during the migration process.
# Authentication
Source: https://api-reference.scale.com/docs/api-reference/authentication
Authentication
# Authentication
Scale uses "HTTP Basic Auth" to authenticate API calls. Scale expects for the API key to be included in all API requests to our platform. "HTTP Basic Auth" supports a username and password. Provide your API key as the basic auth username value. There is not a password, you should leave that blank.
### Getting your API key
Your API keys are conveniently located in your **[dashboard](https://dashboard.scale.com/pro)**. Access to your dashboard is gained by either **[logging in](https://dashboard.scale.com/login)** into your existing account or creating a new one **[signing up](https://dashboard.scale.com/signup)**. Once you're in, navigate to your profile and select 'API Key' from the drop-down menu.
### Don't see the "API Keys" section?.
Have your account admin head over to **[https://dashboard.scale.com/settings/team](https://dashboard.scale.com/settings/team)** and update your **[role](/docs/team-getting-started#section-user-roles)** to be a "Manager" - Only Admins and Managers have access to API Keys for your security.
### Test and Live Modes
To make the API as explorable as possible, accounts have test mode and live mode API keys. There is no "switch" for changing between modes, just use the appropriate key to perform a live or test API requests.
Requests made with test mode credentials are not completed by a human, and therefore have incorrect test responses. Requests made with live mode credentials are always completed by a human and will incur a charge.
### Environments are Separate
The live and test modes of Scale are truly self-contained and separate. If you have created a Project in the live mode, and try to reference it when creating a task in testing mode, you will get an error.
The Scale AI web application has a toggle to switch between viewing the live and test modes below the project list on the left-hand side.
### Callback Authentication
If you'd like to authenticate our **[callbacks](/docs/api-reference/callbacks)**, we set a `**scale-callback-auth**` HTTP header on each of our callbacks. The value will be equal to your `**Live Callback Auth Key**` shown on your dashboard. If this header is not set, or it is set incorrectly, the callback is not from Scale.
### How does Basic Auth work?
The end goal of our authentication is to be able to specify an `**Authorization**` header on the requests we're making to the Scale platform.
Let's pretend our Scale API Key is `**live_ScaleRocks**`.
Basic Auth puts the username and password together separated by a colon so that it looks like this `**username:password**`. Because Scale doesn't have a password, the value is going to be simply `**username:**` (with the trailing colon)
In our example, this value would now be `**live_ScaleRocks:**`
We then need to do a Base64 encoding for this new string. Virtually all programming languages come with a helper function that can convert a string into it's Base64 encoded version.
In our example, our encoded string would now look like `**bGl2ZV9TY2FsZVJvY2tzOg==**`
The authorization header when using basic auth starts with `**Basic **`, and then the encoded string.
Following the example, the `**Authorization**` header on our request would be `**Basic bGl2ZV9TY2FsZVJvY2tzOg==**`
If you look at a request header in Postman or in our docs, you'll see the encoded version of your API key being set directly in the header.
```python Python theme={null}
import requests
from requests.auth import HTTPBasicAuth
url = "https://api.scale.com/v1/tasks"
headers = {"Accept": "application/json"}
auth = HTTPBasicAuth('{{ApiKey}}', '') # No password
response = requests.request("GET", url, headers=headers, auth=auth)
print(response.text)
```
```python Python SDK theme={null}
import scaleapi
client = scaleapi.ScaleClient('{{ApiKey}}')
```
# Batches
Source: https://api-reference.scale.com/docs/api-reference/batches
Create a Batch Finalize Batch Batch Retrieval Batch Status List All Batches Batch Priorization
# Create a Batch
This endpoint facilitates the creation of a new batch within a project.
The name of the project this batch (and its tasks) belong to.
***
Name identifying this batch. Must be unique among all batches belonging to a customer.
***
The full url (including the scheme http\:// or https\://) or email address of the callback that will be used when the task is completed.
***
Only applicable for Rapid projects. Create an calibration batch by setting the calibration\_batch flag to true.
***
Only applicable for Rapid projects. Create a self label batch by setting the self\_label\_batch flag to true.
***
```python Python theme={null}
import requests
# Replace with your actual API key
API_KEY = 'your_api_key_here'
# Define the URL for the API endpoint
url = "https://api.scale.com/v1/batches"
# Define the payload for creating a new batch
payload = {
"project": "kitten_labeling", # The project associated with the batch
"name": "kitten_labeling_2020-07", # The name of the batch
"calibration_batch": False, # Indicates if the batch is a calibration batch
"self_label_batch": False # Indicates if the batch is a self-label batch
}
# Set up the headers for the request
headers = {
"accept": "application/json", # Specify that we want the response in JSON format
"content-type": "application/json" # Specify the content type of the request
}
# Adding authentication to the POST request
# The auth parameter requires a tuple with the API key and an empty string
response = requests.post(url, json=payload, headers=headers, auth=(API_KEY, ''))
# Print the response text to see the result
print(response.text)
```
```python Python SDK theme={null}
import scaleapi
# Initialize the ScaleClient with your API key
client = scaleapi.ScaleClient("YOUR_API_KEY_HERE")
# Define the batch payload
batch_payload = {
"project": "project_name", # The name of the project this batch belongs to
"name": "batch_name", # The unique name for this batch
"callback": "http://www.example.com/callback", # The callback URL or email
"calibration_batch": False, # Only applicable for Rapid projects
"self_label_batch": False # Only applicable for Rapid projects
}
# Create the batch
batch = client.create_batch(**batch_payload)
# Print the created batch's details
print(batch.as_dict())
```
# Finalize Batch
For "Scale Rapid and Studio" customers only, finalizes a batch with name batchName so its tasks can be worked on.
Non-(Rapid/Studio) customers do not need to use this endpoint - calling this endpoint will not do anything, but still return a 200 success status code.
Required batchName to finalize.
***
```python Python theme={null}
import requests
# Replace with your actual API key
API_KEY = 'your_api_key_here'
# Define the URL for the API endpoint
url = "https://api.scale.com/v1/batches/kitten_labeling_2020-07/finalize"
# Set up the headers for the request
headers = {
"accept": "application/json" # Specify that we want the response in JSON format
}
# Adding authentication to the POST request
# The auth parameter requires a tuple with the API key and an empty string
response = requests.post(url, headers=headers, auth=(API_KEY, ''))
# Print the response text to see the result
print(response.text)
```
```python Python SDK theme={null}
import scaleapi
# Initialize the ScaleClient with your API key
client = scaleapi.ScaleClient("YOUR_API_KEY_HERE")
# Define the batch name to finalize
batch_name = "kitten_labeling_2020-07"
# Finalize the batch using the direct method
client.finalize_batch(batch_name=batch_name)
# Alternative method to finalize the batch
batch = client.get_batch(batch_name=batch_name)
batch.finalize()
# Print confirmation
print(f"Batch '{batch_name}' has been finalized.")
```
```json theme={null}
{
"project": "TEST-PROJECT",
"name": "BATCH-NAME",
"callback": "your@email.com",
"status": "in_progress",
"created_at": "2023-08-01T23:04:12.168Z",
"metadata": {}
}
```
# Batch Retrieval
This endpoint returns the details of a batch with the name :batchName.
batchName to retrieve
***
```python Python theme={null}
import requests
# Replace with your actual API key
API_KEY = 'your_api_key_here'
# Define the URL for the API endpoint
url = "https://api.scale.com/v1/batches/kitten_labeling_2020-07"
# Set up the headers for the request
headers = {
"accept": "application/json" # Specify that we want the response in JSON format
}
# Adding authentication to the GET request
# The auth parameter requires a tuple with the API key and an empty string
response = requests.get(url, headers=headers, auth=(API_KEY, ''))
# Print the response text to see the result
print(response.text)
```
```python Python SDK theme={null}
import scaleapi
# Initialize the ScaleClient with your API key
client = scaleapi.ScaleClient("YOUR_API_KEY_HERE")
# Define the batch name to retrieve
batch_name = "kitten_labeling_2020-07"
# Retrieve the batch details
batch = client.get_batch(batch_name=batch_name)
# Print the batch details
print(batch.as_dict())
```
```json theme={null}
{
"project": "PROJECT-NAME",
"name": "BATCH_NAME",
"callback": "your@email.com",
"status": "in_progress",
"created_at": "2023-05-16T19:02:23.149Z",
"metadata": {}
}
```
# Batch Status
This endpoint returns the status of a batch with the name :batchName, as well as the counts of its tasks grouped by task status.
Required batchName to get status.
***
```python Python theme={null}
import requests
# Replace with your actual API key
API_KEY = 'your_api_key_here'
# Define the URL for the API endpoint
url = "https://api.scale.com/v1/batches/kitten_labeling_2020-07/status"
# Set up the headers for the request
headers = {
"accept": "application/json" # Specify that we want the response in JSON format
}
# Adding authentication to the GET request
# The auth parameter requires a tuple with the API key and an empty string
response = requests.get(url, headers=headers, auth=(API_KEY, ''))
# Print the response text to see the result
print(response.text)
```
```python Python SDK theme={null}
import scaleapi
# Initialize the ScaleClient with your API key
client = scaleapi.ScaleClient("YOUR_API_KEY_HERE")
# Define the batch name to retrieve the status
batch_name = "kitten_labeling_2020-07"
# Retrieve the batch status using the direct method
batch_status = client.batch_status(batch_name=batch_name)
print(batch_status)
# Alternative method to retrieve the batch status
batch = client.get_batch(batch_name=batch_name)
batch.get_status() # Refreshes tasks_{status} attributes of Batch
print(f"Tasks Pending: {batch.tasks_pending}, Tasks Completed: {batch.tasks_completed}")
```
```json theme={null}
{
"status": "in_progress",
"tasks_pending": 9,
"tasks_completed": 1
}
```
# List All Batches
This is a paged endpoint for all of your batches. Batches will be returned in descending order based on created\_at. Pagination is based off limit and offset parameters, which determine the page size and how many results to skip.
Project name to filter batches by.
***
Status to filter batches by (staging or in\_progress or completed).
***
Get details about the progress of the batches.
***
The minimum value of created\_at for batches to be returned
***
The maximum value of created\_at for batches to be returned
***
```python Python theme={null}
import requests
# Replace with your actual API key
API_KEY = 'your_api_key_here'
# Define the URL for the API endpoint with query parameters
url = "https://api.scale.com/v1/batches?project=kitten_labeling&status=in_progress&detailed=false&start_time=2020-05-21&end_time=2021-01-01&limit=100&offset=0"
# Set up the headers for the request
headers = {
"accept": "application/json" # Specify that we want the response in JSON format
}
# Adding authentication to the GET request
# The auth parameter requires a tuple with the API key and an empty string
response = requests.get(url, headers=headers, auth=(API_KEY, ''))
# Print the response text to see the result
print(response.text)
```
```python Python SDK theme={null}
import scaleapi
# Initialize the ScaleClient with your API key
client = scaleapi.ScaleClient("YOUR_API_KEY_HERE")
# Define optional filters
project_name = "project_name" # Filter by project name (optional)
batch_status = "in_progress" # Filter by status (optional)
exclude_archived = True # Exclude archived batches (optional)
created_after = "2023-01-01T00:00:00Z" # Filter by start time (optional)
created_before = "2023-12-31T23:59:59Z" # Filter by end time (optional)
# Retrieve the list of all batches with optional filters
batches = client.get_batches(
project_name=project_name,
batch_status=batch_status,
exclude_archived=exclude_archived,
created_after=created_after,
created_before=created_before
)
# Print the details of each batch
for batch in batches:
print(batch.as_dict())
```
```json theme={null}
{
"completed_at": "2023-02-02T10:17:35.379Z",
"created_at": "2023-02-02T10:17:35.379Z",
"metadata": {},
"name": "BATC_NAME",
"project": "PROJECT_NAME",
"status": "in_progress"
}
```
# Batch Priorization
This endpoint updates the priority of a batch.
The batch priority should follow the same parameters as an individual task's priority, namely that priority should be between 10 for the lowest and 30 for the highest priority.
Setting a task's priority will impact the order in which the task is first picked up, but does not guarantee the order in which a task or set of tasks will be returned to you. As a result, tasks that are not yet started can be reprioritized, but tasks that are already started will not be impacted
The name of the batch to update.
***
The new priority for the batch. The priority should be between 10, representing the lowest priority, and 30, representing the highest priority.
***
```python theme={null}
import requests
# Replace with your actual API key
API_KEY = 'your_api_key_here'
# Define the URL for the API endpoint
url = "https://api.scale.com/v1/batches/kitten_labeling_2020-07/prioritize"
# Define the payload to update the batch priority
payload = {
"priority": 10 # Set the priority level
}
# Set up the headers for the request
headers = {
"accept": "application/json", # Specify that we want the response in JSON format
"content-type": "application/json" # Specify the content type of the request
}
# Adding authentication to the POST request
# The auth parameter requires a tuple with the API key and an empty string
response = requests.post(url, json=payload, headers=headers, auth=(API_KEY, ''))
# Print the response text to see the result
print(response.text)
```
```json theme={null}
{
"result": "success"
}
```
# Callbacks
Source: https://api-reference.scale.com/docs/api-reference/callbacks
Callbacks
# Callbacks
On your tasks, you can optionally supply a `**callback_url**`, a fully qualified URL that we will POST with the results of the task when completed. The data will be served as a JSON body (`**application/json**`). Alternately, you can set a default callback URL in your profile, which will be used for tasks that do not specify one.
Additionally, in order to simplify testing and add support for email automation pipelines, you may provide an **email address** as the `**callback_url**`. In this case, each completed task will result in an email sent from `**[hello@scale.ai](mailto:hello@scale.ai)**` with the body as the task's JSON payload.
You should respond to the POST request with a 2xx status code. If we do not receive a 2xx status code, we will continue to retry up to 20 times over the course of the next 24 hours.
If we receive a 2xx status code, the task will be populated with a `**true**` value for the `**callback_succeeded**` parameter. Otherwise, if we do not receive a 2xx status code on any retry, the task will be populated with a `**false**` value for the `**callback_succeeded**` parameter.
```json theme={null}
{
"task": {
"task_id": "576c41bf13e36b0600b02b34",
"completed_at": "2016-06-23T21:54:44.904Z",
"response": {
"category": "red"
},
"created_at": "2016-06-23T20:08:31.573Z",
"callback_url": "http://www.example.com/callback",
"type": "categorization",
"status": "completed",
"instruction": "Is this object red or blue?",
"params": {
"attachment_type": "text",
"attachment": "tomato",
"categories": [
"red",
"blue"
]
},
"metadata": {}
},
"response": {
"category": "red"
},
"task_id": "576c41bf13e36b0600b02b34"
}
```
## Getting Started
If you're just testing and want to try a few requests, the easiest way to get started is to use a **[RequestBin](http://requestbin.com/)** and send requests using the provided URL as the `**callback_url**`. You can also use **[ngrok](https://ngrok.com/)** to expose a local server to the internet for fast prototyping.
We've also found **[Pipedream](https://pipedream.com/)** to be an easy-to-use platform to receive webhooks, view logs, take other actions.
### Authentication
If you'd like to authenticate our callbacks, we set a `**scale-callback-auth**` HTTP header on each of our callbacks. The value will be equal to your `**Live Callback Auth Key**` shown on your dashboard. If this header is not set, or it is set incorrectly, the callback is not from Scale.
### Events that trigger a Callback
Callbacks are sent for the following events:
* Error on Task Creation (see **[Errors](/docs/api-reference/errors)** for more details)
* Task Completion
* Audit Status Changes (Approved, Rejected, Fixed)
* Tasks that are "Recalled" by Scale, meaning an operation on Scale's side converts `**completed**` tasks back to `**pending**` so they can have follow-on work done on them. This conversion is coordinated and communicated with you as the customer if it needs to happen.
### POST Data
| **Property** | **Type** | **Description** |
| ------------ | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **task\_id** | string | The `**task_id**` is the unique identifier for the task. It is identical to `**task.task_id**` |
| **status** | string | The status of the task when it was completed. Normally `**completed**`, but can also be `**error**` in the case that a task failed to process. It is identical to `**task.status**` |
| **response** | object | The response object of the completed request. It is identical to task.response |
| **task** | object | The full **Task Object** for reference and convenience. |
| | | |
## Re-sending a Callback
This endpoint re-sends a callback for a completed or errored task to the `**callback_url**`.If your callback server had gone down, or otherwise missed receiving a task, this endpoint will tell Scale to resend the callback's original data (above) to your server.
```python theme={null}
INSTALLATION
$ python -m pip install requests
---
import requests
url = "https://api.scale.com/v1/task/taskId/send-callback"
headers = {"accept": "application/json"}
response = requests.post(url, headers=headers)
print(response.text)
```
# Create Multi-Stage Task
Source: https://api-reference.scale.com/docs/api-reference/create-multi-stage-task
Create Multi-Stage Task
# Create Multi-Stage Task
This endpoint creates a `multistage` task. Multi-stage tasks are designed to handle complex full-scene labeling that spans multiple annotation types and modalities, and serves as a replacement for Scale’s legacy dependent tasks system (which requires using multiple tasks to fully label a scene).
Use cases for multi-stage tasks include but are not limited to:
* Linking 3D cuboids to their 2D bounding box/polygon projections
* Linking 3D cuboids to relevant top-down annotations
* Conditional labeling (e.g. categorizing scenes based on localization quality to determine whether to label them)
The required parameters for this task are project, attachments, and scene\_format.
The callback\_url is the URL which will be POSTed on task completion, and is described in more detail in the Callback section.
Scale supports the following attachment types for multi-stage tasks:
* (Recommended) Sensor Fusion Scene (.SFS)
* LiDAR frames (.JSON)
The name of the project to associate this task with. See the [Projects](/docs/api-reference/projects) Section for more details.
***
The name of the batch to associate this task with. Note that if a batch is specified, you need not specify the project, as the task will automatically be associated with the batch's project. For Scale Rapid projects specifying a batch is required. See [Batches section](/docs/api-reference/batches) for more details.
***
A markdown-enabled string or iframe embed google doc explaining how to do the task. You can use [markdown](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet) to show example images, give structure to your instructions, and more.
***
The full url (including the scheme `http://` or `https://`) of the callback when the task is completed. See the [Callback section](/docs/api-reference/callbacks) for more details about callbacks.
***
URLs to the scene attachments. Can be a single URL for `.SFS` attachments, or one URL per frame for `.JSON` attachments.
***
Describes what type of file the attachment is. Can be `sensor_fusion` for `.SFS` attachments or `split_base64` for `.JSON` attachments
The ID of the blueprint that you would like to use when creating the multi-stage task. If no blueprint is specified, Scale will default to using the project’s most recent active blueprint.
A set of key/value pairs that you can attach to a task object. It can be useful for storing additional information about the task in a structured format. Max 10KB.
***
A arbitrary ID that you can assign to a task and then query for later. This ID must be unique across all projects under your account, otherwise the task submission will be rejected. See [Avoiding Duplicate Tasks](/docs/api-reference/tasks)[ ](/docs/api-reference/tasks)for more details.
***
A value of 10, 20, or 30 that defines the priority of a task within a project. The higher the number, the higher the priority.
***
If set to be true, if a task errors out after being submitted, the unique id on the task will be unset. This param allows workflows where you can re-submit the same unique id to recover from errors automatically
***
Arbitrary labels that you can assign to a task. At most 5 tags are allowed per task. You can query tasks with specific tags through the task retrieval API.
***
```python theme={null}
import requests
# Replace with your actual API key
API_KEY = "your_api_key_here"
# Define the URL for the API endpoint
url = "https://api.scale.com/v1/task/multistage"
# Define the payload for the multi-stage task
payload = {
"project": "your_project_name_here"
"instruction": "Annotate the *vehicles* and *pedestrians* in the scene.",
"callback_url": "https://example.com/callback",
"attachments": ["https://static.scale.com/uploads/pandaset-demo/scene.sfs"],
"scene_format": "sensor_fusion",
"priority": None
}
# Set up the headers for the request
headers = {
"accept": "application/json", # Specify that we want the response in JSON format
"content-type": "application/json" # Specify the content type of the request
}
# Adding authentication to the POST request
# The auth parameter requires a tuple with the API key and an empty string
response = requests.post(url, json=payload, headers=headers, auth=(API_KEY, ''))
# Print the response text to see the result
print(response.text)
```
```jsx theme={null}
{
"callback_url": "http://www.example.com/callback",
"created_at": "2024-01-16T21:03:33.166Z",
"instruction": "Annotate the *vehicles* and *pedestrians* in the scene.",
"is_test": false,
"params": {},
"status": "pending",
"task_id": "5a99e20de50d4979ce6d291e",
"type": "multistage"
}
```
# Errors
Source: https://api-reference.scale.com/docs/api-reference/errors
API Errors
# API Errors
### Attachment Processing Errors
In the event of one or more task attachments having invalid data, the task callback will be invoked with an error response detailing the problems, and the task's `**status**` will be set to `**error**`. Additional detail about the error and failed attachments will be stored in the task response.
Scale's API uses the following HTTP codes:
| **Error Code** | **Meaning** |
| -------------- | -------------------------------------------------------------------------------------------------------------------- |
| 200 | OK -- Everything worked as expected. |
| 400 | Bad Request -- The request was unacceptable, often due to missing a required parameter. |
| 401 | Unauthorized -- No valid API key provided. |
| 402 | Not enabled -- Please contact **[sales@scaleapi.com](mailto:sales@scaleapi.com)** before creating this type of task. |
| 404 | Not Found -- The requested resource doesn't exist. |
| 409 | Conflict -- The provided idempotency key or `**unique_id**` is already in use for a different request. |
| 429 | Too Many Requests -- Too many requests hit the API too quickly. |
| 500 | Internal Server Error -- We had a problem with our server. Try again later. |
### Example Error Formats
These examples help you understand the structure and content of error messages returned by the API in different situations.
* "Please include an attachment or attachments parameter":
This error occurs when you make a request to the API without including any attachment or attachments parameter. It indicates that you need to provide either a single attachment or an array of attachments in your API request for it to be processed successfully.
* "One or more attachments could not be downloaded":
This error message indicates that there was an issue with downloading one or more attachments specified in your API request. It suggests that there might be connectivity issues or problems with the specified attachment URLs. You may need to ensure that the attachment URLs are valid and accessible for the Scale API to download and process them.
* "One or more attachments could not be downloaded (specific attachment errors)":
This error is similar to the previous one but provides more specific information about the attachments that couldn't be downloaded. It might include additional details such as the URLs or identifiers of the problematic attachments. This information helps you identify and troubleshoot the specific attachments that failed to download.
```json theme={null}
{
"status_code": 400,
"error": "Please include an attachment or attachments parameter."
}
```
```json theme={null}
// example task.response with failed attachment due to access issues
{
"error": "One or more attachments could not be downloaded.", // reason for error
"attachments": [
{
"statusCode": 403, // HTTP code received when fetching attachment
"url": "http://example.com/kitten.png", // attachment URL
}
]
}
```
```json theme={null}
// example task.response with failed attachment processing
{
"error": "One or more attachments could not be downloaded.", // reason for error
"frames": [ // frames which failed processing
{
"frame": 10, // frame #
"message": "Invalid JSON Format: Unexpected end of JSON input",
"attachment": {
"statusCode": 200, // HTTP code received when fetching attachment
"url": "http://example.com/lidar/frame10.json", // attachment URL
}
}
]
}
```
# Fixless Audits
Source: https://api-reference.scale.com/docs/api-reference/fixless-audits
Fixless Audits Fixless Audits improve the quality and accuracy of task auditing by allowing auditors to provide feedback without making any fixes. Fixless Audits are quicker and easier to create when compared to standard
# Fixless Audits
Fixless Audits improve the quality and accuracy of task auditing by allowing auditors to provide feedback without making any fixes. Fixless Audits are quicker and easier to create when compared to standard audits where customers fix tasks.
These audits can be created through the API or in Scale’s LidarLite auditing tool. Fixless Audits submitted via the API can be reviewed and adjusted in LidarLite, allowing auditors to add, delete, or edit Feedback Items as needed.
This guide walks through Fixless Audit creation and review using the API.
## Create a Fixless Audit
Fixless Audits first require some basic information: the relevant task ID, audit result, and the audited task response URL. When you create a fixless audit via the API, you’ll need to create an audit payload in the format outlined in this documentation.
## Retrieve a Fixless Audit
Fixless audits can be retrieved via our endpoint, and are structured in the same format as when you create an audit. An example response format has been included below the Creation Payload below.
```text theme={null}
import requests
# Replace with your actual API key
API_KEY = 'your_api_key_here'
# Set up the headers for the request
headers = {
"accept": "application/json" # Specify that we want the response in JSON format
}
# Create an audit
body = {...} # FixlessAuditCreate
url = "https://api.scale.com/v1/audits"
response = requests.put(url, headers=headers, auth=(API_KEY, ''), json=body) # 201 response FixlessAudit
print(response.text)
# Get audits
url = "https://api.scale.com/v1/audits/?task_id=xxx&id=yyy" # provide task id or audit id
response = requests.get(url, headers=headers, auth=(API_KEY, '')) # 200 response FixlessAudit[]
print(response.text)
```
## Feedback Items
Fixless Audits primarily consist of Feedback Items, which indicate errors, comments, confirmations, or flags on the task.
Fixless Audits can be submitted without any Feedback Items. This is a valid audit and indicates that the task has no errors.
### Feedback Item Type (required)
We use `type` to indicate the purpose of the Feedback Item. The most common type is `error` which indicates a mistake on the task. Other types: `flag`, `confirmation`, `comment`, are less common and are not fixed by Scale or considered in quality score calculations.
* `Error`: The annotation is incorrect
* `Flag`: The annotation needs to be reviewed by a customer auditor, does not get fixed by Scale, and does not affect quality scores.
* `Confirmation`: The annotation is correct, does not affect quality scores
* `Comment`: Generic feedback, does not affect quality scores
### Feedback Item Category (required)
`category` indicates the kind of issue found on the task.
* `missing` - indicates that an annotation is missing. This is also referred to as a false negative.
* `extraneous` - indicates that the annotation should **not** be present. This is also referred to as a false positive.
* `geometry` - indicates that the annotation has the wrong dimensions or shape
* `position` - indicates that the annotation is in the wrong place
* `attribute` - indicates that the annotation has an incorrect attribute property
* `label` - indicates that the annotation label (also referred to as class) is incorrect
**Important**: All error categories except for `missing` are specific to an `annotation_id`. `missing`errors require a `point`/ `polygon` and a `missing_annotation_type`to indicate what type of annotation is missing and where it should be.
### Feedback Item Scope (required)
`scope` indicates the location and duration of the issue. We have 4 different schemas for Feedback Item Scopes, which apply to different error categories. Details on typing and required properties can be found in the \*\*Create Fixless Audit \*\*section, below.
* `FeedbackItemScopeAnnotation` - indicates the annotation ID, frame interval, and attribute where the issue is present. This schema is used when the category is: `extraneous` , `geometry`, `position`, `attribute`, or `label`
* `FeedbackItemScopePolygon` - indicates the 3D or 2D position and frame interval where the issue is present. This schema is used when the category is: `missing`
* `FeedbackItemScopePoint` - indicates the 3D or 2D position and frame interval where the issue is present. This schema is used when the category is: `missing`
* `FeedbackItemScopeScene` - indicates that the issue is related to a scene attribute (not an individual annotation). This schema is used when the category is: `attribute`
### Feedback Item Severity (optional - defaults to standard)
`severity` indicates the seriousness of the issue. Severity can be `mild`, `standard` or `severe`.
* `Mild`: Used when the auditor wants to flag an error, but the error doesn’t exceed our SLA error threshold (e.g. cuboid position is off by \<30cm). \*\* \*\*These errors **will not** affect the quality score
* `Standard:` Used when the auditor wants to flag an error that exceeds our SLA error threshold (e.g. cuboid position is off by >30cm). These errors will affect the quality score
* `Severe`: Used when the auditor wants to flag an error that exceeds our SLA error threshold (e.g. cuboid position is off by >30cm) and the error is critical and requires special attention. These errors will affect the quality score
### Feedback Item Description and Metadata (optional)
`description` is an open text field shown in the tasking and auditing UI. This field can be used to describe the error or fix.
`metadata` is an object that can be used to store data for internal tracking. For instance, in metadata you might store `metadata.is_verified_by_human: true`and `metadata.confidence_level: 0.83` if you were using an automated system to generate Feedback Items.
### Feedback Item State
`state` does \*\*not \*\*need to be included in the Fixless Audit creation payload. Newly created Feedback Items are defaulted to `state: open`. Feedback Item states may change as Scale reviews and fixes tasks.
* `open` - default state. Indicates the issue has not been fixed yet
* `resolved` - indicates the issue has been fixed
* `disputed` - indicates that Scale disagrees with the issue
* `escalated` - indicates the customer disagrees with Scale’s dispute of the issue
* `rejected` - terminal state indicating that the issue was incorrectly reported after review of the escalation
```text theme={null}
FixlessAuditCreate {
type: 'fixless';
result: 'accepted' | 'rejected';
task_id: string;
comments?: string;
target_response_url: string; // from task.response.annotations.url (task.response.ortho for LidarTopdown tasks)
feedback_items?: FeedbackItemCreate[];
metadata?: { [key: string]: any };
}
FeedbackItemCreate {
type: 'comment' | 'error' | 'flag' | 'confirmation';
scope: FeedbackItemScope;
category:
| 'attribute'
| 'extraneous'
| 'geometry'
| 'label'
| 'missing'
| 'position';
severity?: 'mild' | 'standard' | 'severe';
description?: string;
metadata?: { [key: string]: any };
}
FeedbackItemScope =
| FeedbackItemScopeAnnotation
| FeedbackItemScopePolygon
| FeedbackItemScopePoint
| FeedbackItemScopeScene;
// extraneous, geometry, label, position errors should use this scope
// most attribute errors should use this scope. The exception is scene attributes
FeedbackItemScopeAnnotation = {
type: 'annotation';
annotation_id: string;
sensor_id?: string | number; // defaults to the scene's primary sensor (typically the first lidar). Typically used to identify a specific camera for 2D projections
attribute?: string; // attribute name. defined iff scoped on an annotation attribute
interval: FeedbackItemTimestampRange;
}
// Only for missing annotation errors
FeedbackItemScopePolygon {
type: 'polygon';
vertices: [[x0, y0], [x1, y1], [x2, y2], ...];
missing_annotation_type: AnnotationType; // type of missing annotation. see enum definition below
stationary: boolean; // only applies for cuboids and indicates if cuboid is stationary or dynamic. This is important for quality score calculations
missing_annotation_class?: string; // label name for missing annotation
sensor_id?: string; // indicate camera sensor id iff vertices are in camera coordinates. Otherwise, we assume 3d coordinates (z-value assumed to be 0)
interval: FeedbackItemTimestampRange;
}
// Only for missing annotation errors
FeedbackItemScopePoint {
type: 'point';
coordinates: [x, y];
missing_annotation_type: AnnotationType; // type of missing annotation. see enum definition below
stationary: boolean; // only applies for cuboids and indicates if cuboid is stationary or dynamic. This is important for quality score calculations
missing_annotation_class?: string; // label name for missing annotation
sensor_id?: string; // indicate camera sensor id iff coordinates are in camera coordinates. Otherwise, we assume 3d coordinates (z-value assumed to be 0)
interval: FeedbackItemTimestampRange;
}
// only used for scene attribute errors
FeedbackItemScopeScene {
type: 'scene';
attribute?: string; // scene attribute name
interval: FeedbackItemTimestampRange;
}
FeedbackItemInterval = {
type: 'frame_range';
start: number; // frame index
end: number; // frame index
};
// missing annotation types for FeedbackItemScopePoint and FeedbackItemScopePolygon
AnnotationType =
// 3D - task types: sensorFusion, multiStage
'cuboid' |
// 2D - task types: videoAnnotation, multiStage
'box_2d' |
'polygon_2d' |
'polyline_2d' |
'point_2d' |
'event'
// LTD - task types: lidarTopdown, multiStage
'polygon_topdown' |
'polyline' |
'point_topdown';
```
```text theme={null}
FixlessAudit {
id: string;
srn: 'srn:scale:avcv:audit:{{id}}' // SRNs can be used in place of id on endpoints
type: 'fixless';
result: 'accepted' | 'rejected';
task_id: string;
comments?: string;
target_response: { url: string };
feedback_items?: FeedbackItem[];
metadata?: { [key: string]: any };
active: boolean; // whether this is the latest audit
source: 'api' | 'lidarlite' | 'classic';
created_by: string;
created_at: iso_date_string;
updated_at: iso_date_string;
}
```
## Calculating Quality Scores
Scale only uses Feedback Items of `type: error` and `severity: standard | severe` to compute quality scores.
Only the latest Fixless Audit is considered when computing task and batch quality scores.
Invalid Feedback Items are not considered. Invalid feedback items are marked in the `grader_output` property on the Feedback Item. Grader outputs are computed after the Fixless Audit is submitted and may not be available for a few minutes:
* `grader_output.conflict = true`
* `grader_output.explanation = "description of the issue..."`
Below are some examples of situations in which a Feedback Item is marked as invalid:
* Extraneous, attribute, geometry, position, or label error where a valid annotation ID is not provided
* Attribute error where the attribute name does not match an attribute on the annotation
* Missing error where a point or polygon indicating the location of the error is not specified
# GenAI Data Engine
Source: https://api-reference.scale.com/docs/api-reference/genai-data-engine
Scale Generative AI Data Engine enables rapid creation of tailored, high-quality datasets curated by vetted subject matter experts to train the world’s most advanced models. Access customized data annotations, model eval
# Getting Started
Scale Generative AI Data Engine enables rapid creation of tailored, high-quality datasets curated by vetted subject matter experts to train the world’s most advanced models. Access customized data annotations, model evaluations, and RLHF data via API, SDK, or web frontend.
Explore Data Engine integrations in our [Gen AI Data Engine Documentation](https://docs.genai.scale.com).
# GenAI Platform
Source: https://api-reference.scale.com/docs/api-reference/genai-platform
The Scale GenAI Platform empowers modern enterprises to rapidly develop, test and deploy Generative AI applications for custom use cases, using their proprietary data assets. It includes an API, SDK and web frontend whic
# Getting Started
The Scale GenAI Platform empowers modern enterprises to rapidly develop, test and deploy Generative AI applications for custom use cases, using their proprietary data assets. It includes an API, SDK and web frontend which abstract the flexible use of both open and closed-source resources, providing full-stack capabilities that meet enterprise security and scalability standards.
Explore more in our [GenAI Platform Documentation](https://docs.gp.scale.com/home).
# Image & Video Reference
Source: https://api-reference.scale.com/docs/api-reference/image-and-video-reference
Image Annotation Overview Boxes Polygons Lines Ellipses Cuboids Image Response Format Image Annotation Hypothesis Video Annotation Overview Label Nesting and Options
# Image Annotation Overview
This is the recommended task type for annotating images with vector geometric shapes. The available geometries are `box`, `polygon`, `line`, `point`, `cuboid`, and `ellipse`. This endpoint creates an `imageannotation` task. Given an image, Scale will annotate the image with the geometries you specify. The required parameters for this task are `attachment` and `geometries`.
The name of the [project](/docs/api-reference/projects) to associate this task with.
***
The name of the [batch](https://docs.scale.com/reference/batch-overview) to associate this task with. Note that if a batch is specified, you need not specify the project, as the task will automatically be associated with the batch's project. For Scale Rapid projects specifying a batch is required. See [Batches section](https://docs.scale.com/reference/batch-overview) for more details.
***
A markdown-enabled string or iframe embedded Google Doc explaining how to do the task. You can use [markdown](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet) to show example images, give structure to your instructions, and more. See our [instruction best practices](https://scale.com/docs/instructions) for more details. For Scale Rapid projects, DO NOT set this field unless you specifically want to override the project level instructions.
***
The full url (including the scheme `http://` or `https://`) or email address of the [callback](https://docs.scale.com/reference/callbacks) that will be used when the task is completed.
***
A URL to the image you'd like to be annotated.
***
An array of objects in the form of \{"attachment": "\"} to show to taskers as a reference. Context images themselves can not be labeled. Context images will appear [like this](https://i.imgur.com/MJ7ZbMt.mp4) in the UI. You cannot use the task's attachment url as a context attachment's url.
***
This object is used to define which objects need to be annotated and which annotation geometries (`box`, `polygon`, `line`, `point`, `cuboid`, or `ellipse`) should be used for each annotation. Further description of each geometry can be found in each respective section below
***
This field is used to add additional attributes that you would like to capture per annotation. See [Annotation Attributes](https://docs.scale.com/reference/attributes-overview) for more details about annotation attributes.
***
Use this field to define links between annotations. See [Links](https://docs.scale.com/reference/links) for more details about links.
***
Editable annotations that a task should be initialized with. This is useful when you've run a model to prelabel the task and want annotators to refine those prelabels. Must contain the `annotations` field, which has the same format as the `annotations` field in the response.
***
Read-only annotations to be pre-drawn on the task. See the [Layers](https://docs.scale.com/reference/image-layers) section for more details.
***
Editable annotations, with the option to be "locked", that a task should be initialized with. This is useful when you've run a model to prelabel the task and want annotators to refine those prelabels. Must contain the `annotations` field, which has the same format as the `annotations` field in the response.
***
Whether or not new annotations can be added to the task if base\_annotations are used. If set to true, new annotations can be added to the task in addition to base\_annotations. If set to false, new annotations will not be able to be added to the task.
***
Whether or not base\_annotations can be edited in the task. If set to true, base\_annotations can be edited by the tasker (position of annotation, attributes, etc). If set to false, all aspects of base\_annotations will be locked.
***
Whether or not base\_annotations labels can be edited in the task. If set to true, the label of base\_annotations can be edited by the tasker. If set to false, the label will be locked.
***
Whether or not base\_annotations can be removed from the task. If set to true, base\_annotations can be deleted from the task. If set to false, base\_annotations cannot be deleted from the task.
***
This field accepts specified image metadata, supported fields include: - `date_time` - displays the date and time the image is taken - `resolution` - configures the units of the ruler tools, `resolution_ratio` holds the number of `resolution_unit`s corresponding to one pixel; e.g. `\{resolution_ratio: 3, resolution_unit: 'm'\}`, one pixel in the image corresponds to three meters in the real world. - `location` - the real-world location where this image was captured, in the standard geographic coordinate system; e.g. `\{lat: 37.77, long: -122.43\}`
***
A set of key/value pairs that you can attach to a task object. It can be useful for storing additional information about the task in a structured format. Max 10KB. See the [Metadata](https://docs.scale.com/reference/metadata) section for more detail.
***
***
The amount of padding in pixels added to the left and right of the image. Overrides `padding` if set.
***
The amount of padding in pixels added to the top and bottom of the image. Overrides `padding` if set.
***
A value of 10, 20, or 30 that defines the priority of a task within a project. The higher the number, the higher the priority.
***
A arbitrary ID that you can assign to a task and then query for later. This ID must be unique across all projects under your account, otherwise the task submission will be rejected. See [Avoiding Duplicate Tasks](https://docs.scale.com/reference/idempotent-requests) for more details.
***
If set to be true, if a task errors out after being submitted, the unique id on the task will be unset. This param allows workflows where you can re-submit the same unique id to recover from errors automatically
***
Arbitrary labels that you can assign to a task. At most 5 tags are allowed per task. You can query tasks with specific tags through the task retrieval API.
***
```python theme={null}
import requests
url = "https://api.scale.com/v1/task/imageannotation"
payload = {
"instruction": "**Instructions:** Please label all the things",
"attachment": "https://i.imgur.com/iDZcXfS.png",
"geometries": {
"box": {
"min_height": None,
"min_width": None,
"can_rotate": None,
"integer_pixels": None
},
"polygon": {
"min_vertices": None,
"max_vertices": None
},
"line": {
"min_vertices": None,
"max_vertices": None
},
"cuboid": {
"min_height": None,
"min_width": None,
"camera_intrinsics": {
"fx": None,
"fy": None,
"cx": None,
"cy": None,
"skew": None,
"scalefactor": None
},
"camera_rotation_quaternion": {
"w": None,
"x": None,
"y": None,
"z": None
},
"camera_height": None
}
},
"padding": None,
"paddingX": None,
"paddingY": None,
"priority": None
}
headers = {
"accept": "application/json",
"content-type": "application/json"
}
response = requests.post(url, json=payload, headers=headers)
print(response.text)
```
```text theme={null}
{
"task_id": "string",
"created_at": "string",
"type": "imageannotation",
"status": "pending",
"instruction": "string",
"is_test": false,
"urgency": "standard",
"metadata": {},
"project": "string",
"callback_url": "string",
"updated_at": "string",
"work_started": false,
"params": {
"attachment_type": "image",
"attachment": "http://i.imgur.com/3Cpje3l.jpg",
"geometries": {
"box": {
"objects_to_annotate": [
null
],
"min_height": 5,
"min_width": 5
},
"polygon": {
"objects_to_annotate": [
null
]
},
"point": {
"objects_to_annotate": [
null
]
}
},
"annotation_attributes": {
"additionalProp": {
"type": "category",
"description": "string",
"choice": "string"
}
}
}
}
```
# Boxes
Given a `box` entry in `params.geometries`, Scale will annotate your image or video with boxes and return the position and dimensions of the boxes.
A list of `string` or **[LabelDescription](https://docs.scale.com/reference/label-nesting)** objects.
***
The minimum height in pixels of the bounding boxes you'd like to be made.
***
The minimum width in pixels of the bounding boxes you'd like to be made.
***
Allows a tasker to rotate the bounding box.
***
Response fields denoting box location and size (`top`, `left`, `width`, `height`) will be returned as integers instead of floats. This does not work with rotated boxes.
***
### Response Fields
| Key | Type | Description |
| ---------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| uuid | string | A computer-generated unique identifier for this annotation. In video annotation tasks, this can be used to track the same object across frames. |
| type | string | String indicating geometry type: `box` |
| label | string | The label of this annotation, chosen from the `objects_to_annotate` array for its geometry. In video annotation tasks, any annotation objects with the same `uuid` will have the same `label` across all frames. |
| attributes | object | See the [Annotation Attributes](https://docs.scale.com/reference/attributes-overview) section for more details about the `attributes` response field. |
| left | float | The distance, in pixels, between the left border of the bounding box and the left border of the image. |
| top | float | The distance, in pixels, between the top border of the bounding box and the top border of the image. |
| width | float | The width, in pixels, of the bounding box. |
| height | float | The height, in pixels, of the bounding box. |
If `can_rotate` was set to `true`, the following fields will supersede the above fields:
| Key | Type | Description |
| -------- | ----------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| rotation | float | The clockwise rotation in radians |
| vertices | An array of objects with a schema \{x: 0, y: 0} | The vertices of the rotated bounding box |
| left | float | The distance, in pixels, between the left border of the unrotated bounding box and the left border of the image. |
| top | float | The distance, in pixels, between the top border of the unrotated bounding box and the top border of the image. |
```json theme={null}
{
"geometries": {
"box": {
"objects_to_annotate": [
"traffic_sign",
{
"choice": "vehicle",
"subchoices": [
"Car",
{
"choice": "truck_suv",
"display": "truck or SUV"
}
]
},
"pedestrian"
],
"min_height": 5,
"min_width": 5,
"can_rotate": false
},
...
},
...
}
```
```json theme={null}
{
"response": {
"annotations": [
{
"type": "box",
"label": "pedestrian",
"attributes": {
"moving": "yes"
},
"left": 2,
"top": 4,
"width": 3,
"height": 5,
"uuid": "65ec1f52-5902-4b39-bea9-ab6b4d58ef42"
},
{
"type": "box",
"label": "car",
"attributes": {
"moving": "yes"
},
"left": 7,
"top": 5,
"width": 14,
"height": 5,
"uuid": "0a6cd019-a014-4c67-bd49-c269ba08028a"
},
{ ... },
{ ... }
]
},
"task_id": "5774cc78b01249ab09f089dd",
"task": {
// populated task for convenience
...
}
}
```
```json theme={null}
{
"response": {
"annotations" : [
{
"label" : "car",
"attributes" : {},
"uuid" : "122a4270-f9b2-4f66-a9ca-2e06f0de66e5",
"width" : 121.878523862864,
"height" : 71.6961921895555,
"rotation" : 1.2440145049532,
"left" : 613.440037825633,
"top" : 199.208745812549,
"type" : "box",
"vertices" : [
{
"x" : 688.769014855216,
"y" : 165.835344251165
},
{
"x" : 727.891633787782,
"y" : 281.264089660824
},
{
"x" : 659.989584658913,
"y" : 304.27833956349
},
{
"x" : 620.866965726348,
"y" : 188.84959415383
}
]
}
{ ... },
{ ... }
]
},
"task_id": "5774cc78b01249ab09f089dd",
"task": {
// populated task for convenience
...
}
}
```
# Polygons
Given a `polygon` entry in `params.geometries`, Scale will annotate your image or video with polygons and return the vertices of the polygons.
A list of `string` or **[LabelDescription](https://docs.scale.com/reference/label-nesting)** objects.
***
The minimum number of vertices in a valid line annotation for your request.
***
The maximum number of vertices in a valid line annotation for your request. Must be at least `min_vertices`.
***
### Response Fields
| Key | Type | Description |
| ---------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| uuid | string | A computer-generated unique identifier for this annotation.
In video annotation tasks, this can be used to track the same object across frames. |
| type | string | String to indicate geometry type: `polygon` |
| label | string | The label of this annotation, chosen from the `objects_to_annotate` array for its geometry. In video annotation tasks, any annotation objects with the same `uuid` will have the same `label` across all frames. |
| attributes | object | See the [Annotation Attributes](https://docs.scale.com/reference/attributes-overview) section for more details about the `attributes` response field. |
| vertices | array | An array of vertex objects describing the vertices of the polygon, listed in the order they were annotated. In other words, the point order will be either clockwise or counter-clockwise for each annotation. |
\*\*Definition: \*\*`Vertex`
| Key | Type | Description |
| --- | ------ | ----------------------------------------------------------------------------- |
| x | number | The distance, in pixels, between the vertex and the left border of the image. |
| y | number | The distance, in pixels, between the vertex and the top border of the image. |
```json theme={null}
{
"geometries": {
"polygon": {
"objects_to_annotate": [
"traffic_sign",
{
"choice": "vehicle",
"subchoices": [
"Car",
{
"choice": "truck_suv",
"display": "truck or SUV"
}
]
},
"pedestrian"
],
"min_vertices": 4,
"max_vertices": 15
},
...
},
...
}
```
```json theme={null}
{
"response": {
"annotations": [
{
"type": "polygon",
"label": "car",
"vertices": [
{
"x": 123,
"y": 10
},
{
"x": 140,
"y": 49
},
{
"x": 67,
"y": 34
}
],
"uuid": "65ec1f52-5902-4b39-bea9-ab6b4d58ef42"
},
{ ... },
{ ... }
]
},
"task_id": "5774cc78b01249ab09f089dd",
"task": {
// task inlined for convenience
...
}
}
```
# Lines
Given a `line` entry in `params.geometries`, Scale will annotate your image or video with polylines (segmented lines) and return the vertices of the lines.
A list of `string` or **[LabelDescription](https://docs.scale.com/reference/label-nesting)** objects.
***
The minimum number of vertices in a valid line annotation for your request.
***
The maximum number of vertices in a valid line annotation for your request. Must be at least `min_vertices`.
***
### Response Fields
| **Key** | **Type** | Description |
| ---------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| uuid | string | A computer-generated unique identifier for this annotation.
In video annotation tasks, this can be used to track the same object across frames. |
| type | string | String to indicate geometry type: `line` |
| label | string | The label of this annotation, chosen from the `objects_to_annotate` array for its geometry. In video annotation tasks, any annotation objects with the same `uuid` will have the same `label` across all frames. |
| attributes | object | See the [Annotation Attributes](https://docs.scale.com/reference/attributes-overview) section for more details about the `attributes` response field. |
| vertices | array | An array of *vertex* objects describing the vertices of the polygon, listed in the order they were annotated. In other words, the point order will be either clockwise or counter-clockwise for each annotation. |
### Definition: `Vertex`
| **Key** | **Type** | **Description** |
| ------- | -------- | ----------------------------------------------------------------------------- |
| x | number | The distance, in pixels, between the vertex and the left border of the image. |
| y | number | The distance, in pixels, between the vertex and the top border of the image. |
```json theme={null}
{
"geometries": {
"line": {
"objects_to_annotate": [
"unmarked_lane",
{
"choice": "marked lanes",
"subchoices": [
"solid",
{
"choice": "dashed",
"display": "dashed or dotted"
}
]
},
"shoulder"
],
"min_vertices": 2,
"max_vertices": 15
},
...
},
...
}
```
```json theme={null}
{
"response": {
"annotations": [
{
"type": "line",
"label": "solid line",
"vertices": [
{
"x": 123,
"y": 10
},
{
"x": 140,
"y": 49
},
{
"x": 67,
"y": 34
}
],
"uuid": "65ec1f52-5902-4b39-bea9-ab6b4d58ef42"
},
{ ... },
{ ... }
]
},
"task_id": "5774cc78b01249ab09f089dd",
"task": {
// populated task for convenience
...
}
}
```
# Ellipses
Given an `ellipse` entry in `params.geometries`, Scale will annotate your image or video with ellipses and return the extremal points of the ellipses. The ellipses may be rotated relative to the X and Y axes.
A list of `string` or **[LabelDescription](https://docs.scale.com/reference/label-nesting)** objects.
***
### Response Fields
| Key | Type | Description |
| ---------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| uuid | string | A computer-generated unique identifier for this annotation.
In video annotation tasks, this can be used to track the same object across frames. |
| type | string | String to indicate geometry type: `ellipse` |
| label | string | The label of this annotation, chosen from the `objects_to_annotate` array for its geometry. In video annotation tasks, any annotation objects with the same `uuid` will have the same `label` across all frames. |
| attributes | object | See the [Annotation Attributes](https://docs.scale.com/reference/attributes-overview) section for more details about the `attributes` response field. |
| vertices | array | A list of **Vertex** objects of length 4 describing the extremal vertices of the ellipse |
```json theme={null}
{
...
"geometries": {
"ellipse": {
"objects_to_annotate": ["wheel"]
}
},
"annotation_attributes": {
"position": {
"type": "category",
"description": "What is the position of this wheel?",
"choices": [
"front_left",
"front_right",
"back_left",
"back_right",
]
}
},
...
}
```
```json theme={null}
{
"response": {
"annotations": [
{
"type": "ellipse",
"label": "wheel",
"attributes": {
"position": "front_left"
},
"vertices": [
{
"x": 123,
"y": 92
},
{
"x": 173,
"y": 113
},
{
"x": 123,
"y": 134
},
{
"x": 73,
"y": 113
}
],
"uuid": "65ec1f52-5902-4b39-bea9-ab6b4d58ef42"
},
{ ... },
{ ... }
]
},
"task_id": "5774cc78b01249ab09f089dd",
"task": {
// task inlined for convenience
...
}
}
```
# Cuboids
Given a `cuboid` entry in `params.geometries`, Scale will annotate your image or video with perspective cuboids and return the vertices of the cuboids. If camera intrinsics and extrinsics are provided as well, Scale will return scale-invariant 3D coordinates with respect to the camera, i.e. assuming the camera is at the origin. See [https://scale.com/blog/3d-cuboids-annotations](https://scale.com/blog/3d-cuboids-annotations) for a detailed explanation of how we can augment 2D cuboid responses.
A list of `string` or **[LabelDescription](https://docs.scale.com/reference/label-nesting)** objects.
***
The minimum height in pixels of the cuboids you'd like to be made.
***
The minimum width in pixels of the cuboids you'd like to be made.
***
An object that defines camera intrinsics, in format `\{fx: number, fy: number, cx: number, cy: number, scalefactor: number, skew: number\}` (`skew` defaults to 0, `scalefactor` defaults to 1). `scalefactor` is used if the image sent is of different dimensions from the original photo (if the attachment is half the original, set `scalefactor` to 2) to correct the focal lengths and offsets. Use in conjunction with `camera_rotation_quaternion` and `camera_height` to get perspective-corrected cuboids and 3d points.
***
Object that defines the rotation of the camera in relation to the world. Expressed as a quaternion, in format `\{w: number, x: number, y: number, z: number\}`. Use in conjunction with `camera_intrinsics` to get perspective-corrected cuboids and 3d points. Note that the z-axis of the camera frame represents the camera's optical axis. Use in conjunction with `camera_intrinsics` and `camera_height` to get perspective-corrected cuboids and 3d points.
***
The height of camera above the ground, in meters. Use in conjunction with `camera_rotation_quaternion` and `camera_intrinsics` to get perspective-corrected cuboids and 3d points.
***
### Response Fields
| Key | Type | Description |
| ---------- | ----------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| uuid | string | A computer-generated unique identifier for this annotation.
In video annotation tasks, this can be used to track the same object across frames. |
| type | string | String to indicate geometry type: `cuboid` |
| label | string | The label of this annotation, chosen from the `objects_to_annotate` array for its geometry. In video annotation tasks, any annotation objects with the same `uuid` will have the same `label` across all frames. |
| attributes | object | See the [Annotation Attributes](https://docs.scale.com/reference/attributes-overview) section for more details about the `attributes` response field. |
| vertices | array of `Vertex` objects | A list of `Vertex` objects defining all visible vertices of the cuboid. See the Vertex section for more details. |
| edges | array of `Edge` objects | A list of `Edge` objects defining the edges of the cuboid.. See the Edge section for more details. |
| points\_2d | array of `\{x, y\}` coordinate objects | If `camera_rotation_quaternion`, `camera_intrinsics`, and `camera_height` were provided, contains projected 2D coordinates of all 8 vertices of the cuboid after perspective correction. See diagram below for the order that the points are returned in. |
| points\_3d | array of `\{x, y, z\}` coordinate objects | If `camera_rotation_quaternion`, `camera_intrinsics`, and `camera_height` were provided, contains 3D coordinates (arbitrarily scaled, relative to the camera location) of all 8 vertices of the cuboid after perspective correction. See diagram below for the order that the points are returned in. |
### Definition: `Vertex`
| Key | Type | Description |
| ----------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| x | number | The distance, in pixels, between the vertex and the left border of the image. |
| y | number | The distance, in pixels, between the vertex and the top border of the image. |
| type | string | Always `vertex`. |
| description | string | An enum describing the position of the vertex, which is one of: `face-topleft` `face-bottomleft` `face-topright` `face-bottomright` `side-topcorner` `side-bottomcorner` |
### Definition: `Edge`
| Key | Type | Description |
| ----------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| x1 | number | The distance, in pixels, between the first vertex of the edge and the left border of the image. |
| y1 | number | The distance, in pixels, between the first vertex of the edge and the top border of the image. |
| x2 | number | The distance, in pixels, between the second vertex of the edge and the left border of the image. |
| y2 | number | The distance, in pixels, between the second vertex of the edge and the top border of the image. |
| type | string | Always `edge`. |
| description | string | An enum describing the position of the edge, which is one of:: `face-top` `face-bottom` `face-left` `face-right` `side-top` `side-bottom` |
```json theme={null}
{
...
"geometries": {
"cuboid": {
"objects_to_annotate": [
"car"
],
"min_height": 10,
"min_width": 10,
"camera_intrinsics": {
"fx": 986.778503418,
"fy": 984.4254150391,
"cx": 961.078918457,
"cy": 586.9694824219,
"skew": 0,
"scale_factor": 1
},
"camera_rotation_quaternion": {
"w": 0.0197866653,
"x": 0.0181939654,
"y": 0.6981190587,
"z": -0.715476937
},
"camera_height": -0.2993970777
}
},
...
}
```
```text theme={null}
Points on the cuboid are returned in this order for both points_2d and points_3d:
3-------2
/| /|
/ | / |
0-------1 |
| 7----|--6
| / | /
4-------5
```
```json theme={null}
{
...,
"response": {
"annotations": [
{
"label": "car",
"vertices": [
{
"description": "face-topleft",
"y": 270,
"x": 293,
"type": "vertex"
},
{
"description": "face-bottomleft",
"y": 437,
"x": 293,
"type": "vertex"
},
{
"description": "face-topright",
"y": 270,
"x": 471,
"type": "vertex"
},
{
"description": "face-bottomright",
"y": 437,
"x": 471,
"type": "vertex"
},
{
"description": "side-topcorner",
"y": 286,
"x": 607,
"type": "vertex"
},
{
"description": "side-bottomcorner",
"y": 373,
"x": 607,
"type": "vertex"
}
],
"edges": [
{
"description": "face-top",
"x1": 293,
"y1": 270,
"x2": 471,
"y2": 270,
"type": "edge"
},
{
"description": "face-right",
"x1": 471,
"y1": 270,
"x2": 471,
"y2": 437,
"type": "edge"
},
{
"description": "face-bottom",
"x1": 471,
"y1": 437,
"x2": 293,
"y2": 437,
"type": "edge"
},
{
"description": "face-left",
"x1": 293,
"y1": 437,
"x2": 293,
"y2": 270,
"type": "edge"
},
{
"description": "side-top",
"x1": 471,
"y1": 270,
"x2": 607,
"y2": 286,
"type": "edge"
},
{
"description": "side-bottom",
"x1": 471,
"y1": 437,
"x2": 607,
"y2": 373,
"type": "edge"
}
],
"points_2d": [
{
"y": 270,
"x": 293
},
{
"y": 437,
"x": 293
},
{
"y": 270,
"x": 471
},
{
"y": 437,
"x": 471
},
{
"y": 286,
"x": 607
},
{
"y": 373,
"x": 607
},
{
"y": 373,
"x": 607
},
{
"y": 373,
"x": 607
}
],
"points_3d": [
{
"z": 0,
"y": 270,
"x": 293
},
{
"z": 0,
"y": 437,
"x": 293
},
{
"z": 0,
"y": 270,
"x": 471
},
{
"z": 0,
"y": 437,
"x": 471
},
{
"z": 0,
"y": 286,
"x": 607
},
{
"z": 0,
"y": 373,
"x": 607
},
{
"z": 0,
"y": 373,
"x": 607
},
{
"z": 0,
"y": 373,
"x": 607
}
],
}
]
},
...
}
```
# Image Response Format
The `response` field, which is part of the callback POST request and permanently stored as part of the task object, will contain an `annotations` field (and a `global_attributes` field, if [Global Attributes](https://docs.scale.com/reference/global-attributes) were specified in the task creation request).
The annotations field will contain an array of **Annotation** objects. The schema of each **Annotation** object depends on the Geometry of the **Annotation**. See the [Boxes](https://docs.scale.com/reference/boxes), [Polygons](https://docs.scale.com/reference/polygons), [Lines](https://docs.scale.com/reference/lines), [Points](https://docs.scale.com/reference/points), [Cuboids](https://docs.scale.com/reference/cuboids), and [Ellipses](https://docs.scale.com/reference/ellipses) sections for descriptions of the schemas.
```json theme={null}
{
"response": {
"annotations": [
{
"type": "box",
"label": "small vehicle",
"attributes": {
"moving": "yes"
},
"left": 2,
"top": 4,
"width": 3,
"height": 5,
"uuid": "65ec1f52-5902-4b39-bea9-ab6b4d58ef42"
},
{
"type": "box",
"label": "large vehicle",
"attributes": {
"moving": "yes"
},
"left": 7,
"top": 5,
"width": 14,
"height": 5,
"uuid": "0a6cd019-a014-4c67-bd49-c269ba08028a"
},
{
"type": "polygon",
"label": "car",
"vertices": [
{
"x": 123,
"y": 10
},
{
"x": 140,
"y": 49
},
{
"x": 67,
"y": 34
}
],
"uuid": "65ec1f52-5902-4b39-bea9-ab6b4d58ef43"
},
{ ... },
{ ... }
],
"global_attributes": {
"driving": "Yes",
"night": "No"
}
},
"task_id": "5774cc78b01249ab09f089dd",
"task": {
// populated task for convenience
...
}
}
```
# Image Annotation Hypothesis
When creating a `imageannotation` task, you can provide prelabels in the `hypothesis` field, so that workers don't have to start from scratch to annotate the image.
In order to add pre-labels in a task using hypothesis, you’ll need to provide these in the `hypothesis` field of the payload when creating the task. The schema of the hypothesis object must match the schema of the task response.
1. Verify the task response field schema for the desired task type.
2. Review your project taxonomy (label names, attribute conditions, annotation types, etc).
3. Generate pre-labels that are formatted to match the aforementioned schema and taxonomy.
4. Create a task, including a hypothesis field that contains the pre-labels at the same top-level as other task fields such as project and instructions.
The hypothesis format will largely mirror Scale’s task response format. In this particular task type, `annotations` field array is mandatory inside the hypothesis object for simple annotations.
**Note:** UUIDs are not mandatory, if you want to use a particular UUID to track an annotation you can add it to the hypothesis, if not, Scale will generate one for you.
\_For Image Annotation, you can also add Global Attributes in the hypothesis object at the same level of annotations in the \_`global_attributes` field.
```json theme={null}
{
...
"attachment": "https://example.com/attachment.png",
"hypothesis": {
"annotations": [
{
"label": "car",
"left": 90,
"top": 66,
"height": 94,
"width": 96,
"type": "box"
}
]
},
...
}
```
```text theme={null}
{
"geometries": {
"box": {
"objects_to_annotate": [
"car"
],
"min_height": 10,
"min_width": 10
}
},
"annotation_attributes": {}
}
```
```json theme={null}
{
"links": [],
"annotations": [
{
"label": "car",
"uuid": "xfb506ca-d742-4e75-bb52-0725f099b238",
"left": 115,
"top": 68,
"height": 97,
"width": 69,
"type": "box"
},
],
"global_attributes": {}
}
```
# Video Annotation Overview
### **Note: Scale VideoAnnotation has been deprecated in favor of Video V2 (/task/videoplayback)**.
**Note: Scale Video is only available for our Enterprise customers**. If you want to learn more, please contact our [sales team](https://scale.com/sales).
This endpoint creates a `videoannotation` task. Given a series of images sampled from a video (which we will refer to as "frames"), Scale will annotate each frame with the Geometries (`box`, `polygon`, `line`, `point`, `cuboid,` and `ellipse`) you specify.
The required parameter for this task is `geometries`.
You can optionally provide additional markdown-enabled or Google Doc-based [instructions](https://scale.com/docs/instructions) via the `instruction` parameter.
You may also optionally specify `events_to_annotate`, a list of strings describing [events section](https://docs.scale.com/reference/events) to annotate in the video.
If the request is successful, Scale will return the generated task object, at which point you should store the `task_id` to have a permanent reference to the task.
# Label Nesting and Options
There are often annotation tasks that have too many label choices for a tasker to efficiently sort through them all at once, or times when you want to show one version of a label name to a tasker, but would like another version in the response.
In those cases, you can utilize `LabelDescription` objects to support nested labels, where labels may have subcategories within them, as well as setting `display` values for the label.
When declaring `objects_to_annotate` in your task parameters, we accept a mixed array of strings and the more complex `LabelDescription` objects.
### Definition: `LabelDescription`
A simple example is illustrated in the example JSON below, where `objects_to_annotate` can simply be a string, a nested label with choices and subchoices, or a nested label where the subchoices themselves are `LabelDescription` objects with a display value.
While there may be a large number of total labels, using subchoices a tasker can first categorize an object as a road, pedestrian, or vehicle, and based on that choice, further select the specific type of pedestrian or vehicle.
Nested labels may be specified both for the object labels (the `objects_to_annotate` array parameter), as well as in the `choices` array of a categorical annotation attribute. In both cases, you would specify a nested label by using a `LabelDescription` object instead of a string.
For example, for an `objects_to_annotate` array of `\["Vehicle", "Pedestrian"\]`, you could instead add a nested label by passing an array, like `\["Vehicle", \{"choice": "Pedestrian", "subchoices": \["Animal", "Adult", "Child"\]\}\]`. Then, if a tasker selected "Pedestrian" for an annotation, they would be further prompted to choose one of the corresponding subchoices for that annotation.
The `LabelDescription` object has the following structure:
| **Parameter** | **Type** | **Description** |
| --------------- | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| choice\* | string | The name of the label. This should be singular and descriptive (ex: `car`, `background`, `pole`).
When both a `choice` and `subchoices` are defined, the `choice` will not be selectable, it will only be used for UX navigation. Only the "leaf" nodes will be returned in Scale's response. |
| subchoices | Array\ | Optional: Descriptions of the sub-labels to be shown under this parent label. Array can be a mix of LabelDescription objects or strings. |
| instance\_label | boolean default `false` | Optional: For Segmentation-based Tasks - Whether this label should be segmented on a per-instance basis. For example, if you set `instance_label` to `true`, each individual car would get a separate mask in the image, allowing you to distinguish between them. |
| display | string default `choice` | Optional: The value to be shown to a Tasker for a given label. Visually overrides the `choice` field in the user experience, but does not affect the task response or conditionality. |
```python theme={null}
objects_to_annotate = [
"Road",
{
"choice": "Vehicle",
"subchoices": ["Car", "Truck", "Train", "Motorcycle"]
},
{
"choice": "Pedestrian",
"subchoices": [
"Animal",
{"choice": "Ped_HeightOverMeter", "display": "Adult" },
{"choice": "Ped_HeightUnderMeter", "display": "Child" },
]
}
]
```
# Image & Video Tasks
Source: https://api-reference.scale.com/docs/api-reference/image-and-video-tasks
Create Semantic Segmentation Annotation Task Create General Video Annotation Task Create Video Playback Annotation Task
# Create Semantic Segmentation Annotation Task
This endpoint creates a `**segmentannotation**` task. In this task, one of our labelers will view the given image and classify pixels in the image according to the labels provided. You will receive a semantic, pixel-wise, dense segmentation of the image. We also support **instance-aware** semantic segmentations, also called **panoptic segmentation**, via **[LabelDescription](/docs/api-reference/labels)** objects. The required parameters for this task are `**attachment**` and `**labels**`. The `**attachment**` is a URL to an image you'd like to be segmented. `**labels**` is an array of strings or **[LabelDescription](/docs/api-reference/labels)** objects describing the different types of objects you'd like to segment the image with. You can optionally provide additional markdown-enabled or Google Doc-based **[instructions](/docs/write-your-instructions)** via the `**instruction**` parameter. You can also optionally set `**allow_unlabeled**` to true, which will allow the existence of unlabeled pixels in the task response - otherwise, all pixels in the image will be classified (in which case it's important that there are labels for everything in the image, to avoid misclassification). The response you will receive will be a series of images where each pixel's value corresponds to the label, either via a numerical index or a color mapping. You will also get separate masks for each label for convenience.If the request successful, Scale will return the generated task object, at which point you should store the `**task_id**` to have a permanent reference to the task.
The name of the **[project](/docs/api-reference/projects)** to associate this task with.
***
A markdown-enabled string or iframe embed google doc explaining how to do the segmentation. You can use markdown to show example images, give structure to your instructions, and more. See our instruction best practices for more details. For Scale Rapid projects, DO NOT set this field unless you specifically want to override the project level instructions.
***
The full url (including the scheme `**http://**` or `**https://**`) or email address of the [**callback**](/docs/api-reference/callbacks) that will be used when the task is completed.
***
A URL to the image you'd like to be segmented.
***
Describes what type of file the attachment is. We currently only support image for the segmentannotation.
***
An array of strings or LabelDescription objects describing the different types of objects you'd like to be used to segment the image. You may include at most 50 labels.
***
This field is used to add additional attributes that you would like to capture per annotation. This only applies to instance annotations. See Annotation Attributes for more details about annotation attributes.
***
Whether or not this image can be completed without every pixel being labeled.
***
Editable annotations that a task should be initialized with. This is useful when you've run a model to prelabel the task and want annotators to refine those prelabels. Review the Segmentation Hypothesis Format for more details.
***
A set of key/value pairs that you can attach to a task object. It can be useful for storing additional information about the task in a structured format. Max 10KB. See the Metadata section for more detail.
***
An array of objects in the form of \{"attachment": "\"} to show to taskers as a reference. Context images themselves can not be labeled. Context images will appear like this in the UI. You cannot use the task's attachment url as a context attachment's url.
***
A arbitrary ID that you can assign to a task and then query for later. This ID must be unique across all projects under your account, otherwise the task submission will be rejected. See **[Avoiding Duplicate Tasks](/docs/api-reference/data-engine-reference#avoiding-duplicate-tasks)** for more details.
***
If set to be true, if a task errors out after being submitted, the unique id on the task will be unset. This param allows workflows where you can re-submit the same unique id to recover from errors automatically
***
Arbitrary labels that you can assign to a task. At most 5 tags are allowed per task. You can query tasks with specific tags through the task retrieval API.
***
```python Python theme={null}
import requests
# Replace with your actual API key
API_KEY = 'your_api_key_here'
# Define the URL for the API endpoint
url = "https://api.scale.com/v1/task/segmentannotation"
# Define the payload for the segment annotation task
payload = {
"instruction": "**Instructions:** Please label all the things",
"attachment": "https://i.imgur.com/iDZcXfS.png",
"attachment_type": "image",
"annotation_attributes": {
"newKey": {
"type": "type",
"description": "description",
"choices": "choices",
"conditions": {
"label_condition": ["car", "car2"],
"attribute_conditions": {
"newKey": "New Value",
"newKey-1": "New Value"
}
}
}
},
"allow_unlabeled": False,
"metadata": {
"newKey": "New Value",
"newKey-1": "New Value"
},
"project": "Project Name",
"batch": "Batch Name",
"callback_url": "http://www.example.com/callback",
"labels": [["vehicle"], "vehicle 2", "vehicle 3"],
"context_attachments": [{"attachment": "attachment"}, {"attachment": "attachment2"}],
"unique_id": "unique_id",
"clear_unique_id_on_error": True,
"tags": ["tag", "tag2"]
}
# Set up the headers for the request
headers = {
"accept": "application/json", # Specify that we want the response in JSON format
"content-type": "application/json" # Specify the content type of the request
}
# Adding authentication to the POST request
# The auth parameter requires a tuple with the API key and an empty string
response = requests.post(url, json=payload, headers=headers, auth=(API_KEY, ''))
# Print the response text to see the result
print(response.text)
```
```python Python SDK theme={null}
from scaleapi.tasks import TaskType
from scaleapi.exceptions import ScaleDuplicateResource
payload = dict(
"instruction": "**Instructions:** Please label all the things",
"attachment": "https://i.imgur.com/iDZcXfS.png",
"attachment_type": "image",
"annotation_attributes": { "newKey": {
"type": "type",
"description": "description",
"choices": "choices",
"conditions": {
"label_condition": ["car", "car2"],
"attribute_conditions": {
"newKey": "New Value",
"newKey-1": "New Value"
}
}
} },
"allow_unlabeled": False,
"metadata": {
"newKey": "New Value",
"newKey-1": "New Value"
},
"project": "Project Name",
"batch": "Batch Name",
"callback_url": "http://www.example.com/callback",
"labels": [["vehicle"], "vehicle 2", "vehicle 3"],
"context_attachments": [{ "attachment": "attachment" }, { "attachment": "attachment2" }],
"unique_id": "unique_id",
"clear_unique_id_on_error": True,
"tags": ["tag", "tag2"]
}
)
try:
client.create_task(TaskType.Segmentannotation, **payload)
except ScaleDuplicateResource as err:
print(err.message) # If unique_id is already used for a different task
```
```json theme={null}
{
"task_id": "string",
"created_at": "string",
"type": "segmentannotation",
"status": "pending",
"instruction": "string",
"is_test": false,
"urgency": "standard",
"metadata": {},
"project": "string",
"callback_url": "string",
"updated_at": "string",
"work_started": false,
"params": {
"allow_unlabeled": false,
"labels": [
null
],
"instance_labels": [
null
],
"attachment_type": "image",
"attachment": "https://i.imgur.com/SudOKhq.jpg"
}
}
```
# Create General Video Annotation Task
This endpoint creates a `**videoannotation**` task. Given a series of images sampled from a video (which we will refer to as "frames"), Scale will annotate each frame with the Geometries (box, polygon, line, point, cuboid, and ellipse) you specify.
The required parameter for this task is `**geometries**`.
You can optionally provide additional markdown-enabled or Google Doc-based instructions via the `**instruction**` parameter.
You may also optionally specify `**events_to_annotate**`, a list of strings describing events section to annotate in the video.
If the request is successful, Scale will return the generated task object, at which point you should store the `**task_id**` to have a permanent reference to the task.
The name of the **[project](/docs/api-reference/projects)** to associate this task with.
***
The name of the **[batch](/docs/api-reference/batches)** to associate this task with. Note that if a batch is specified, you need not specify the project, as the task will automatically be associated with the batch's project. For Scale Rapid projects specifying a batch is required. See **[Batches section](/docs/api-reference/data-engine-reference#batches-object-overview)** for more details.
***
A markdown-enabled string or iframe embed google doc explaining how to do the task. You can use **[markdown](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet)** to show example images, give structure to your instructions, and more. See our **[instruction best practices](/docs/write-your-instructions)** for more details. For Scale Rapid projects, DO NOT set this field unless you specifically want to override the project level instructions.
***
The full url (including the scheme `**http://**` or `**https://**`) or email address of the **[callback](/docs/api-reference/callbacks)** that will be used when the task is completed.
***
An array of URLs for the frames you'd like to be annotated. These image frames are stitched together to create a video. This is required if attachment\_type is image and must be omitted if attachment\_type is video.
***
A URL pointing to the video file attachment. Only the mp4, webm, and ogg formats are supported.
***
Describes what type of file the attachment(s) are. The only options are image and video.
***
An object mapping `**box**`, `**polygon**`, `**line**`, `**point**`, `**cuboid**`, or `**ellipse**` to Geometry objects
***
See the **[Annotation Attributes](/docs/api-reference/data-engine-reference#annotation-attributes-overview)** section for more details about annotation attributes.
***
The list of events to annotate.
***
Use this field to define links between annotations. See **[Links](/docs/api-reference/data-engine-reference#linked-attributes)** for more details about links.
***
The number of frames per second to annotate.
***
The amount of padding in pixels added to the top, bottom, left, and right of each video frame. This allows labelers to extend annotations outside of the frames.
***
The amount of padding in pixels added to the left and right of each video frame. Overrides `**padding**` if set.
***
The amount of padding in pixels added to the top and bottom of each video frame. Overrides padding if set.
***
Editable annotations that a task should be initialized with. This is useful when you've run a model to prelabel the task and want annotators to refine those prelabels. Review the Segmentation Hypothesis Format for more details.
***
Editable annotations, with the option to be "locked", that a task should be initialized with. This is useful when you've run a model to prelabel the task and want annotators to refine those prelabels. Must contain the annotations field, which has the same format as the annotations field in the response.
***
Whether or not new annotations can be added to the task if base\_annotations are used. If set to true, new annotations can be added to the task in addition to base\_annotations. If set to false, new annotations will not be able to be added to the task.
***
Whether or not base\_annotations can be edited in the task. If set to true, base\_annotations can be edited by the tasker (position of annotation, attributes, etc). If set to false, all aspects of base\_annotations will be locked.
***
Whether or not base\_annotations labels can be edited in the task. If set to true, the label of base\_annotations can be edited by the tasker. If set to false, the label will be locked.
***
Whether or not base\_annotations can be removed from the task. If set to true, base\_annotations can be deleted from the task. If set to false, base\_annotations cannot be deleted from the task.
***
A set of key/value pairs that you can attach to a task object. It can be useful for storing additional information about the task in a structured format. Max 10KB.
***
A value of 10, 20, or 30 that defines the priority of a task within a project. The higher the number, the higher the priority.
***
A arbitrary ID that you can assign to a task and then query for later. This ID must be unique across all projects under your account, otherwise the task submission will be rejected. See **[Avoiding Duplicate Tasks](/docs/api-reference/data-engine-reference#avoiding-duplicate-tasks)** for more details.
***
If set to be true, if a task errors out after being submitted, the unique id on the task will be unset. This param allows workflows where you can re-submit the same unique id to recover from errors automatically
***
Arbitrary labels that you can assign to a task. At most 5 tags are allowed per task. You can query tasks with specific tags through the task retrieval API.
***
```python Python theme={null}
import requests
# Replace with your actual API key
API_KEY = 'your_api_key_here'
# Define the URL for the API endpoint
url = "https://api.scale.com/v1/task/videoannotation"
# Define the payload for the video annotation task
payload = {
"instruction": "**Instructions:** Please label all the things",
"attachments": [
"https://static.scale.com/scaleapi-lidar-images/2011_09_29_drive_0071_sync/image_02/data/0000000005.png",
"https://static.scale.com/scaleapi-lidar-images/2011_09_29_drive_0071_sync/image_02/data/0000000008.png"
],
"attachment_type": "image",
"geometries": {
"box": {
"min_height": 10,
"min_width": 10,
"can_rotate": True,
"integer_pixels": False
},
"polygon": {
"min_vertices": 10,
"max_vertices": 20,
"objects_to_annotate": ["large vehicle"]
},
"line": {
"min_vertices": 10,
"max_vertices": 20,
"objects_to_annotate": ["large vehicle"]
},
"point": {
"objects_to_annotate": ["large vehicle", "large vehicle"]
},
"cuboid": {
"min_height": 10,
"min_width": 10,
"camera_intrinsics": {
"fx": 10,
"fy": 10,
"cx": 10,
"cy": 10,
"skew": 10,
"scalefactor": 10
},
"camera_rotation_quaternion": {
"w": 10,
"x": 10,
"y": 10,
"z": 10
},
"camera_height": 10
},
"ellipse": {
"objects_to_annotate": ["large vehicle"]
}
},
"events_to_annotate": ["event_1_name", "event_2_name"],
"frame_rate": 1,
"start_time": 10,
"padding": 10,
"paddingX": 10,
"metadata": {
"newKey": "New Value",
"newKey-1": "New Value"
},
"priority": 30,
"project": "Project Name",
"batch": "Batch Name",
"callback_url": "http://www.example.com/callback",
"attachment": "attachment_url",
"duration_time": 10,
"paddingY": 10,
"unique_id": "unique_id",
"clear_unique_id_on_error": True,
"tags": ["tag1", "tag2"]
}
# Set up the headers for the request
headers = {
"accept": "application/json", # Specify that we want the response in JSON format
"content-type": "application/json" # Specify the content type of the request
}
# Adding authentication to the POST request
# The auth parameter requires a tuple with the API key and an empty string
response = requests.post(url, json=payload, headers=headers, auth=(API_KEY, ''))
# Print the response text to see the result
print(response.text)
```
```python Python SDK theme={null}
from scaleapi.tasks import TaskType
from scaleapi.exceptions import ScaleDuplicateResource
payload = dict(
"instruction": "**Instructions:** Please label all the things",
"attachments": ["https://static.scale.com/scaleapi-lidar-images/2011_09_29_drive_0071_sync/image_02/data/0000000005.png", "https://static.scale.com/scaleapi-lidar-images/2011_09_29_drive_0071_sync/image_02/data/0000000008.png"],
"attachment_type": "image",
"geometries": {
"box": {
"min_height": 10,
"min_width": 10,
"can_rotate": True,
"integer_pixels": False
},
"polygon": {
"min_vertices": 10,
"max_vertices": 20,
"objects_to_annotate": ["large vehicle"]
},
"line": {
"min_vertices": 10,
"max_vertices": 20,
"objects_to_annotate": ["large vehicle"]
},
"point": { "objects_to_annotate": ["large vehicle", "large vehicle"] },
"cuboid": {
"min_height": 10,
"min_width": 10,
"camera_intrinsics": {
"fx": 10,
"fy": 10,
"cx": 10,
"cy": 10,
"skew": 10,
"scalefactor": 10
},
"camera_rotation_quaternion": {
"w": 10,
"x": 10,
"y": 10,
"z": 10
},
"camera_height": 10
},
"ellipse": { "objects_to_annotate": ["large vehicle"] }
},
"events_to_annotate": ["event_1_name", "event_2_name"],
"frame_rate": 1,
"start_time": 10,
"padding": 10,
"paddingX": 10,
"metadata": {
"newKey": "New Value",
"newKey-1": "New Value"
},
"priority": 30,
"project": "Project Name",
"batch": "Batch Name",
"callback_url": "http://www.example.com/callback",
"attachment": "attachment_url",
"duration_time": 10,
"paddingY": 10,
"unique_id": "unique_id",
"clear_unique_id_on_error": True,
"tags": ["tag1", "tag2"]
)
try:
client.create_task(TaskType.VideoAnnotation, **payload)
except ScaleDuplicateResource as err:
print(err.message) # If unique_id is already used for a different task
```
```json theme={null}
{
"task_id": "string",
"created_at": "string",
"type": "videoannotation",
"status": "pending",
"instruction": "string",
"is_test": false,
"urgency": "standard",
"metadata": {},
"project": "string",
"callback_url": "string",
"updated_at": "string",
"work_started": false,
"params": {
"attachment_type": "website",
"attachment": [
null
],
"geometries": {
"box": {
"objects_to_annotate": [
null
],
"min_height": 10,
"min_width": 10
},
"polygon": {
"objects_to_annotate": [
null
]
},
"point": {
"objects_to_annotate": [
null
]
}
},
"annotation_attributes": {
"additionalProp": {
"description": "string",
"choice": "string"
}
},
"events_to_annotate": [
null
],
"with_labels": true
}
}
```
# Create Video Playback Annotation Task
This endpoint creates a `**videoplaybackannotation**` task. In this task, we will view the given video file and draw annotations around the specified objects.
You are required to provide a URL to the video file as the `**attachment**`. It can be in `**mp4**`, `**webm**`, or `**ogg**` format.
You can optionally provide additional markdown-enabled or Google Doc-based **[instructions](/docs/write-your-instructions)** via the `**instruction**` parameter.
You may optionally specify a `**frame_rate**`, which will determine how many frames per second will be used to annotate the given video. The default value is `**1**`.
You may also optionally specify `**events_to_annotate**`, a list of strings describing **events section** to annotate in the video.
If the request is successful, Scale will return the generated task object, at which point you should store the `**task_id**` to have a permanent reference to the task.
The name of the **[project](/docs/api-reference/projects)** to associate this task with.
***
The name of the **[batch](/docs/api-reference/batches)** to associate this task with. Note that if a batch is specified, you need not specify the project, as the task will automatically be associated with the batch's project. For Scale Rapid projects specifying a batch is required. See **[Batches section](/docs/api-reference/batches)** for more details.
***
A markdown-enabled string or iframe embed google doc explaining how to do the task. You can use **[markdown](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet)** to show example images, give structure to your instructions, and more. See our **[instruction best practices](/docs/write-your-instructions)** for more details. For Scale Rapid projects, DO NOT set this field unless you specifically want to override the project level instructions.
***
The full url (including the scheme `**http://**` or `**https://**`) or email address of the **[callback](/docs/api-reference/callbacks)** that will be used when the task is completed.
***
An array of URLs for the frames you'd like to be annotated. These image frames are stitched together to create a video. This is required if attachment\_type is image and must be omitted if attachment\_type is video.
***
A URL pointing to the video file attachment. Only the mp4, webm, and ogg formats are supported.
***
Describes what type of file the attachment(s) are. The only options are image and video.
***
An object mapping box, polygon, line, point, cuboid, or ellipse to Geometry objects
***
See the **[Annotation Attributes](/docs/api-reference/data-engine-reference#annotation-attributes-overview)** section for more details about annotation attributes.
***
The list of events to annotate.
***
The duration of the video in seconds. This is ignored if attachment\_type is image. Default is full video length.
***
The number of frames to capture in one second. This is ignored if attachment\_type is image.
***
The start time in seconds. This is ignored if attachment\_type is image.
***
The amount of padding in pixels added to the top, bottom, left, and right of each video frame. This allows labelers to extend annotations outside of the frames.
***
The amount of padding in pixels added to the left and right of each video frame. Overrides padding if set.
***
The amount of padding in pixels added to the top and bottom of each video frame. Overrides padding if set.
***
Editable annotations, with the option to be "locked", that a task should be initialized with. This is useful when you've run a model to prelabel the task and want annotators to refine those prelabels. Must contain the annotations field, which has the same format as the annotations field in the response.
***
Whether or not new annotations can be added to the task if base\_annotations are used. If set to true, new annotations can be added to the task in addition to base\_annotations. If set to false, new annotations will not be able to be added to the task.
***
Whether or not base\_annotations can be edited in the task. If set to true, base\_annotations can be edited by the tasker (position of annotation, attributes, etc). If set to false, all aspects of base\_annotations will be locked.
***
Whether or not base\_annotations labels can be edited in the task. If set to true, the label of base\_annotations can be edited by the tasker. If set to false, the label will be locked.
***
Whether or not base\_annotations can be removed from the task. If set to true, base\_annotations can be deleted from the task. If set to false, base\_annotations cannot be deleted from the task.
***
A set of key/value pairs that you can attach to a task object. It can be useful for storing additional information about the task in a structured format. Max 10KB.
***
A value of 10, 20, or 30 that defines the priority of a task within a project. The higher the number, the higher the priority.
***
A arbitrary ID that you can assign to a task and then query for later. This ID must be unique across all projects under your account, otherwise the task submission will be rejected. See **[Avoiding Duplicate Tasks](/docs/api-reference/data-engine-reference#avoiding-duplicate-tasks)** for more details.
***
If set to be true, if a task errors out after being submitted, the unique id on the task will be unset. This param allows workflows where you can re-submit the same unique id to recover from errors automatically
***
Arbitrary labels that you can assign to a task. At most 5 tags are allowed per task. You can query tasks with specific tags through the task retrieval API.
***
```python Python theme={null}
import requests
# Replace with your actual API key
API_KEY = 'your_api_key_here'
# Define the URL for the API endpoint
url = "https://api.scale.com/v1/task/videoplaybackannotation"
# Define the payload for the video playback annotation task
payload = {
"instruction": "**Instructions:** Please label all the things",
"attachments": [
"https://static.scale.com/scaleapi-lidar-images/2011_09_26_drive_0051_sync/image_02/data/0000000000.png",
"https://static.scale.com/scaleapi-lidar-images/2011_09_26_drive_0051_sync/image_02/data/0000000001.png"
],
"attachment": "https://scale-static-assets.s3-us-west-2.amazonaws.com/demos/multimodal-video.mp4",
"attachment_type": "image",
"geometries": {
"box": {
"min_height": 10,
"min_width": 10
},
"polygon": {
"min_vertices": 1,
"max_vertices": " "
},
"line": {
"min_vertices": 1,
"max_vertices": " "
},
"point": {
"x": " ",
"y": " "
},
"cuboid": {
"min_height": 0,
"min_width": 0,
"camera_intrinsics": {
"fx": " ",
"fy": " ",
"cx": " ",
"cy": " ",
"skew": 0,
"scalefactor": 1
},
"camera_rotation_quaternion": {
"w": " ",
"x": " ",
"y": " ",
"z": " "
},
"camera_height": " "
}
},
"frame_rate": 1,
"padding": 0,
"paddingX": 0,
"paddingY": 0,
"priority": 30
}
# Set up the headers for the request
headers = {
"accept": "application/json", # Specify that we want the response in JSON format
"content-type": "application/json" # Specify the content type of the request
}
# Adding authentication to the POST request
# The auth parameter requires a tuple with the API key and an empty string
response = requests.post(url, json=payload, headers=headers, auth=(API_KEY, ''))
# Print the response text to see the result
print(response.text)
```
```text Python SDK theme={null}
from scaleapi.tasks import TaskType
from scaleapi.exceptions import ScaleDuplicateResource
payload = dict(
"instruction": "**Instructions:** Please label all the things",
"attachments": ["https://static.scale.com/scaleapi-lidar-images/2011_09_26_drive_0051_sync/image_02/data/0000000000.png", "https://static.scale.com/scaleapi-lidar-images/2011_09_26_drive_0051_sync/image_02/data/0000000001.png"],
"attachment": "https://scale-static-assets.s3-us-west-2.amazonaws.com/demos/multimodal-video.mp4",
"attachment_type": "image",
"geometries": {
"box": {
"min_height": 10,
"min_width": 10
},
"polygon": {
"min_vertices": 1,
"max_vertices": 10
},
"line": {
"min_vertices": 1,
"max_vertices": 10
},
"point": {
"x": " ",
"y": " "
},
"cuboid": {
"min_height": 0,
"min_width": 0,
"camera_intrinsics": {
"fx": 10,
"fy": 10,
"cx": 10,
"cy": 10,
"skew": 0,
"scalefactor": 1
},
"camera_rotation_quaternion": {
"w": 10,
"x": 10,
"y": 10,
"z": 10
},
"camera_height": 10
}
},
"frame_rate": 1,
"padding": 0,
"paddingX": 0,
"paddingY": 0,
"priority": 30
)
try:
client.create_task(TaskType.VideoPlaybackAnnotation, **payload)
except ScaleDuplicateResource as err:
print(err.message) # If unique_id is already used for a different task
```
```json theme={null}
{
"task_id": "string",
"created_at": "string",
"type": "imageannotation",
"status": "pending",
"instruction": "string",
"is_test": false,
"urgency": "standard",
"metadata": {},
"project": "string",
"callback_url": "string",
"updated_at": "string",
"work_started": false,
"params": {
"attachment_type": "image",
"attachment": "http://i.imgur.com/3Cpje3l.jpg",
"geometries": {
"box": {
"objects_to_annotate": [
null
],
"min_height": 5,
"min_width": 5
},
"polygon": {
"objects_to_annotate": [
null
]
},
"point": {
"objects_to_annotate": [
null
]
}
},
"annotation_attributes": {
"additionalProp": {
"type": "category",
"description": "string",
"choice": "string"
}
}
}
}
```
# Introduction to Scale API
Source: https://api-reference.scale.com/docs/api-reference/introduction-to-scale-api
API Reference
# API Reference
The Scale API is designed around the principles of REST. It uses resource-oriented URLs for predictable interactions, processes form-encoded request bodies, delivers JSON-encoded responses, and employs standard HTTP response codes, authentication protocols, and verbs.
The Scale API provides a sandbox mode, allowing you to test your integrations without affecting your live data or interacting with production systems. The API key used during request authentication determines whether the interaction occurs in live mode or sandbox mode.
Please note, the Scale API does not support bulk updates. Each request is tailored to work on a single object.
```shell Python theme={null}
pip install --upgrade scaleapi
conda install -c conda-forge scaleapi # If using Anaconda package management
```
```shell Javascript theme={null}
$ npm install scaleapi --save
```
**Don't see your favorite language?** **[Let us know](mailto:hello@scaleapi.com)** if you want (or are interested in writing) a library for a language not represented here!
# Multi-Stage Reference
Source: https://api-reference.scale.com/docs/api-reference/multi-stage-reference
Multi-Stage Overview Blueprints All multi-stage tasks are created with a blueprint, which is defined separately prior to task creation. Each task’s blueprint dictates its stages, stage dependencies (pipeline), their resp
# Multi-Stage Overview
Multi-stage is our most advanced and flexible task type.
It is designed to handle complex full-scene labeling that spans multiple annotation types and modalities, and serves as a replacement for Scale’s legacy dependent tasks system (which requires using multiple tasks to fully label a scene).
On a multi-stage task, labeling is split up into multiple stages. Stages are organized in a pipeline such that they are dependent on one another, and can run sequentially or in parallel. As a task passes through a stage, Scale adds labels corresponding to taxonomy classes enabled for that stage. Once all stages are completed, the task is then finalized and delivered.
# Blueprints
All multi-stage tasks are created with a blueprint, which is defined separately prior to task creation. Each task’s blueprint dictates its stages, stage dependencies (pipeline), their respective taxonomies, and the values of any stage-specific parameters.
Blueprints may also include conditional logic, so that certain stages are only enabled if specific criteria are met (e.g. if there is an attribute with a specific value in the prior stage).
Multi-stage projects may contain one or many blueprints, but only one blueprint can be “active” at any given time. New multi-stage tasks will use the project’s active blueprint by default, but may also use a different blueprint from the project if specified during task creation.
A list of all blueprints available for each multi-stage project is available in the Scale dashboard.
# Dependent Tasks vs Multi-Stage Tasks
| | Dependent Tasks | Multi-Stage Tasks |
| ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Task Creation | - Multiple tasks per scene - Complex payloads containing fully defined dependencies and taxonomies - Error-prone, frequently requires manual backfills to fix payload errors | - One task per scene - Simple payload referencing a single pre-defined blueprint - On-rails, payloads are much less likely to contain errors |
| Project Management | - Many projects, one for each dependency - Difficult to track all the different tasks across all projects for each scene | - One project - Each scene is labeled with just one task |
| Task Delivery | - Annotations are delivered across multiple tasks, each with a separate response | - Annotations are delivered in a single task with a single response (task-level delivery), or delivered separately for each stage (stage-level delivery) |
| Auditing | - Multiple audits across dependent tasks to fully audit a scene - Support for Standard audits only | - One audit is sufficient to review all annotations in the scene - Supports both Standard Audits (stage-level delivery only) and Fixless Audit |
# Task-level Delivery vs Stage-level Delivery
Each multi-stage project can be configured to use either task-level delivery or stage-level delivery.
With **task-level delivery**, a multi-stage task is delivered when all of its stages are completed. Customer auditors can create Fixless Audits, and all annotations from all stages will be available for review. Quality will be tracked at the task-level as well, covering all annotations across all stages.
With **stage-level delivery,** annotations from each stage are delivered as soon as the stage is completed (even if other stages are still in-progress). Customer auditors can create either Standard or Fixless Audits, which will be conducted separately for each stage. Quality will also be tracked separately for each stage, covering only the annotations added in that stage.
# Projects
Source: https://api-reference.scale.com/docs/api-reference/projects
Project Retrieval List All Projects Create Project Update Project Parameters
# Project Retrieval
The project retrieval endpoint allows you to get details about a specific project using its unique ID. It provides access to project-related data, such as name, status, task count, and metadata. You can use this endpoint to integrate with the Scale API for programmatic project management and monitoring. Don't forget to authenticate your requests with a valid API key for successful access.
***
```python Python theme={null}
import requests
# Replace with your actual API key
API_KEY = 'your_api_key_here'
# Define the URL for the API endpoint
url = "https://api.scale.com/v1/projects/kitten_labeling"
# Set up the headers for the request
headers = {
"accept": "application/json" # Specify that we want the response in JSON format
}
# Adding authentication to the GET request
# The auth parameter requires a tuple with the API key and an empty string
response = requests.get(url, headers=headers, auth=(API_KEY, ''))
# Print the response text to see the result
print(response.text)
```
```python Python SDK theme={null}
import scaleapi
# Initialize the ScaleClient with your API key
client = scaleapi.ScaleClient("YOUR_API_KEY_HERE")
# Define the project name you want to retrieve
project_name = "kitten_labeling"
# Retrieve the project details
project = client.get_project(project_name=project_name)
# Print the project details
print(project.as_dict())
```
```json theme={null}
{
"type": "imageannotation",
"name": "kitten_labeling",
"param_history": [
{
"instruction": "Instructions",
"version": 0,
"created_at": "2021-04-25T07:38:32.368Z"
}
],
"created_at": "2021-04-25T07:38:32.368Z"
}
```
# List All Projects
List information for all projects. Note: No parameters required. Optionally, set a boolean value for `archived` to only list information for all (un)archived projects.
```python Python theme={null}
import requests
from requests.auth import HTTPBasicAuth
# URL endpoint for the API call
url = "https://api.scale.com/v1/projects"
# Headers to specify the type of response we accept (JSON in this case)
headers = {"accept": "application/json"}
# Authentication using HTTP Basic Auth; provide your API key as the username
# No password is required, so the password field is empty
auth = HTTPBasicAuth('{{ApiKey}}', '')
# Performing a GET request to the API with headers and authentication
response = requests.get(url, headers=headers, auth=auth)
# Printing the text content of the response from the API
print(response.text)
```
```python Python SDK theme={null}
import scaleapi
# Initialize the ScaleClient with your API key
client = scaleapi.ScaleClient("YOUR_API_KEY_HERE")
# Retrieve the list of all projects
projects = client.get_projects()
# Iterate over the projects and print their details
for project in projects:
print(project.as_dict())
```
```json theme={null}
{
"type": "imageannotation",
"name": "project_name",
"param_history": [
{
"instruction": "Instructions",
"version": 0,
"created_at": "2021-04-25T07:38:32.368Z"
}
],
"created_at": "2021-04-25T07:38:32.368Z"
}
```
# Create Project
The project creation endpoint enables you to programmatically create new projects on the Scale API platform. It allows automation of project creation, streamlining data annotation workflows, and efficient management of multiple projects. You need valid authentication credentials (API key) and relevant project information to use this endpoint successfully.
The task type of all the tasks belonging to this project
***
Name identifying this project. Must be unique among all your projects. When creating tasks for this project, you should add a project: \ parameter to the task creation request to associate the task with this project.
***
Whether the project being created is a Scale Rapid project. Enterprise and On-Demand customers should omit this value.
***
Whether the project being created is a Scale Studio project. Enterprise and On-Demand customers should omit this value.
***
Default parameters for tasks created under this project. Any parameters specified here will be set if omitted in a task request; they will be overridden by any values sent in the task request. params.instruction behaves slightly differently; values here will instead be appended after the task-level instruction.
***
Studio projects only. Specify the pipeline of the project. Must use either standard\_task or consensus\_task.
***
Studio consensus projects only. Specify the number of attempts
***
```python Python theme={null}
import requests
# Replace with your actual API key
API_KEY = 'your_api_key_here'
# Define the URL for the API endpoint
url = "https://api.scale.com/v1/projects"
# Define the payload for creating a new project
payload = {
"type": "imageannotation", # Type of the project
"name": "kitten_labeling", # Name of the project
"rapid": False, # Indicates if the project is a rapid project
"studio": False, # Indicates if the project uses Scale AI's Studio interface
"params": {
"instruction": "Please label the kittens" # Instructions for the annotation task
}
}
# Set up the headers for the request
headers = {
"accept": "application/json", # Specify that we want the response in JSON format
"content-type": "application/json" # Specify the content type of the request
}
# Adding authentication to the POST request
# The auth parameter requires a tuple with the API key and an empty string
response = requests.post(url, json=payload, headers=headers, auth=(API_KEY, ''))
# Print the response text to see the result
print(response.text)
```
```python Python SDK theme={null}
import scaleapi
from scaleapi.tasks import TaskType
# Initialize the ScaleClient with your API key
client = scaleapi.ScaleClient("YOUR_API_KEY_HERE")
# Define the project payload
project_payload = {
"type": TaskType.ImageAnnotation,
"name": "project_name",
"rapid": False, # Set to True if creating a Scale Rapid project
"studio": True, # Set to True if creating a Scale Studio project
"params": {
"instruction": "Instructions"
},
"pipeline": "pipeline_name" # Specify the pipeline name for Studio projects
}
# Create the project
project = client.create_project(**project_payload)
# Print the created project's details
print(project.as_dict())
```
```json theme={null}
{
"type": "imageannotation",
"pipelineName": "pipeline_name",
"numReviews": 0,
"numConsensus": 0,
"created_at": "2023-08-04T15:11:57.508Z",
"created_by": "user_id",
"created_with": "api",
"param_history": [],
"projectType": "project_type",
"name": "project_name",
"pinned": false,
"archived": false,
"nucleusDatasetId": "nucleus_id",
"customGradersUsedOrDismissed": false,
"edgeCaseAlertsEnabled": false,
"turnOnLongHints": true,
"isAudioTranscription": false,
"useOldOnboarding": false,
"containsAdultContent": false,
"useRapidSmartTraining": false,
"useTextCollectionTemplate": false,
"datasetLinks": [
{
"datasetId": "dataset_id",
"datasetName": "dataset_name"
}
],
"taxonomy": null,
"isSampleProject": false,
"modelEvalProject": false
}
```
# Update Project Parameters
You can set parameters on a project. Project-level-parameters will be set on future tasks created under this project if they are not set in the task request. Any parameters specified in the task request will override any project parameter.
Projects keep a history of the parameters that they were set with. Tasks created under a project inherit the latest params of the project (the last entry in param\_history), one can also specify project\_version when creating a task to inherit from an older set of params (or use -1 to skip parameter inheritance altogether.)
Tasks have a `**project_param_version**` field pointing to the project version in place at the time a task was created.
If you haven't already, check out our **[guide to using Project-level parameters](/docs/api-reference/data-engine-reference#using-project-level-parameters)**[ ](/docs/api-reference/data-engine-reference#using-project-level-parameters)to learn more about how this workflow comes togeher.
Just want to see some examples? **[We've added those as well](/docs/api-reference/projects)**.
When generating the newest set of project parameters, whether or not to combine the most recent project parameters with the parameters specified in this request. Defaults to false. For example, if the current params contains \{ instruction: 'label the cats in the image' } and the latest call to this API contains the task parameter \{ objects\_to\_annotate: \['cat'] }, whether or not patch is additionally set will determine if the new project parameters also contain the instruction “label the cats in the image”. Note that any fields set in the API request completely override any fields from the previous version; object or array values will not be merged.
***
instruction field to combine with any specified task-level instruction.
***
Default parameters for tasks created under this project. Any parameters specified here will be set if omitted in a task request; they will be overridden by any values sent in the task request.
***
```python Python theme={null}
import requests
# Replace with your actual API key
API_KEY = 'your_api_key_here'
# Define the URL for the API endpoint
url = "https://api.scale.com/v1/projects/kitten_labeling/setParams"
# Define the payload to update project parameters
payload = {
"patch": "false" # Parameter to be updated
}
# Set up the headers for the request
headers = {
"accept": "application/json", # Specify that we want the response in JSON format
"content-type": "application/json" # Specify the content type of the request
}
# Adding authentication to the POST request
# The auth parameter requires a tuple with the API key and an empty string
response = requests.post(url, json=payload, headers=headers, auth=(API_KEY, ''))
# Print the response text to see the result
print(response.text)
```
```python Python SDK theme={null}
import scaleapi
# Initialize the ScaleClient with your API key
client = scaleapi.ScaleClient("YOUR_API_KEY_HERE")
# Function to set project parameters
def set_project_params(client, project_name: str, params: dict):
"""
Set parameters for a project.
Args:
client (ScaleClient): The ScaleClient instance.
project_name (str): The name of the project to update.
params (dict): A dictionary of parameters to set.
"""
response = client.update_project(project_name, **params)
return response
# Example usage
project_name = "example_project"
params = {
"instruction": "Please label the objects in the image.",
"priority": 10,
"additional_param": "value"
}
response = set_project_params(client, project_name, params)
print(response)
```
```json theme={null}
{
"type": "imageannotation",
"name": "kitten_labeling",
"param_history": [
{
"instruction": "please label the kittens in the image",
"version": 0,
"created_at": "2021-04-25T07:38:32.368Z"
}
],
"created_at": "2021-04-25T07:38:32.368Z"
}
```
# Secure Result URLs
Source: https://api-reference.scale.com/docs/api-reference/secure-result-urls
Secure Result URLs For tasks like 2d segmentation, video, and lidar where results cannot simply be put in the task JSON, task responses are uploaded to an AWS S3 bucket scaleapi-results, publicly accessible with a URL in
# Secure Result URLs
For tasks like 2d segmentation, video, and lidar where results cannot simply be put in the task JSON, task responses are uploaded to an AWS S3 bucket `**scaleapi-results**`, publicly accessible with a URL in the form `**https://scaleapi-results.s3.us-west-1.amazonaws.com/:uuid**`. This is generally secure enough for our customers, as the UUIDs are unlikely to be guessed and cannot be linked back to a user or the data that the annotations correspond to.
We do have the ability to, rather than return publicly accessible URLs, instead, return URLs that require authentication for each request. This ensures that third parties without access to your Scale account will not be able to download your task responses, even if they manage to get a hold of or guess the URL.
Secure result URLs will be of the form `**https://api.scale.com/v1/task/:taskId/response_url/:uuid**`, and in order to be fetched HTTP basic auth (with your account's live API key) must be used.
Please contact **[Scale Support](/docs/customer-support)** to enable secure result URLs for your account.
# Sensor Fusion / Lidar Reference
Source: https://api-reference.scale.com/docs/api-reference/sensor-fusion-lidar-reference
Lidar Annotation Overview Lidar Segmentation Overview 2D / 3D Lidar Linking Overview
# Lidar Annotation Overview
### Data Types and the Frame Objects
The input into our sensor fusion application will be a series of points, radar points, and camera images that will be rendered and labeled. Because of the size of the objects involved, we will require that the data be JSON-encoded (or protobuf-encoded) and accessible via a URL passed in through the task request. Basically, in order to annotate a point cloud frame, format the data in one of our accepted formats, upload the data as a file, and then send a request to the Scale API, similar to the way that we would process image files.Below are our definitions for our various object types for the JSON format, and for an entire point cloud frame. The protobuf format is largely identical, and can be downloaded **[here](https://static.scale.com/protos/lidar_frame-1.1.proto)**; the difference is that camera intrinsic parameters are encoded as a `**oneof**` within the `**CameraImage**` message type, and thus no `**camera_model**` field is needed.
Definition: `Vector2`
`**Vector2**` objects are used to represent positions, and are JSON objects with 2 properties.
| x | float | x value |
| - | ----- | ------- |
| y | float | y value |
```json theme={null}
{
"x": 1,
"y": 2
}
```
# Lidar Segmentation Overview
### Start From Completed LIDAR Task
Instead of creating Lidar Segmentation tasks from scratch, we can bring already completed work from a **[Lidar Annotation](/docs/api-reference/sensor-fusion-lidar-tasks#create-lidar-cuboid-annotation)** task into the Lidar Segmentation task. This will persist the cuboids that you got from the annotation step. Furthermore, you can specify which subset of frames should be included as the source material.
The following differences from the previous approach will take effect:
* A new parameter name `**lidar_task**` will contain the identifier of the LiDAR Annotation task to be used as source. The `**lidar_task**` needs to be in a `**completed**` state.
* You don't need to add the `**attachments**` and `**attachment_type**` parameters as the `**Frame**` objects will be taken from the source LiDAR Annotation task.
* A new optional parameter appears, named `**lidar_task_frames**`, allows you to specify an array of frame indexes to select which subset of frames you want to use from the LiDAR Annotation task. If omitted all frames will be used.
* For example, assuming we start from a completed LiDAR Annotation task with five frames and we wanted to use all frames except the last one, the parameter will look like `**lidar_task_frames: \[0, 1, 2, 3\]**`.
* The `**labels**` parameter needs to be a super set of the set used on the original LiDAR Annotation task.
## 2D / 3D Lidar Linking Overview
### Inherited Lidar Attributes
Object attributes set in a **`lidarannotation`** task can be inherited by the corresponding object in **`lidarlinking`** tasks created from the **`lidarannotation`** task.
### Option 1:
Inherited lidar attributes can be enabled by using the **`copy_all_lidar_task_attributes`** flag when creating the **`lidarlinking`** task.
If using this option, you can NOT set the same attributes in the **`lidarlinking`** task, because it will be copied over automatically from the **`lidarannotation`** task.
### Option 2:
Setting **`copy_from_lidar_task: true`** on one or more **`annotation_attributes`** defined in the lidar linking task.
These attributes will be copied from the **`lidarannotation`** task to the **`lidarlinking`** task and cannot be modified in annotations derived from the **`lidarannotation`** task.
In both cases, for any new annotations that are added in, they can be used.
```json theme={null}
{ // ... in the linking task payload
"annotation_attributes": {
"Size": {
"copy_from_lidar_task": true,
"type": "category",
"description": "An attribute that was set in the original lidar task. Note that the attribute name must match the original attribute name. If this attribute is copied from an annotation existing in the lidar task, its value cannot be changed.",
"choices": [ "Large", "Small" ]
},
"Shape": {
"type": "category",
"description": "This is a new attribute that is specific to the 2d task",
"choices": [ "Parallelogram", "Square", "Rhombus" ]
}
}
}
```
# Sensor Fusion / Lidar Tasks
Source: https://api-reference.scale.com/docs/api-reference/sensor-fusion-lidar-tasks
Create Lidar Topdown Tasks Create Lidar Cuboid Annotation Create Lidar Segmentation Annotation Create Lidar Linking Annotation Change Dependent Task Options Force Dependent Tasks Creation
# Create Lidar Topdown Tasks
This endpoint creates a lidartopdown task for annotating a collection of lidar Frames in top down, with vector geometric shapes. The available geometries are polygon, line, and point.
Given a collection of LiDAR Frames, and optional camera data, Scale will annotate the top down images with the specified geometries. The callback\_url is the URL which will be POSTed on task completion, and is described in more detail in the Callback section. The attachments will be a list of links to external JSON files, each following the definition of a Frame.
The name of the project to associate this task with. See the [Projects](/docs/api-reference/projects) Section for more details.
***
The name of the batch to associate this task with. Note that if a batch is specified, you need not specify the project, as the task will automatically be associated with the batch's project. See [Batches section](/docs/api-reference/batches) for more details.
***
A markdown-enabled string or iframe embed google doc explaining how to do the task. You can use [markdown](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet) to show example images, give structure to your instructions, and more. See our [instruction best practices](/docs/write-your-instructions) for more details.
***
The full url (including the scheme `http://` or `https://`) of the callback when the task is completed. See the [Callback section](/docs/api-reference/callbacks) for more details about callbacks.
***
required if `attachments` is not specified. The full url of an image (png, jpg), to serve as the TopDown aerial imagery for a task to be labeled upon. If an `attachment` is submitted, the `attachments` fields should be empty, and vice versa. If a LiDAR task and not an aerial image task, an `attachment` will be automatically generated by projecting the points in the set of `attachments`.
***
A list of URLs to the `Frame` objects you’d like to be labeled. The frames should be time-ordered as is natural. The URLs should link to JSON files that follows the specification above, [Callback section](/docs/api-reference/callbacks), or protobuf files that encode `LidarFrame` messages as defined in the .proto file.
***
required for aerial imagery tasks when submitting type `world_camera`. This object allows Scale to perform the correct transformation from lon/lat world coordinates to pixels. It allows Scale to identify the pixel coordinates of the camera location on the provided aerial imagery task. If this is not provided, the camera context images will not render on the task.
***
required for lidar tasks. This Object crops the attachments’ points to a rectangle on the XY plane centered around position with rotation counterclockwise to the z-axis. This must be submitted for any LiDAR TopDown annotation tasks, and defines the bounds to which the point cloud should be restricted to for annotation.
***
This object is used to define which objects need to be annotated and which annotation geometries (box, polygon, line, poin) should be used for each annotation.
If taxonomy service is enabled, this field will overwrite the geometries defined in the taxonomy version.
Required if not using taxonomy service
***
List of labels under the “line” category in geometries that should have directionality. Note, the label names must be matched exactly.
If taxonomy service is enabled, this field will overwrite the directed lines defined in the taxonomy version.
***
This field is used to add additional attributes that you would like to capture per annotation. See [Annotation Attributes](/docs/api-reference/annotation-attributes) for more details about annotation attributes.
If taxonomy service is enabled, this field will overwrite the annotation attributes defined in the taxonomy version.
***
Use this field to define links between annotations. See [Links](/docs/api-reference/annotation-attributes) for more details about links.
If taxonomy service is enabled, this field will overwrite the links defined in the taxonomy version.
***
Editable annotations, with the option to be 'locked', that a task should be initialized with. This is useful when you've run a model to prelabel the task and want annotators to refine those prelabels. Must contain the annotations field, which has the same format as the annotations field in the response.
***
A list of groups that this label belongs to. If this choice has subchoices, those subchoices will also belong to these groups. This is used to provide additional info to each LabelDescription, as defined in [LabelDescription nesting](/docs/api-reference/annotation-attributes). Example: The label Single Solid belongs to groups Roundabout Edge and Colored Line, whereas the label Double Solid only belongs to the group Roundabout Edge.
***
Use this field to define relationships between annotations. If using line annotations to form polygon annotations, the labels of the involved annotations are set here.
***
By default, when a LidarTopDown task is created as a dependent task of a LidarAnnotation task, the LidarAnnotation’s cuboids are projected as polygons in the LidarTopDown task. By setting this property to true, that behavior is disabled and no LidarAnnotation cuboids will be projected to the LidarTopDown Task. Note that this parameter only takes effect if the LidarTopdown task is a dependent task.
***
A set of key/value pairs that you can attach to a task object. It can be useful for storing additional information about the task in a structured format. Max 10KB.
***
A value of 10, 20, or 30 that defines the priority of a task within a project. The higher the number, the higher the priority.
***
A arbitrary ID that you can assign to a task and then query for later. This ID must be unique across all projects under your account, otherwise the task submission will be rejected. See [Avoiding Duplicate Tasks](/docs/api-reference/tasks) for more details.
***
If set to be true, if a task errors out after being submitted, the unique id on the task will be unset. This param allows workflows where you can re-submit the same unique id to recover from errors automatically
***
Arbitrary labels that you can assign to a task. At most 5 tags are allowed per task. You can query tasks with specific tags through the task retrieval API.
***
Task ID of a completed lidar task used to construct this Lidar TopDown task. Annotation information from the Lidar task will be used as a prior for the LTD task. This is only used when creating a Lidar TopDown task from a Lidar Cuboids Task
***
The height of the lidar device relative to the ground in meters. If a point on the ground has height z in the device coordinate frame, then z + deviceHeight should be about 0. Used to filter out points that are too high/low more accurately.
***
Use this field to specify a taxonomy version to use from the taxonomy service when it is enabled. If this field is empty, the task will use the most recently submitted taxonomy
```python theme={null}
import requests
# Replace with your actual API key
API_KEY = 'your_api_key_here'
# Define the URL for the API endpoint
url = "https://api.scale.com/v1/task/lidartopdown"
# Define the payload for the LIDAR top-down task
payload = {
"instruction": "**Instructions:** Please label all the things",
"callback_url": "https://example.com/callback",
"attachment": ["https://s3-us-west-1.amazonaws.com/scaleapi-cust-lidar/kitti-road-2011_10_03_drive_0047/frames/frame1.json"],
"attachments": ["https://s3-us-west-1.amazonaws.com/scaleapi-cust-lidar/kitti-road-2011_10_03_drive_0047/frames/frame1.json"],
"geometries": {
"newKey": "New Value",
"newKey-1": "New Value_1"
},
"priority": None
}
# Set up the headers for the request
headers = {
"accept": "application/json", # Specify that we want the response in JSON format
"content-type": "application/json" # Specify the content type of the request
}
# Adding authentication to the POST request
# The auth parameter requires a tuple with the API key and an empty string
response = requests.post(url, json=payload, headers=headers, auth=(API_KEY, ''))
# Print the response text to see the result
print(response.text)
```
```json theme={null}
{
"callback_url": "http://www.example.com/callback",
"created_at": "2019-01-16T21:03:33.166Z",
"instruction": "**Instructions:** Please label all the things",
"is_test": false,
"params": {},
"status": "pending",
"task_id": "5a99e20de50d4979ce6d291e",
"type": "lidartopdown"
}
```
# Create Lidar Cuboid Annotation
This endpoint creates a `**lidarannotation**` task. In this task, one of our Scalers view outputs from a series of LIDAR frames, along with optional radar and camera data, and annotate where different objects exist in the 3D space with 3D cuboids. The required parameters for this task are `**attachments**`, `**labels**`, and `**attachment_type**`. `**The callback_url**` is the URL which will be POSTed on task completion, and is described in more detail in the **[Callback section](/docs/api-reference/callbacks)**. The `**attachments**` will be a list of links to external JSON files, each following the definition of a `**Frame**` as specified below.
The name of the project to associate this task with. See the [Projects](/docs/api-reference/projects) Section for more details.
***
The name of the batch to associate this task with. Note that if a batch is specified, you need not specify the project, as the task will automatically be associated with the batch's project. For Scale Rapid projects specifying a batch is required. See [Batches section](/docs/api-reference/batches) for more details.
***
A markdown-enabled string or iframe embed google doc explaining how to do the task. You can use [markdown](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet) to show example images, give structure to your instructions, and more. See our [instruction best practices](/docs/rapid-or-how-it-works) for more details. For Scale Rapid projects, DO NOT set this field unless you specifically want to override the project level instructions.
***
The full url (including the scheme `http://` or `https://`) of the callback when the task is completed. See the [Callback section](/docs/api-reference/callbacks) for more details about callbacks.
***
A list of URLs to the `Frame` objects you’d like to be labeled. The frames should be time-ordered as is natural. The URLs should link to JSON files that follows the specification above, [Callback section](/docs/api-reference/callbacks), or protobuf files that encode `LidarFrame` messages as defined in the .proto file.
***
Describes what type of file the attachment is. Defaults to `json`, but should be set to protobuf if attachments are being sent in `protobuf` format.
***
An array of strings or objects describing the different types of objects you’d like to be used to segment the image. You may include at most 50 objects. See [Label Nesting](/docs/api-reference/annotation-attributes) and Options for more details about label objects.
***
This field is used to add additional attributes that you would like to capture per annotation. See [Annotation Attributes](https://docs.scale.com/reference/attributes-overview) for more details about annotation attributes.
***
The maximum distance in meters from the sensor for which an object should be labeled. If undefined, all visible objects will be labeled.
***
The conversion rate of a unit scalar in the point data to a meter in the real world. e.g. if a unit vector represents 10 meters in real world distance, then this value should be 10.
***
The frequency of the frames per second.
***
The sample rate of frames which will be fully labeled. If you are capturing 10Hz LIDAR but only want labels in 2Hz, you can set this parameter to 5 and achieve that. If the sample rate is `k` and there are `n` frames total, we will fully label (1-indexed), the `1`st, `k + 1`-th, `2k + 1`-th, ... , `floor((n - 1) / k) * k + 1`-th, and `n`th frames.
***
Enables polygon annotations, see the Polygons section in [Response on Callback](https://docs.scale.com/reference/lidar-callback-format#definition-polygon) for more details.
***
An array of strings describing the different types of polygons you'd like to be annotated in the scene.
***
A set of key/value pairs that you can attach to a task object. It can be useful for storing additional information about the task in a structured format. Max 10KB.
***
A value of 10, 20, or 30 that defines the priority of a task within a project. The higher the number, the higher the priority.
***
A arbitrary ID that you can assign to a task and then query for later. This ID must be unique across all projects under your account, otherwise the task submission will be rejected. See [Avoiding Duplicate Tasks](/docs/api-reference/tasks)[ ](/docs/api-reference/tasks)for more details.
***
If set to be true, if a task errors out after being submitted, the unique id on the task will be unset. This param allows workflows where you can re-submit the same unique id to recover from errors automatically
***
Arbitrary labels that you can assign to a task. At most 5 tags are allowed per task. You can query tasks with specific tags through the task retrieval API.
***
```python Python theme={null}
import requests
# Replace with your actual API key
API_KEY = 'your_api_key_here'
# Define the URL for the API endpoint
url = "https://api.scale.com/v1/task/lidarannotation"
# Define the payload for the LIDAR annotation task
payload = {
"instruction": "Annotate the *vehicles* and *pedestrians* in the image.",
"callback_url": "https://example.com/callback",
"attachments": ["https://s3-us-west-1.amazonaws.com/scaleapi-cust-lidar/kitti-road-2011_10_03_drive_0047/frames/frame1.json"],
"attachment_type": "json",
"labels": ["Vehicle"],
"priority": None
}
# Set up the headers for the request
headers = {
"accept": "application/json", # Specify that we want the response in JSON format
"content-type": "application/json" # Specify the content type of the request
}
# Adding authentication to the POST request
# The auth parameter requires a tuple with the API key and an empty string
response = requests.post(url, json=payload, headers=headers, auth=(API_KEY, ''))
# Print the response text to see the result
print(response.text)
```
```python Python SDK theme={null}
from scaleapi.tasks import TaskType
from scaleapi.exceptions import ScaleDuplicateResource
payload = dict(
"instruction": "Annotate the *vehicles* and *pedestrians* in the image.",
"callback_url": "https://example.com/callback",
"attachments": ["https://s3-us-west-1.amazonaws.com/scaleapi-cust-lidar/kitti-road-2011_10_03_drive_0047/frames/frame1.json"],
"attachment_type": "json",
"labels": ["Vehicle"],
"priority": None
)
try:
client.create_task(TaskType.LidarAnnotation, **payload)
except ScaleDuplicateResource as err:
print(err.message) # If unique_id is already used for a different task
```
```json theme={null}
{
"callback_url": "http://www.example.com/callback",
"created_at": "2019-01-16T21:03:33.166Z",
"instruction": "Annotate the *vehicles* and *pedestrians* in the image.",
"is_test": false,
"params": {},
"status": "pending",
"task_id": "5a99e20de50d4979ce6d291e",
"type": "lidarannotation"
}
```
# Create Lidar Segmentation Annotation
This endpoint creates a `**lidarsegmentation**` task. In this task, one of our Taskers views outputs from a series of LIDAR frames, along with optional camera data, and annotates where different objects exist in the 3D space by assigning a class to each `**[LidarPoint](/docs/api-reference/sensor-fusion-lidar-reference#definition-lidarpoint)**`.\n\nThis type of task can be created on its own, or you can create a task **[based on an already completed Lidar Annotation task](/docs/api-reference/sensor-fusion-lidar-tasks)**.\n\nThe required parameters for this task are `**labels**`, `**attachments**`, and `**attachment_type**`. \n\n\* The `**callback_url**` is the URL which will be POSTed on task completion, and is described in more detail in the **[callbacks](/docs/api-reference/callbacks)** section. \n\* The `**labels**` array lists the object classes for which semantic information is desired.\n\t\* Instance labels are supported, by specifying `**instance_label: true**` when defining the label. For example, `**\['Road', \{'choice': 'Pedestrian', 'instance_label': true\}\]**`.\n\t\* Nested labels are also supported for these labels, and may be specified in the same format as noted in our **[documentation](/docs/api-reference/labels)**. For example, `**\['Vehicle', \{'choice': 'Pedestrian', 'subchoices': \['Adult', 'Child'\]\}\]**`.\n\* The `**attachments**` will be a list of links to external JSON files, each following the definition of a `**Frame**` as specified **[here](/docs/api-reference/sensor-fusion-lidar-reference#definition-frame)**.
You should provide additional **[markdown-enabled](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet)** instructions via the instruction parameter.\n\nIt is **strongly recommended** for you to flesh out your Markdown instructions with many examples of tasks being done correctly and incorrectly.\n\nIf successful, Scale will immediately return the generated task object, of which you should at least store the `**task_id**`
The name of the batch to associate this task with. Note that if a batch is specified, you need not specify the project, as the task will automatically be associated with the batch's project. For Scale Rapid projects specifying a batch is required. See Batches section for more details.
***
The name of the batch to associate this task with. Note that if a batch is specified, you need not specify the project, as the task will automatically be associated with the batch's project. For Scale Rapid projects specifying a batch is required. See Batches section for more details.
***
A markdown-enabled string or iframe embed google doc explaining how to do the task. You can use markdown to show example images, give structure to your instructions, and more. See our instruction best practices for more details. For Scale Rapid projects, DO NOT set this field unless you specifically want to override the project level instruction
***
The full url (including the scheme http\:// or https\://) of the callback when the task is completed. See the Callback section for more details about callbacks.
***
A list of URLs to the Frame objects you’d like to be labeled. The frames should be time-ordered as is natural. The URLs should link to JSON files that follows the specification above Callback section, or protobuf files that encode LidarFrame messages as defined in the .proto file.
***
Describes what type of file the attachment is. Defaults to json, but should be set to protobuf if attachments are being sent in protobuf format.
***
An array of strings or objects describing the different types of objects you’d like to be used to segment the image. You may include at most 50 objects. See Label Nesting and Options for more details about label objects.
***
A set of key/value pairs that you can attach to a task object. It can be useful for storing additional information about the task in a structured format. Max 10KB.
***
A value of 10, 20, or 30 that defines the priority of a task within a project. The higher the number, the higher the priority.
***
A arbitrary ID that you can assign to a task and then query for later. This ID must be unique across all projects under your account, otherwise the task submission will be rejected. See **[Avoiding Duplicate Tasks](/docs/api-reference/data-engine-reference#avoiding-duplicate-tasks)** for more details
***
If set to be true, if a task errors out after being submitted, the unique id on the task will be unset. This param allows workflows where you can re-submit the same unique id to recover from errors automatically
***
Arbitrary labels that you can assign to a task. At most 5 tags are allowed per task. You can query tasks with specific tags through the task retrieval API.
***
```python theme={null}
import requests
# Replace with your actual API key
API_KEY = 'your_api_key_here'
# Define the URL for the API endpoint
url = "https://api.scale.com/v1/task/lidarsegmentation"
# Define the payload for the LIDAR segmentation task
payload = {
"instruction": "**Instructions:** Please label all the things",
"callback_url": "https://example.com/callback",
"attachments": ["https://s3-us-west-1.amazonaws.com/scaleapi-cust-lidar/kitti-road-2011_10_03_drive_0047/frames/frame1.json"],
"attachment_type": "json",
"labels": ["Vegetation"],
"priority": None
}
# Set up the headers for the request
headers = {
"accept": "application/json", # Specify that we want the response in JSON format
"content-type": "application/json" # Specify the content type of the request
}
# Adding authentication to the POST request
# The auth parameter requires a tuple with the API key and an empty string
response = requests.post(url, json=payload, headers=headers, auth=(API_KEY, ''))
# Print the response text to see the result
print(response.text)
```
```json theme={null}
{
"callback_url": "http://www.example.com/callback",
"created_at": "2019-01-16T21:03:33.166Z",
"instruction": "Segment the *Vegetation* in the image.",
"is_test": false,
"params": {},
"status": "pending",
"task_id": "5a99e20de50d4979ce6d291e",
"type": "lidarsegmentation"
}
```
# Create Lidar Linking Annotation
This endpoint creates a lidarlinking task. Sometimes camera calibrations can be incorrect in 3D tasks, leading to inaccurate projections of the cuboid vertices onto 2D images (despite the 3D cuboids being accurate). The 2D/3D linking API allows users to request corrected 2D projections, each labeled with the same ID as the corresponding cuboid in 3D.
The required parameters for this task are lidar\_task, annotation\_type, and instruction. The lidar\_task is the ID of the completed lidar task to request corrected 2D projections. The annotation\_type is the 2D annotation type to return, either imageannotation (preferred), annotation, cuboidannotation, or polygonannotation. The format of these annotation types is described in more detail in box, cuboid, polygon, and imageannotation documentation, respectively.
You must provide additional markdown-enabled instructions via the instruction parameter.
It is strongly recommended for you to flesh out your Markdown instructions with many examples of tasks being done correctly and incorrectly.
If successful, Scale will immediately return the generated task object, of which you should at least store the task\_id.
lidarlinking tasks can also be created automatically after a lidarannotation or lidarsegmentation task is completed. To learn more about this, see Dependent Tasks.
The name of the [project](/docs/api-reference/projects) to associate this task with.
***
The name of the batch to associate this task with. Note that if a batch is specified, you need not specify the project, as the task will automatically be associated with the batch's project. For Scale Rapid projects specifying a batch is required. See **[Batches section](/docs/api-reference/batches)** for more details.
***
A markdown-enabled string or iframe embed google doc explaining how to do the task. You can use [markdown](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet) to show example images, give structure to your instructions, and more. See our [instruction best practices](/docs/write-your-instructions) for more details. For Scale Rapid projects, DO NOT set this field unless you specifically want to override the project level instructions.
***
The full url (including the scheme `http://` or `https://`) or email address of the [callback](/docs/api-reference/callbacks) that will be used when the task is completed.
***
The 2D annotation type to return, either `imageannotation` (preferred), `annotation`, `cuboidannotation`, or `polygonannotation`
***
The ID of the completed lidar task to request corrected 2D projections for.
***
This field is used to add additional attributes that you would like to capture per annotation. See [Annotation Attributes](/docs/api-reference/annotation-attributes) for more details about annotation attributes.
***
Indices of the CameraImages in the `lidarannotation` task to request corrected 2D projections for. Defaults to all cameras.
***
The list of events to annotate. By default, we will annotate every event in each camera, but we can specify which camera we want to annotate specific events in (see event\_camera\_ids)
***
Can be used to specify which cameras we want to annotate each event in. By default we annotate every event in for every camera. Every event specified here must also be in events\_to\_annotate
***
Whether or not to allow labelers to draw additional annotations onto the images (note that newly drawn annotations may not have consistent IDs across cameras). True by default.
***
Whether or not to allow labelers to modify any aspect of an annotation (labels, attributes and position). True by default.
***
Whether or not to allow labelers to modify an annotation's position (but not neceesarily its label or attributes). `can_edit_annotations` must also be set to true. True by default.
***
Whether or not to allow labelers to delete an annotation that was carried over from the `lidarannotation` task. `can_edit_annotations` must also be set to true. True by default.
***
The sample rate of frames whose 2D projections will be adjusted. If you are capturing 10Hz LIDAR but only want adjusted labels in 2Hz, you can set this parameter to 5 and achieve that. If the sample rate is `k` and there are `n` frames total, we will adjust (1-indexed), the `1`st, `k + 1`-th, `2k + 1`-th, ... , `floor((n - 1) / k) * k + 1`-th, and `n`th frames' projections. Note that `labeling_sample_rate` samples from the set of frames that are left after the sampling done in the original lidar task; e.g. if you submit a lidar task with 20 frames and `labeling_sample_rate=4`, frames \[1, 5, 9, 13, 17, 20] will be labeled with cuboids. A subsequent lidar linking task with `labeling_sample_rate=2` performed on the aforementioned task will label frames \[1, 9, 17, 20].
***
An array of strings or [LabelDescription](/docs/api-reference/annotation-attributes) objects to be merged with the original `lidarannotation` task's list of labels. Defaults to an empty array. Do not use if `annotation_type=imageannotation`; specify additional labels using `geometries` instead.
***
Labels to skip projection generation for (must be a subset of the `labels` param of the original `lidarannotation` task). Defaults to an empty array.
***
(required if `annotation_type=imageannotation`). An object mapping `box`, `polygon`, `line`, `point`, `cuboid`, or `ellipse`to Geometry objects, indicating the geometry with which annotations should be drawn and the geometry of generated projections.
***
(required if `annotation_type=imageannotation`). The default geometry to use when creating projections for annotations if the label to geometry mapping isn't explicitly specified in `geometries`. Must be `box`, `cuboid`, or `polygon`.
***
If set, all attributes from the `lidar_task` will be copied to the linking task -- see [Inherited Lidar Attributes](/docs/api-reference/annotation-attributes)[ ](/docs/api-reference/annotation-attributes)for details.
***
The amount of padding in pixels added to the top, bottom, left, and right of each video frame. This allows labelers to extend annotations outside of the image. `0` by default.
***
The amount of padding in pixels added to the left and right of each video frame. Overrides `padding` if set. `0` by default.
***
The amount of padding in pixels added to the top and bottom of each video frame. Overrides `padding` if set. `0` by default.
***
Read-only shapes to be drawn on each frame of the lidarlinking task. Each `LidarLinkingLayers` object has a required `url` field, which is a `string` link to a Scale-accessible file containing an array of [`Layers`](/docs/api-reference/sensor-fusion-lidar-reference), one for each frame of the `lidarlinking` task. See [example file](https://scale-static-assets.s3-us-west-2.amazonaws.com/uploads/lidarLinkingLayers0.json) for reference. Each `LidarLinkingLayers` object also has a required `camera_id` field (which is an `integer` describing the camera ID to which the `Layers` correspond to). It is not required to define `LidarLinkingLayers` for every camera.
***
A set of key/value pairs that you can attach to a task object. It can be useful for storing additional information about the task in a structured format. Max 10KB.
***
A value of 10, 20, or 30 that defines the priority of a task within a project. The higher the number, the higher the priority.
***
A arbitrary ID that you can assign to a task and then query for later. This ID must be unique across all projects under your account, otherwise the task submission will be rejected. See [Avoiding Duplicate Tasks](/docs/api-reference/tasks) for more details.
***
If set to be true, if a task errors out after being submitted, the unique id on the task will be unset. This param allows workflows where you can re-submit the same unique id to recover from errors automatically
***
Arbitrary labels that you can assign to a task. At most 5 tags are allowed per task. You can query tasks with specific tags through the task retrieval API.
***
```python theme={null}
import requests
# Replace with your actual API key
API_KEY = 'your_api_key_here'
# Define the URL for the API endpoint
url = "https://api.scale.com/v1/task/lidarlinking"
# Define the payload for the LIDAR linking task
payload = {
"instruction": "**Instructions:** Please label all the things",
"annotation_type": "imageannotation",
"can_add_annotations": True,
"can_edit_annotations": True,
"can_edit_annotation_positions": True,
"can_delete_annotations": True,
"lidar_task": "607385eadfd77d0029a84084"
}
# Set up the headers for the request
headers = {
"accept": "application/json", # Specify that we want the response in JSON format
"content-type": "application/json" # Specify the content type of the request
}
# Adding authentication to the POST request
# The auth parameter requires a tuple with the API key and an empty string
response = requests.post(url, json=payload, headers=headers, auth=(API_KEY, ''))
# Print the response text to see the result
print(response.text)
```
```json theme={null}
{
"task_id": "string",
"created_at": "string",
"type": "lidarlinking",
"status": "pending",
"instruction": "string",
"is_test": false,
"urgency": "standard",
"metadata": {},
"project": "string",
"callback_url": "string",
"updated_at": "string",
"work_started": false,
"params": {
"labeling_sample_rate": 1,
"geometries": [],
"annotation_type": "imageannotation",
"task_id": "607385eadfd77d0029a84084",
"default_geometry": "box"
}
}
```
# Change Dependent Task Options
This endpoint is used to change the options associated with dependent tasks. This can only be done if the original task is not complete, not just if dependent tasks have not been created.
POST this endpoint with a `**dependents**` object to update the dependent tasks options
Root task of the dependent tasks.
***
Definitions of the tasks that will be created once this task is complete.
***
Whether or not to wait for a customer audit to fix/approve a task before creating the dependent tasks.
***
```python theme={null}
import requests
# Replace with your actual API key and task ID
API_KEY = 'your_api_key_here'
TASK_ID = 'your_task_id_here'
# Define the URL for the API endpoint
url = f"https://api.scale.com/v1/task/{TASK_ID}/dependents/options"
# Define the payload for the dependent task options
payload = {
"defs": [
{
"labels": ["label1", "label2"],
"type": "lidarsegmentation",
"instruction": "**Instructions",
"callback_url": "http://www.example.com/callback",
"annotation_type": "imageannotation"
}
]
}
# Set up the headers for the request
headers = {
"accept": "application/json", # Specify that we want the response in JSON format
"content-type": "application/json" # Specify the content type of the request
}
# Adding authentication to the POST request
# The auth parameter requires a tuple with the API key and an empty string
response = requests.post(url, json=payload, headers=headers, auth=(API_KEY, ''))
# Print the response text to see the result
print(response.text)
```
# Force Dependent Tasks Creation
This endpoint creates dependent tasks and skips the audit (assuming require\_audit = true on a particular task). This will fail if the task is not completed, or dependent tasks have already been created.
Root task of the dependent tasks.
***
```python theme={null}
import requests
# Replace with your actual API key and task ID
API_KEY = 'your_api_key_here'
TASK_ID = 'your_task_id_here'
# Define the URL for the API endpoint
url = f"https://api.scale.com/v1/task/{TASK_ID}/dependents/force_creation"
# Set up the headers for the request
headers = {
"accept": "application/json", # Specify that we want the response in JSON format
"content-type": "application/json" # Specify the content type of the request
}
# Adding authentication to the POST request using the 'auth' parameter
# The auth parameter requires a tuple with the API key and an empty string
response = requests.post(url, headers=headers, auth=(API_KEY, ''))
# Print the response text to see the result
print(response.text)
```
# Sensor Fusion Reference
Source: https://api-reference.scale.com/docs/api-reference/sensor-fusion-reference
Sensor Fusion Scene Format Overview What is a Sensor Fusion Scene? A Sensor Fusion Scene (SFS) file is a container format designed to represent a 3D scene by synchronizing data from multiple sensor types like LiDAR, came
# Sensor Fusion Scene Format Overview
## What is a Sensor Fusion Scene?
A Sensor Fusion Scene (SFS) file is a container format designed to represent a 3D scene by synchronizing data from multiple sensor types like LiDAR, cameras, and radar. Its strength lies in its efficiency and synchronization. An SFS file stores data in compressed formats like MP4 or in binary typed arrays, synchronized to a single time scale. This structure makes SFS files smaller and faster to process, while also allowing Scale to leverage work across 2D and 3D pipelines through projection and the linking of objects.
## Core SFS Functionality
To build an SFS file, you'll primarily work with classes from the [`scale_sensor_fusion_io`](https://pypi.org/project/scale-sensor-fusion-io/) library. The most common are `PosePath`, `CameraSensor`, and `LidarSensor`.
* **`PosePath`**
* A `PosePath` defines an object's movement and orientation through the scene over time. It's constructed from two main components: an `index` containing an array of timestamps, and `data` containing a corresponding array of poses. Each pose is represented as an array of seven numbers: three for the position (x, y, z) and four for the orientation as a scalar-last quaternion (qx, qy, qz, qw).
* **`CameraSensor`**
* This class represents a single camera in the scene. A `CameraSensor` requires a unique `id`, a `PosePath` to describe its position over time, and its `intrinsics` (focal length, principal point, etc.). For workflows where 3D annotations are not generated, the intrinsics can be a dummy variable; their use is in reprojection of cuboids into 2D space. Its most important feature is the `video` object, which holds the video content as a binary byte array (`Uint8Array`) along with an array of timestamps for each frame and the video's FPS.
* **`LidarSensor`**
* This class represents a single LiDAR. Like the camera, it requires a unique `id` and a `PosePath`. The point cloud data itself is organized into a list of `frames`. Each frame has a start `timestamp` and a `points` object containing the actual data for that capture period. The `points` object holds several binary arrays:
* `positions`: A `Float32Array` of all point (x, y, z) coordinates.
* `intensities`: A `Uint8Array` of intensity values for each point.
* `timestamps`: A `Uint32Array` of per-point timestamps, which is valuable for "frameless" or high-frequency data.
* `colors`: An optional `Uint8Array` of RGB values, which can be generated by projecting camera colors onto the point cloud.
SFS also has functionality for a generic `PointSensor` as well as a `RadarSensor`. All of these data classes can be found within the `scale_sensor_fusion_io` library. Additional reference material can also be found at the bottom of this page.
## Sensor Fusion Scene Creation Workflow
Producing an SFS file can be completed with a series of steps which focus on data synchronization, object creation, and scene assembly. A high level overview of these steps is below with code samples included for each step to the right.
### Step 1: Synchronize Timestamps
This is the most critical preparation step. All sensor data must exist on a single, unified timeline. This timeline is not tied to a specific sensor, but exists across the entire span of the scene.
1. First, gather all frame-level timestamps from every sensor.
2. Identify the single earliest timestamp among them. This will become your scene's "zero" point, or `t=0`.
3. Convert all timestamps to microseconds and subtract the "zero" timestamp from every timestamp in your dataset. The original start time is saved separately and stored as the `time_offset` in the final scene file.
```text theme={null}
### IDENTIFY MINIMUM TIMESTAMP ###
USEC_IN_SEC = 1e6
min_camera_timestamps = []
min_lidar_timestamps = []
for camera in cameras:
min_camera_timestamps.append(camera_timestamps[camera.name].min())
for lidar in lidars:
min_lidar_timestamps.append(lidar["t"].min())
minimum_timestamp = min(
min_camera_timestamps + min_lidar_timestamps) * USEC_IN_SEC
```
### Step 2: Ingest and Format Data
With timestamps aligned, you can load and format the raw sensor data.
* **For cameras**: Convert your image sequences into MP4 video files. There are several utility functions within `scale_sensor_fusion_io` which can assist with this. The result should be a byte array for each camera's video stream.
* **For LiDAR**: Load the point cloud data for each frame. The goal is to get the position, intensity, and per-point timestamp data into NumPy arrays or similar structures that can be easily converted to the required binary format.
* **For Poses**: Load all pose data format it into the (N, 7) array structure required by the `PosePath` class. Since each sensor needs its own `PosePath`, the `PosePath` should be specific to the sensor; each sensor will need to be adjusted using its extrinsic calibration with respect to the frame of the Ego `PosePath`. If pose data does not have the same timestamps as your `CameraSensor` or `LidarSensor`, the poses can be interpolated using helper functions like `apply_interpolated_transform_to_points` or the class function `PosePath.interpolate(timestamps)`.
```text theme={null}
### GENERATE POSES ###
''' The poses need to be normalized with the minimum timestamp and should be converted to world coordinates.
If poses are already recorded in world coordinates, this step can be skipped. '''
import scale_sensor_fusion_io as sfio
with open("pose.json", "r") as f:
poses_json = json.load(f) # array of poses for each frame
pose_values = np.array([list(pose.values()) for pose in poses_json])
pose_values[:, 0] = pose_values[:, 0] * USEC_IN_SEC - minimum_timestamp
all_poses = sfio.PosePath(
data=np.fromiter(
(
(
pose["tx"],
pose["ty"],
pose["tz"],
pose["qx"],
pose["qy"],
pose["qz"],
pose["qw"],
)
for pose in poses_json
),
dtype=np.dtype((float, 7)),
count=len(poses_json),
),
index=pose_values[:, 0],
)
# converts poses to world coordinates from ego coordinates
all_poses_world = sfio.PosePath(all_poses.invert().as_matrix()[0] @ all_poses)
```
### Step 3: Instantiate Sensor Objects
Now, use the formatted data to create your sensor objects.
* For each camera: Create a `PosePath(timestamps, pose_data)` and then a `CameraSensor(id, intrinsics, PosePath, video_data)`.
* For the LiDAR: Create a `PosePath(timestamps, pose_data)`. Then, for each lidar frame, package point cloud data as `LidarSensorPoints(timestamps, positions, intensities)` and create a `LidarSensor(id, PosePath, list_of_lidar_frames)`.
```text Generate Camera Sensor theme={null}
### GENERATE CAMERA SENSORS ###
from pyquaternion import Quaternion
from scale_lidar_io import transform
def generate_camera_sensor(camera_name: str, timestamps: List[int], poses: List[dict]) -> sfio.CameraSensor:
### CAMERA INTRINSICS ###
# Load the camera intrinsics and distortion parameters from the json file
with open("camera_param.json", "r") as f:
intrinsics_json = json.load(f)
intrinsics = sfio.CameraIntrinsics(
fx=intrinsics_json[camera_name]["intrinsics"]["fx"],
fy=intrinsics_json[camera_name]["intrinsics"]["fy"],
cx=intrinsics_json[camera_name]["intrinsics"]["cx"],
cy=intrinsics_json[camera_name]["intrinsics"]["cy"],
width=CAMERA_WIDTH,
height=CAMERA_HEIGHT,
distortion=sfio.CameraDistortion.from_dict(
intrinsics_json[camera_name]
),
)
### CAMERA POSES ###
''' poses_json and pose_values were already calculated earlier. These poses are in the GPS/IMU frame,
so we need to adjust poses to account for the camera extrinsics. '''
lidar_to_cam_transform = transform.Transform.from_Rt(
R=Quaternion([extrinsics[camera_name]["qw"],
extrinsics[camera_name]["qx"],
extrinsics[camera_name]["qy"],
extrinsics[camera_name]["qz"]]),
t=np.array([extrinsics[camera_name]["tx"],extrinsics[camera_name]["ty"],extrinsics[camera_name]["tz"]])
)
poses = poses @ lidar_to_cam_transform.matrix
poses = poses.interpolate(timestamps)
### GENERATE VIDEO ###
# use this utility function to encode the video correctly
sfio.utils.video_helpers.generate_video(
image_files=sorted(
glob.glob(os.path.join(SAMPLE_CAMERA_PATH, camera_name, f"*.jpg"))
),
target_file=os.path.join(SAMPLE_CAMERA_PATH, camera_name, "video.mp4"),
fps=CAMERA_FPS,
)
video = sfio.CameraSensorVideo(
timestamps=timestamps,
content=np.fromfile(
os.path.join(SAMPLE_CAMERA_PATH, camera_name, "video.mp4"), dtype=np.uint8
),
fps=CAMERA_FPS,
)
return sfio.CameraSensor(
id=camera_name,
intrinsics=intrinsics,
video=video,
poses=poses,
)
```
```text Generate Lidar Sensor theme={null}
### GENERATE LIDAR SENSOR ###
def generate_lidar_sensor(
dataframes: List[pd.DataFrame], lidar_timestamps: List[int], poses: List[dict]) -> sfio.LidarSensor:
### CREATE LIDAR FRAMES ###
lidar_interp_poses = poses.interpolate(lidar_timestamps)
interp_points = [
sfio.utils.pose_path_helpers.apply_interpolated_transform_to_points(
lidar_df[["x", "y", "z"]].values,
lidar_df["time (s)"].values,
poses,
)
for lidar_df in dataframes
]
lidar_points = [
sfio.LidarSensorPoints(
positions=interpted.astype(np.float32),
timestamps=df["time (s)"].to_numpy(dtype=np.uint32),
intensities=df["intensity"].to_numpy(dtype=np.uint8),
)
for df, interpted in zip(dataframes, interp_points)
]
frames = [
sfio.LidarSensorFrame(points=points, timestamp=timestamp)
for points, timestamp in zip(lidar_points, lidar_timestamps)
]
### LIDAR POSES ###
for i in range(len(dataframes)):
dataframes[i][["x", "y", "z"]] = interp_points[i]
# this sensor is using the world frame, make sure we indicate that here
return sfio.LidarSensor(
id="lidar_0", poses=lidar_interp_poses, frames=frames, coordinates="world"
)
```
### Step 4: Assemble and Serialize the Scene
Combine all the created objects into a single root `Scene` object.
1. Create a list containing all the `CameraSensor` and `LidarSensor` objects you instantiated.
2. Instantiate the main `Scene` object, passing it your list of `sensors` and the `time_offset` you calculated in Step 1.
3. Finally, use an SFS-specific encoder (like the `JSONBinaryEncoder` from the library) to write your `Scene` object to an .sfs file. This encoder correctly handles the conversion of your data into the efficient binary format.
```text theme={null}
### CREATE SENSORS ###
lidar_sensor = generate_lidar_sensor(lidar_dataframes, lidar_timestamps, all_poses_world)
camera_sensors = [
generate_camera_sensor(camera.name, camera_timestamps[camera.name], all_poses_world)
for camera in cameras
]
sensors = camera_sensors + [lidar_sensor]
### ASSEMBLE SCENE ####
scene = sfio.Scene(
sensors=sensors, time_offset=minimum_timestamp, time_unit="microseconds" # type: ignore
)
# convert the Scene object to an sfs object
sfs_scene = sfio.model_converters.sfs.to_scene_spec_sfs(scene)
encoder = sfio.JSONBinaryEncoder()
encoder.write_file(os.path.join(SAMPLE, "example_scene.sfs"), sfs_scene) # type: ignore
```
### Step 5: Verify the Output
After saving the file, you should verify its integrity. You can do this programmatically by using the library's `parse_and_validate_scene` function, which checks for structural correctness. After confirming the structural integrity of the SFS file, you can visualize the output of the scene using your Scale Dashboard `debug` entrypoint, located [here.](https://dashboard.scale.com/lidarlite/?entrypoint=debug) You must be logged in to open the Scale Dashboard.
```text theme={null}
### VERIFY SCENE ###
import pprint
pp = pprint.PrettyPrinter(depth=6)
def test0(sfs_scene_path):
print("Test 0")
raw_data = read_file(sfs_scene_path)
result = parse_and_validate_scene(raw_data)
if not result.success:
pp.pprint(asdict(result))
else:
print("Scene parsed successfully")
test0(os.path.join(SAMPLE, "example_scene.sfs"))
```
## Scale Annotations
After Scale has completed annotating your task, the next step is to [retrieve](https://scale.com/docs/api-reference/tasks#retrieve-a-task) that task and ingest the annotations. Scale’s API allows you to easily retrieve single tasks or a large number of tasks, but you’ll need to parse the annotations to leverage them for model training.
If you submitted a *legacy* task, you will receive a JSON response which corresponds to the legacy format you submitted. You can find more information about those responses in the corresponding *Task Reference* page.
If you submitted a Sensor Fusion or Multi-Stage task, you will receive a response in an .SFS or .BS5 format. As covered in the *Sensor Fusion Task Reference*, an SFS/BS5 file is a custom format used by Scale to more efficiently store 2D and 3D data, and under the hood, it is comprised of 3 sections: a JSON Object header, a zero byte padding, and a binary array.
The JSON header contains information about the file format, the time unit, time offset, as well as annotations and attributes. A final field, labeled \$items, contains pointers to the binary arrays if relevant to help reconstruct certain response formats. It is at this point that the procedure for parsing annotation depends on the type of annotation required.
### Sparse Data Annotations
For annotations like 3D cuboids, 2D bounding boxes, keypoints, text, or other types which do not require comprehensive data context to return, responses can be converted to a common JSON type with minor manipulation. The byte array for these responses will not contain any data, so removing the zero byte buffer will yield a standard JSON object.
```text theme={null}
response = requests.get(url)
response.raise_for_status()
cleaned_text = response.text.rstrip('\x00')
data = json.loads(cleaned_text)
```
### Dense Data Annotations
For tasks in Lidar Semantic Segmentation (LSS) or Image Semantic Segmentation, reconstructing the annotations requires large amounts of data, since every point or pixel respectively has a label assigned. The data required to assemble the full annotation is stored in the binary array portion of the BS5 or SFS response, and must be read using our binary decoder. A sample of that has been included below:
```text theme={null}
from scale_json_binary import read_file
import os
import urllib.request
def save_file(url: str, filename: str) -> None:
"""
Downloads a file from a URL, like the Scale Annotation response
"""
urllib.request.urlretrieve(url, filename)
def read_binary_json(url: str) -> dict:
"""
Save the binary JSON locally, read into a JSON object, and then delete the tmp file
"""
response_filename = 'tmp_task_response.json'
save_file(url, response_filename)
response = read_file(response_filename)
os.remove(response_filename)
return response
```
### Response Format
Once you have the annotations for your task ingested, you’ll notice that for each annotation, there are a number of fields you can retrieve data from, some of which have Scale annotations, some of which include task data like time offset and time unit, and one which is available for customer metadata, assigned upon task creation.
You may notice that each annotation does not have values for every timestamp in the task, in SFS responses, we only include position information when the value has changed over time. Static objects and objects seen when the vehicle is at rest may not have a value for every timestamp; this is expected behavior, and.
```text theme={null}
{
"version": "1.0", # or "5.1"
"annotations": [
{
"id": "UUID",
"type": "Annotation Type", # cuboid, box, line, etc
"stationary": "Boolean",
"label": "string", # defined by taxonomy
"path": {
"timestamps" : {},
"values" : {}
},
"attributes" : {}, # defined by taxonomy
},
... ]
"attributes" :
{}
"time_offset": int, # from SFS creation
"time_unit" : "microseconds", # from SFS creation
"metadata" : {}, # customer defined metadata
"$items" : [], # representation of Binary Arrays
}
```
### Prelabel Generation
Including model predictions in your task submission is one feature that customers leverage when they want to Scale to assess their model’s performance while also ensuring that their data is annotated with the highest quality and level of review.
Annotations can be included in the task by creating Annotation Objects in an attached “hypothesis” SFS file. These objects can be any of the many annotation types which Scale delivers, including but not limited to:
* 3D Cuboids
* 2D Bounding Boxes
* Polylines
* Polygons
* Lidar Point Labels (LSS)
Each annotation type requires different arguments to generate, which can be found in scale\_sensor\_fusion\_io in the types/spec.py portion of the library or in the extended reference at the bottom of this page. For any geometric annotations which require an AnnotationPath, please make sure that they reflect the coordinate system of your scene, whether ego or world.
An example payload including the hypothesis is shown below:
```python theme={null}
payload = {
"project": project,
"scene_format": "sensor_fusion",
"attachments": ["s3://bucket/sample.sfs"],
"instruction": "This is a test task",
"hypothesis": {"annotations": {"url": "s3://bucket/hypothesis.sfs"}},
}
```
There are a few common errors we see when creating Sensor Fusion Scenes for the first time
1. `timestamps` not recorded in microseconds (1e-6)
`timestamps` must be saved in microseconds across all sensors and poses. If your sensors have a higher or lower sample rate than 1MHz, simply divide or multiply the timestamps accordingly
1. Tasks erroring on Scale’s platform
`SensorFusionScene` objects greater than 1.5 gigabytes can sometimes cause timeouts when being fetched on the Scale platform. We would encourage you to experiment with methods to reduce the size of your SFS scenes to below 1.5gb by adjusting the length of your scene, voxelization of pointclouds, or reduction in video frame rate where appropriate. Within the [`scale_sensor_fusion_io`](https://pypi.org/project/scale-sensor-fusion-io/) there are functions like `generate_video` which can leverage MP4 compression to significantly reduce the size of SFS files. Feel free to reach out to your technical team to strategize a process that works best for you.
1. Egocentric 3D scenes
With a proper `PosePath`, you should see point clouds that represent static objects as static in our SFS visualization tool. If you notice that the vehicle is static while point clouds move around you, check to make sure that you’re adjusting point clouds to compensate for the path of your vehicle. A helpful function for this is `apply_interpolated_transform_to_points` located in the [`scale_sensor_fusion_io`](https://pypi.org/project/scale-sensor-fusion-io/) library.
1. Incorrect Camera Projections
When hovering over each camera view in your debug viewer, if the camera’s Field of View (FOV) doesn’t match the expected orientation of the camera, confirm that each camera has been transformed in relation to a common pose path. If each camera has an extrinsic calibration, ensure that they’re calibrated against a common point on the vehicle.
A `PosePath` represents the position and orientation of a sensor or an object in the scene at different `timestamps`. The timestamps numbers may not be the same timestamps used in other Sensor objects, as pose timestamps might have been interpolated.
PosePath has the following field:
* `timestamps`: An array of numbers representing the timestamps at which the sensor or object was at a specific position and orientation.
* `values`: An array of arrays containing the \[x, y, z, qx, qy, qz, qw] components of the pose (position and scalar-last quaternion).
- These values will be in either ego-centric or world coordinates, depending on the “coordinates” field of the sensor. If there are no “coordinates” field provided, the field will be parsed as world coordinates
- If “ego” value is provided, a GPSSensor must be provided to use as the “ego” pose.
### **Points Sensor**
A PointsSensor is a sensor that captures points in 3D space. It has the following fields:
* `id` : The unique identifier of the sensor.
* `type` : The string "points" indicating this is a points sensor.
* `parent_id` (optional): The unique identifier of the parent sensor if it exists.
* `points` : An object containing the following fields:
* positions : An array of 3D positions represented as a Float32Array.
* colors (optional): An array of RGB colors represented as a Uint8Array.
### **Lidar Sensor**
A LidarSensor is a sensor that captures 3D points with optional intensity, colors and per-point timestamp data. It has the following fields:
* `id` : The unique identifier of the sensor.
* `type` : The string "lidar" indicating this is a lidar sensor.
* parent\_id (optional): The unique identifier of the parent sensor if it exists.
* `poses` : A PosePath object that defines the path of the sensor.
* coordinates (optional): A string representing the coordinate system the lidar is in, either "ego" or "world". It’s world by default.
* `frames` : An array of frame objects containing the following fields:
* `timestamp` : The start timestamp of the frame.
* `points` : An object containing the following fields:
* `positions` : A binary array of 3D positions represented as a Float32Array.
* `colors` (optional): A binary array of RGB colors represented as a Uint8Array
* `intensities` (optional): A binary array of intensity values represented as a Uint8Array.
* `timestamps` (optional): A binary array of timestamps represented as a Uint32Array or Uint64Array . If scene.time\_unit == "nanosecond" this field will be parsed as Uint64Array, otherwise it will be parsed as Uint32Array
### **Radar Sensor**
A RadarSensor is a sensor that captures 3D radar points with optional direction and length data values. It has the following fields:
* `id `: The unique identifier of the sensor.
* `type` : The string "radar" indicating this is a radar sensor.
* `parent_id `(optional): The unique identifier of the parent sensor if it exists.
* `poses` : A PosePath object that defines the path of the sensor.
* coordinates (optional): A string representing the coordinate system the lidar is in, either "ego" or "world". It’s world by default.
* `frames` : An array of frame objects containing the following fields:
* `timestamp` : The start timestamp of the frame.
* `points` : An object containing the following fields:
* `positions` : A binary array of 3D positions represented as a Float32Array.
* `directions` (optional): A 3D binary array of directions represented as a Float32Array.
* `lengths` (optional): A binary array of length values represented as a Float32Array.
* `timestamps` (optional): A binary array of timestamps represented as a Uint32Array or Uint64Array . If scene.time\_unit == "nanosecond" this field will be parsed as Uint64Array, otherwise it will be parsed as Uint32Array
### **Camera Sensor**
A CameraSensor is a sensor that captures 2D images or video. It has the following fields:
* `id`: The unique identifier of the sensor.
* `type` : The string "camera" indicating this is a camera sensor.
* `parent_id`(optional): The unique identifier of the parent sensor if it exists.
* `poses`: A PosePath object that defines the path of the camera.
* `coordinates` (optional): A string representing the coordinate system the lidar is in, either "ego" or "world". It’s world by default.
* `intrinsics` : An object containing the intrinsic parameters of the camera:
* `fx` : The focal length in the x direction.
* `fy `: The focal length in the y direction.
* `cx` : The x coordinate of the principal point.
* `cy` : The y coordinate of the principal point.
* `width`: The width of the camera image.
* `height` : The height of the camera image.
* distortion (optional): An object containing the following fields:
* `model` : A string representing the distortion model used, one of "brown\_conrady", "mod\_equi\_fish", "mod\_kannala", "fisheye", "fisheye\_rad\_tan\_prism", or "cylindrical".
* `params` : An array of floats representing the distortion parameters required to apply the model.
* `video` (optional): An object containing the following fields if the camera captures video:
* `timestamps` : An array of timestamps for each frame indicating the start of the frame
* `content` : A binary Uint8Array containing the video data encoded as mp4.
* `fps` : The frames per second of the video.
* images (optional): An array of objects containing the images:
* `timestamp` : The timestamp of the image.
* `content` : A binary Uint8Array containing the image encoded as jpg.
### **Odometry Sensor**
An OdometrySensor is a sensor that captures the movement of the vehicle (or "ego") that the sensors are attached to. It has the following fields:
* `id `: The unique identifier of the sensor.
* `type` : The string "odometry" indicating this is an odometry sensor.
* `parent_id` (optional): The unique identifier of the parent sensor if it exists.
* `poses` : A PosePath object that defines the poses of the odometry.
## Cuboid Annotation
A CuboidAnnotation is an annotation that labels an object as a cuboid. It has the following fields:
* id : The unique identifier of the annotation.
* type : The string cuboid indicating this is a cuboid annotation.
* parent\_id (optional): The id of a parent annotation.
* stationary (optional): A boolean indicating whether the object is stationary or not.
* label (optional): A string representing the label of the object.
* path : An object that defines the path of the cuboid annotation, with the following fields:
* timestamps: The timestamps of the keyframes of the cuboid path.
* values: An array of arrays containing the \[x, y, z, px, py, pz, roll, pitch, yaw] components of the cuboid at each path timestamp
* attributes (optional): An array of AttributePath objects that define the attributes of the annotation and per-sensor attributes.
* activations (optional): An array of objects that defines the per-sensor activations of the annotation in frameless scenes. There is one object per sensor, and each object contains the following fields:
* sensor\_id The unique identifier of the sensor for which the cuboid is activated.
* timestamps : An array of timestamps (in microseconds) for each cuboid activation.
* durations : An array of durations (in microseconds) for each cuboid activation.
* cuboids (optional): An array of arrays containing the \[dx, dy, dz, px, py, pz, pitch, roll, yaw] components of a computed cuboid for each activation timestamp.
* projections (optional): An array of objects that define per-sensor projections of the annotation, with the following fields:
* sensor\_id: The unique identifier of the 2D sensor for which the cuboid is projected.
* timestamps: An array of camera timestamps for each cuboid projection.
* boxes: An array of arrays containing the \[x, y, width, height] components of the 2D bounding box of the object in the image. It could contain undefined if the box could not be projected or was deleted by the user.
* confirmed (optional): An array of boolean values indicating whether a value is confirmed or not by a user.
* cuboids (optional): An array of arrays containing the \[dx, dy, dz, px, py, pz, pitch, roll, yaw] components of the cuboid for each projection timestamp.
## 2D Box Annotation
The Box2DAnnotation type represents a 2D bounding box annotation in the scene. It has the following fields:
* id : The unique identifier of the annotation.
* type : The string box\_2d indicating this is a box annotation.
* parent\_id (optional): The id of a parent annotation.
* stationary (optional): A boolean indicating whether the object is stationary or not.
* label (optional): A string representing the label of the object.
* attributes (optional): An array of AttributePath objects that define the attributes of the annotation and per-sensor attributes.
* sensor\_id: The unique identifier of the sensor if the annotation is sensor-specific.
* path : An object that defines the path of the box annotation, with the following fields:
* timestamps: The timestamps of the path.
* values: An array of arrays containing the \[left, top, width, height] components of the box.
## 2D Polyline Annotation
The Polyline2DAnnotation type represents a 2d polyline annotation in the scene. It has the following fields:
* id : The unique identifier of the annotation.
* type : The string polyline\_2d indicating this is a polyline annotation.
* parent\_id (optional): The id of a parent annotation.
* stationary (optional): A boolean indicating whether the object is stationary or not.
* label (optional): A string representing the label of the object.
* attributes (optional): An array of AttributePath objects that define the attributes of the annotation and per-sensor attributes.
* sensor\_id: The unique identifier of the sensor if the annotation is sensor-specific.
* path : An object that defines the path of the polyline annotation, with the following fields:
* timestamps: The timestamps of the path.
* values: An array of arrays containing the \[\[1st timestamp's x\_0, y\_0, x\_1, y\_1, ..., x\_n, y\_n], \[2nd timestamp's x\_0, y\_0, ...]] vertices of the polyline per timestamp
## 2D Polygon Annotation
The Polygon2DAnnotation type represents a 2D polygon annotation in the scene. It has the following fields:
* id : The unique identifier of the annotation.
* type : The string polygon\_2d indicating this is a 2D polygon annotation.
* parent\_id (optional): The id of a parent annotation.
* stationary (optional): A boolean indicating whether the object is stationary or not.
* label (optional): A string representing the label of the object.
* attributes (optional): An array of AttributePath objects that define the attributes of the annotation and per-sensor attributes.
* sensor\_id: The unique identifier of the sensor if the annotation is sensor-specific.
* path : An object that defines the path of the polygon annotation, with the following fields:
* timestamps: The timestamps of the path.
* values: An array of arrays containing the \[\[1st timestamp's x\_0, y\_0, x\_1, y\_1, ..., x\_n, y\_n], \[2nd timestamp's x\_0, y\_0, ...]] vertices of the polygon.
## 2D Point Annotation
The Point2DAnnotation type represents a 2d Point annotation in the scene. It has the following fields:
* id : The unique identifier of the annotation.
* type : The string point\_2d indicating this is a point annotation.
* parent\_id (optional): The id of a parent annotation.
* stationary (optional): A boolean indicating whether the object is stationary or not.
* label (optional): A string representing the label of the object.
* attributes (optional): An array of AttributePath objects that define the attributes of the annotation and per-sensor attributes.
* sensor\_id: The unique identifier of the sensor if the annotation is sensor-specific.
* path : An object that defines the path of the point annotation, with the following fields:
* timestamps: The timestamps of the path.
* values: An array of arrays containing the \[\[1st timestamp's x, y], \[2nd timestamp's x, y]] point coordinates
## Polygon Annotation
The PolygonAnnotation type represents a polygon annotation in the scene. Polygon has the invariant that the points are on a plane. It has the following fields:
* id : The unique identifier of the annotation.
* type : The string polygon indicating this is a polygon annotation.
* parent\_id (optional): The id of a parent annotation.
* stationary (optional): A boolean indicating whether the object is stationary or not.
* label (optional): A string representing the label of the object.
* attributes (optional): An array of AttributePath objects that define the attributes of the annotation and per-sensor attributes.
* sensor\_id (optional): The unique identifier of the sensor if the annotation is sensor-specific.
* path : An object that defines the path of the polygon annotation, with the following fields:
* timestamps: The timestamps of the path.
* values: An array of arrays containing the \[\[1st timestamp's x\_0, y\_0, x\_1, y\_1, ..., x\_n, y\_n], \[2nd timestamp's x\_0, y\_0, ...]] vertices of the polygon.
## Topdown Polygon Annotation
The TopdownPolygonAnnotation type represents a topdown 2D polygon annotation with elevation data. Points in a topdown polygon don’t need to lie on a plane. It has the following fields:
* id : The unique identifier of the annotation.
* type : The string polygon\_topdown indicating this is a polygon annotation.
* parent\_id (optional): The id of a parent annotation.
* stationary (optional): A boolean indicating whether the object is stationary or not.
* label (optional): A string representing the label of the object.
* attributes (optional): An array of AttributePath objects that define the attributes of the annotation and per-sensor attributes.
* sensor\_id (optional): The unique identifier of the sensor if the annotation is sensor-specific.
* path : An object that defines the path of the polygon annotation, with the following fields:
* timestamps: The timestamps of the path.
* values: An array of arrays containing the \[\[1st timestamp's x\_0, y\_0, x\_1, y\_1, ..., x\_n, y\_n], \[2nd timestamp's x\_0, y\_0, ...]] vertices of the polygon.
## Polyline Annotation
The PolylineAnnotation type represents a polyline annotation in the scene. It has the following fields:
* id : The unique identifier of the annotation.
* type : The string polyline indicating this is a polyline annotation.
* is\_closed (optional): Whether or not this annotation is closed. If true, the first and last vertices will be connected to represent a “3D polygonal loop”
* parent\_id (optional): The id of a parent annotation.
* stationary (optional): A boolean indicating whether the object is stationary or not.
* label (optional): A string representing the label of the object.
* attributes (optional): An array of AttributePath objects that define the attributes of the annotation and per-sensor attributes.
* sensor\_id (optional): The unique identifier of the sensor if the annotation is sensor-specific.
* path : An object that defines the path of the polyline annotation, with the following fields:
* timestamps: The timestamps of the path.
* values: An array of arrays containing the \[\[1st timestamp's x\_0, y\_0, z\_0, x\_1, y\_1, z\_1, ..., x\_n, y\_n, z\_n], \[2nd timestamp's x\_0, y\_0, z\_0,...]] vertices of the polyline.
## Points Annotation / Keypoints
Both LidarTopdown 3D points and LidarAnnotation keypoints are represented by this
* id : The unique identifier of the annotation.
* type : The string points indicating this is a point annotation.
* parent\_id (optional): The id of a parent annotation.
* stationary (optional): A boolean indicating whether the object is stationary or not.
* labels (optional): An array of strings representing the label of each point, respecting the index of each point.
* attributes (optional): An array of AttributePath objects along with an additional point\_index field .
* sensor\_id (optional): The unique identifier of the sensor if the annotation is sensor-specific.
* paths : An object that defines the path of the point annotation, with the following fields:
* id: Unique identifier for each point within the annotation (optional in old files).
* timestamps: The timestamps of the path.
* values: An array of arrays containing the \[x, y, z] coordinates of the points.
* projections (optional): An array of arrays containing objects that define per-sensor projections of the annotation, with the following fields:
* sensor\_id: The unique identifier of the 2D sensor for which the cuboid is projected.
* timestamps: An array of camera timestamps for each cuboid projection.
* points: An array of arrays containing the \[x, y] components of the 2D bounding box of the object in the image. It could contain undefined if the point could not be projected or was deleted by the user.
* confirmed (optional): An array of boolean values indicating whether a value is confirmed or not by a user.
* positions (optional): An array of arrays containing the \[x, y, z] components of the point for each projection timestamp.
## Event Annotation
The EventAnnotation type represents an event. It has the following fields:
* id : The unique identifier of the annotation.
* type : The string event indicating this is a event annotation.
* parent\_id (optional): The id of a parent annotation.
* label (optional): A string representing the label of the object.
* attributes (optional): An array of AttributePath objects that define the attributes of the annotation and per-sensor attributes.
* start: The timestamp of the start of the event.
* duration (optional): The duration of the event
* sensor\_id: The unique identifier of the sensor if the event is sensor-specific.
## Labeled Points Annotation (LSS)
The LabeledPointsAnnotation interface represents an annotation of labeled points in a lidar sensor.
* id: A unique identifier for the annotation.
* type: The string labeled\_points indicating this is a cuboid annotation.
* parent\_id (optional): The id of a parent annotation.
* label: a string representing the label assigned to the points in the annotation.
* is\_instance: a boolean value indicating whether the annotation represents an instance or a class.
* labeled\_points: an array of objects representing the labeled points grouped by sensor and frame. Each object contains:
* sensor\_id: the unique identifier for the sensor containing the labeled points.
* sensor\_frame (optional):the frame number of the sensor, if the sensor has frames.
* point\_ids: a Uint32Array containing the indices of the labeled points in the sensor frame.
## Localization Adjustment Annotation
The LocalizationAdjustmentAnnotation represents a PosePath applied a scene to fix localization issues or convert from ego to world coordinates.
* id : The unique identifier of the annotation.
* type : The string localization\_adjustment indicating this is a localization adjustment annotation.
* parent\_id (optional): The id of a parent annotation.
* poses : A PosePath object that defines the poses of the adjustment.
## Camera Calibration Annotation
The CameraCalibrationAnnotation represents .
* id : The unique identifier of the annotation.
* type : The string camera\_calibration indicating this is a camera calibration annotation.
* sensor\_id : The unique identifier of the camera
* time\_offset (optional): Timestamp offset to apply to the camera. This is used if the camera timestamps are obviously not correct, but can be corrected with minor timestamps changes.
* parent\_id (optional): The id of a parent annotation.
* poses : A PosePath object that representing the diff between the initial camera and calibrated camera extrinsics. These poses should be applied after the current camera extrinsics are applied.
* intrinsics: Intrinsics for the calibrated camera
## Object Annotation
The ObjectAnnotation represents an object within a scene, serving as a way to group related annotations associated with a specific object.
* id : The unique identifier of the annotation.
* type : The string event indicating this is an event annotation.
* parent\_id (optional): The id of a parent annotation.
* label (optional): A string representing the label of the object.
* attributes (optional): An array of AttributePath objects that define the attributes of the annotation and per-sensor attributes.
## Link Annotation
The LinkAnnotation represents a link between two annotations. This used to represent relationships between two annotations.
* id : The unique identifier of the annotation.
* type : The string link indicating this is a link annotation.
* label: A string representing the label of the object.
* is\_bidirectional: Whether this link is a bidirectional relationship
* from\_id : The id of the annotation that this links from
* to\_id: The id of the annotation this links to
* parent\_id (optional): The id of a parent annotation.
* attributes (optional): An array of AttributePath objects that define the attributes of the annotation and per-sensor attributes.
## Group Annotation
The GroupAnnotation represents a group to which multiple child annotations can belong to, via parent\_id.
* id : The unique identifier of the annotation.
* type : The string group indicating this is a link annotation.
* label (optional): A string representing the label of the object.
* parent\_id (optional): The id of a parent annotation.
A Scene is an interface that represents a sensor fusion scene. It has the following fields:
* `version`: A string representing the version of the scene format. It should be 1.0
* `sensors`(optional): An array of sensor objects that describe the sensors in the scene.
* `annotations`(optional): An array of annotation objects that describe the annotations in the scene.
* `attributes `(optional): An array of AttributePath objects that describe the attributes of the scene-level and sensor-level attributes.
* `time_offset `(optional): Scene-level field used to set a reference point in time after making the scene relative to that specific moment.
This is used as the path key in all geometric annotations. It captures the geometry per timestamp:
* `timestamps`: List\[int] - the timestamps of the path
* `values`: List\[List\[float]] - An array of arrays containing \[\[1st timestamp's x\_0, y\_0, x\_1, y\_1, ..., x\_n, y\_n], \[2nd timestamp's x\_0, y\_0, ...]]
Attributes are used to add additional information to annotations. The value of an attribute can be a string, number, or an array of strings.
## Attribute Value
An attribute value can be string, number or string\[].
## Attribute Path
An AttributePath represents the values of an attribute at different timestamp. It has the following fields:
* `name`: A string representing the name of the attribute.
* `sensor_id `(optional): The unique identifier of the sensor if the attribute is sensor-specific.
* `static` (optional): A boolean indicating whether the attribute is static or not. Default is false.
* `timestamps` : An array of timestamps for each value.
* `values` : An array of AttributeValue representing the values of the attribute at each timestamp.
# Sensor Fusion Tasks
Source: https://api-reference.scale.com/docs/api-reference/sensor-fusion-tasks
Create Sensor Fusion Tasks
# Create Sensor Fusion Tasks
\[**Recommended**] We recommend using this [**SDK documentation**](https://github.com/scaleapi/scaleapi-python-client) to create Sensor Fusion tasks.
This endpoint creates a `sensorfusion` task. In this task, annotators load the sensor fusion scene and apply various annotations to mark objects or segment the point cloud. Sensor fusion scenes may contain 3D point clouds (from LiDARs, radars, and/or point sensors) and 2D images/videos (from camera sensors). Supported annotations include 3D cuboids, point cloud segmentation, 3D keypoints, and top-down polygons.
PosePath, CameraSensor, and LidarSensor.
List of URLs to the Sensor Fusion Scene objects you'd like to be labeled. For sensor fusion tasks, there must only be one url provided
***
The format of the scene provided. Currently only supports `sensor_fusion`
The name of the project to associate this task with.
***
A markdown-enabled string or iframe embed google doc explaining how to do the task. You can use [markdown](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet) to show example images, give structure to your instructions, and more.
List of annotation definitions to use for labeling.
The full url (including the scheme http\:// or https\://) of the callback when the task is completed.
***
A arbitrary ID that you can assign to a task and then query for later. This ID must be unique across all projects under your account, otherwise the task submission will be rejected. See **[Avoiding Duplicate Tasks](/docs/api-reference/data-engine-reference#avoiding-duplicate-tasks)** for more details
***
If set to be true, if a task errors out after being submitted, the unique id on the task will be unset. This param allows workflows where you can re-submit the same unique id to recover from errors automatically
***
Whether to label as a frameless task
Arbitrary labels that you can assign to a task. At most 5 tags are allowed per task. You can query tasks with specific tags through the task retrieval API.
```python theme={null}
import requests
url = "https://api.scale.com/v1/task/sensorfusion"
payload = {
"scene_format": "\"sensor_fusion\"",
"instruction": "**Instructions:** Please label all the things",
"callback_url": "https://example.com/callback"
}
headers = {
"accept": "application/json",
"content-type": "application/json"
}
response = requests.post(url, json=payload, headers=headers)
print(response.text)
```
# Tasks
Source: https://api-reference.scale.com/docs/api-reference/tasks
Create a task Retrieve a task Retrieve Multiple Tasks Cancel Task Set Task Metadata Update unique_id Delete unique_id Add Task Tag Set Task Tag Delete Task Tag Avoiding Duplicate Tasks
# Create a task
A task represents an individual unit of work to be done by a Contributor. There's a 1:1 mapping between a task and the data to be labeled. For example, there'd be 1 task for each image, video, or lidar sequence needing to be labeled.
You specify how the labeling should be done for a given task when making an API call specifying a set of task parameters to the endpoint you'd like to leverage.
Tasks have a type such as "Image Annotation", "Video Annotation", "Lidar Segmentation", or "Document Transcription".
**For information on how to create specific task types, you can click on the links below:**
* [Image Annotation Task](https://scale.com/docs/api-reference/image-and-video-tasks#create-image-annotation-task)
* [Semantic Segmentation Annotation Task](https://scale.com/docs/api-reference/image-and-video-tasks#create-semantic-segmentation-annotation-task)
* [General Video Annotation Task](https://scale.com/docs/api-reference/image-and-video-tasks#create-general-video-annotation-task)
* [Lidar Cuboid Annotation](https://scale.com/docs/api-reference/sensor-fusion-lidar-tasks#create-lidar-cuboid-annotation)
* [Lidar Segmentation Annotation](https://scale.com/docs/api-reference/sensor-fusion-lidar-tasks#create-lidar-segmentation-annotation)
* [Lidar Linking Annotation](https://scale.com/docs/api-reference/sensor-fusion-lidar-tasks#create-lidar-linking-annotation)
* [Lidar Topdown Tasks](https://scale.com/docs/api-reference/sensor-fusion-lidar-tasks#create-lidar-topdown-tasks)
* [Text Collection Task](https://scale.com/docs/api-reference/generative-ai#create-text-collection-task)
* [Named Entity Recognition Task](https://scale.com/docs/api-reference/generative-ai#create-named-entity-recognition-task)
### Task Metadata
Tasks objects have a metadata parameter. You can use this parameter to attach key-value data to tasks.
Metadata is useful for storing additional, structured information on an object - especially information that can help you ingest the task response or keep track of what content this task corresponds to.
Metadata is not used by Scale (e.g., to affect how the task is done).
Common use-cases for metadata:
* Internal identifiers
* File paths
* Scenario / Run / Case identifiers
* Environment details (time of day, location)
* Sensor information
* Guideline / Taxonomy versions
```json theme={null}
{
"task_id": "576ba74eec471ff9b01557cc",
"created_at": "2016-06-23T09:09:34.752Z",
"updated_at": "2016-06-23T09:10:02.798Z",
"completed_at": "2016-06-23T09:10:02.798Z",
"type": "categorization",
"status": "completed",
"instruction": "Would you say this item is big or small?",
"params": {
"attachment_type": "text",
"attachment": "car",
"categories": [
"big",
"small"
]
},
"callback_url": "http://www.example.com/callback",
"callback_completed": true,
"response": {
"category": "big"
},
"metadata": {},
"audits": [
{
"audited_by": "david@company.com",
"audited_at": "2016-06-24T15:32:03.585Z",
"audit_time_secs": 120,
"audit_result": "accepted",
"audit_source": "customer"
},
{
"audited_by": "auditor@scale.com",
"audited_at": "2016-06-23T10:01:02.352Z",
"audit_time_secs": 511,
"audit_result": "fixed",
"audit_source": "scale"
}
],
"tags": ["experiment_1", "owner:david"],
"unique_id": "product_experiment_dg3d9x83"
}
```
# Retrieve a task
Efficiently retrieve detailed task information, including the ability to retrieve a specific task using a task ID. This resourceful functionality allows seamless integration and thorough analysis of tasks, enhancing your workflow's data-driven capabilities. Explore task details effortlessly through this essential endpoint.
***
```python Python theme={null}
import requests
# Replace with your actual API key
API_KEY = 'your_api_key_here'
# Define the URL for the API endpoint
url = "https://api.scale.com/v1/task/576ba74eec471ff9b01557cc"
# Set up the headers for the request
headers = {
"accept": "application/json" # Specify that we want the response in JSON format
}
# Adding authentication to the GET request
# The auth parameter requires a tuple with the API key and an empty string
response = requests.get(url, headers=headers, auth=(API_KEY, ''))
# Print the response text to see the result
print(response.text)
```
```python Python SDK theme={null}
import scaleapi
# Initialize the ScaleClient with your API key
client = scaleapi.ScaleClient("YOUR_API_KEY_HERE")
# Define the task ID to retrieve
task_id = "601ba74eec471ff9b01557cc" # Replace with your actual task ID
# Retrieve the task details
task = client.get_task(task_id)
# Print the task details
print(task.as_dict())
```
```json theme={null}
{
"task_id": "601ba74eec471ff9b01557cc",
"created_at": "2021-06-23T09:09:34.752Z",
"callback_url": "http://www.example.com/callback",
"type": "imageannotation",
"status": "canceled",
"instruction": "Label every object in this image",
"params": {
"attachment": "https://example.com/image.jpg",
"geometries": {
"box": {
"objects_to_annotate": [
"vehicle",
"pedestrian"
]
}
}
},
"metadata": {
"key": "value",
"key2": "value2"
}
}
```
# Retrieve Multiple Tasks
This is a paginated endpoint that retrieves a list of your tasks.
The tasks will be returned in descending order based on created\_at time. All time filters expect an **[ISO 8601-formatted string](https://timestampgenerator.com/)**, like '2021-04-25' or '2021-04-25T03:14:15-07:00'
The pagination is based on the limit and next\_token parameters, which determine the page size and the current page we are on. The value of next\_token is a unique pagination token for each page (**[nerdy details if you were curious](https://www.mixmax.com/engineering/api-paging-built-the-right-way)**). Make the call again using the returned token to retrieve the next page.
The minimum value of created\_at for tasks to be returned
***
The maximum value of `created_at` for tasks to be returned
The minimum value of `completed_at` for tasks to be returned
The maximum value of `completed_at` for tasks to be returned
The minimum value of `updated_at` for tasks to be returned
The maximum value of `updated_at` for tasks to be returned
The status of the task - can be: `completed`, `pending`, or `canceled`
The type of the task.
The name of the project that the returned tasks must belong to.
The status of the audit result of the task can be: `accepted`, `fixed`, `commented`, or `rejected`; multiple status values can be specified at the same time either as a joint string separated by comma (e.g. `customer_review_status=accepted,fixed`), or as an array (e.g. `customer_review_status=accepted&customer_review_status=fixed`)
The name of the batch that the returned tasks must belong to
A number between 1 and 100, the maximum number of results to display per page | optional, default 100
The unique\_id of a task; multiple unique IDs can be specified at the same time either as a joint string separated by comma (e.g. `unique_id=a1,a2`), or as an array (e.g. `unique_id=a1&unique_id=a2`)
The tags of a task; multiple tags can be specified at the same time either as a joint string separated by comma (e.g. `tags=t1,t2`), or as an array (e.g. `tags=t1&tags=t2`)
Set to true if the returned Task Object should include presigned attachment urls.
A token used to retrieve the next page of results if there are more. You can find the `next_token` in your last request
```python Python theme={null}
import requests
# Replace with your actual API key
API_KEY = 'your_api_key_here'
# Define the URL for the API endpoint with query parameters
url = "https://api.scale.com/v1/tasks?status=completed&type=imageannotation&project=kitten_labeling&batch=kitten_labeling_2020-07&customer_review_status=accepted&limit=100&include_attachment_url=true"
# Set up the headers for the request
headers = {
"accept": "application/json" # Specify that we want the response in JSON format
}
# Adding authentication to the GET request
# The auth parameter requires a tuple with the API key and an empty string
response = requests.get(url, headers=headers, auth=(API_KEY, ''))
# Print the response text to see the result
print(response.text)
```
```python Python SDK theme={null}
import scaleapi
# Initialize the ScaleClient with your API key
client = scaleapi.ScaleClient("YOUR_API_KEY_HERE")
# Define optional filters (adjust as necessary)
filters = {
"project_name": "your_project_name", # Replace with your project name
"status": "completed", # Filter by task status (optional)
"created_after": "2023-01-01T00:00:00Z", # Filter by start time (optional)
"created_before": "2023-12-31T23:59:59Z", # Filter by end time (optional)
}
# Retrieve the list of tasks with optional filters
tasks = client.get_tasks(**filters)
# Print the details of each task
for task in tasks:
print(task.as_dict())
```
```json theme={null}
{
"docs": [
{
"task_id": "601ba74eec471ff9b01557cc",
"created_at": "2021-06-23T09:09:34.752Z",
"callback_url": "http://www.example.com/callback",
"type": "imageannotation",
"status": "canceled",
"instruction": "Label every object in this image",
"params": {
"attachment": "https://example.com/image.jpg",
"geometries": {
"box": {
"objects_to_annotate": [
"vehicle",
"pedestrian"
]
}
}
},
"metadata": {
"key": "value",
"key2": "value2"
}
}
],
"total": 220,
"limit": 100,
"has_more": true,
"next_token": "eyJ0YXNrX2lkIjoiNjBkYjgwZTFkYmRkNTMwMDExNDZlMzg5IiwiY3JlYXRlZF9hdCI6IjIwMjEtMDYtMjlUMjA6MjE6NTMuMjg5WiJ9"
}
```
# Cancel Task
You may only cancel pending tasks, and the endpoint will return a 400 error code if you attempt to cancel a completed task
If the task to be cancled had a unique id, specifying `**clear_unique_id=true**` will remove the unique id. Canceling tasks is idempotent such that calling this endpoint multiple times will still return a 200 success response.
***
If true, will clear a task's unique\_id, thus allowing the same unique id to be used in future tasks.
***
```python Python theme={null}
import requests
# Replace with your actual API key
API_KEY = 'your_api_key_here'
# Define the URL for the API endpoint with query parameters
url = "https://api.scale.com/v1/task/576ba74eec471ff9b01557cc/cancel?clear_unique_id=true"
# Set up the headers for the request
headers = {
"accept": "application/json" # Specify that we want the response in JSON format
}
# Adding authentication to the POST request
# The auth parameter requires a tuple with the API key and an empty string
response = requests.post(url, headers=headers, auth=(API_KEY, ''))
# Print the response text to see the result
print(response.text)
```
```python Python SDK theme={null}
import scaleapi
# Initialize the ScaleClient with your API key
client = scaleapi.ScaleClient("YOUR_API_KEY_HERE")
# Define the task ID to cancel
task_id = "601ba74eec471ff9b01557cc" # Replace with your actual task ID
# Cancel the task
client.cancel_task(task_id)
# Print confirmation
print(f"Task '{task_id}' has been canceled.")
```
```json theme={null}
{
"task_id": "601ba74eec471ff9b01557cc",
"created_at": "2021-06-23T09:09:34.752Z",
"callback_url": "http://www.example.com/callback",
"type": "imageannotation",
"status": "canceled",
"instruction": "Label every object in this image",
"params": {
"attachment": "https://example.com/image.jpg",
"geometries": {
"box": {
"objects_to_annotate": [
"vehicle",
"pedestrian"
]
}
}
},
"metadata": {
"key": "value",
"key2": "value2"
}
}
```
# Set Task Metadata
This endpoint sets the `**metadata**` field on a task.
You may set the `**metadata**` field on any existing task using valid key-value data.
Updating a task's `**metadata**` field is idempotent such that calling this endpoint multiple times will still return a 200 success response.
***
```python Python theme={null}
import requests
# Replace with your actual API key
API_KEY = 'your_api_key_here'
# Define the URL for the API endpoint
url = "https://api.scale.com/v1/task/576ba74eec471ff9b01557cc/setMetadata"
# Set up the headers for the request
headers = {
"accept": "application/json", # Specify that we want the response in JSON format
"content-type": "application/json" # Specify the content type of the request
}
# Define the payload for setting metadata
payload = {
# Add your metadata here
# For example: "metadata_key": "metadata_value"
}
# Adding authentication to the POST request
# The auth parameter requires a tuple with the API key and an empty string
response = requests.post(url, headers=headers, json=payload, auth=(API_KEY, ''))
# Print the response text to see the result
print(response.text)
```
```python Python SDK theme={null}
import scaleapi
# Initialize the ScaleClient with your API key
client = scaleapi.ScaleClient("YOUR_API_KEY_HERE")
# Define the task ID and the metadata to set
task_id = "601ba74eec471ff9b01557cc" # Replace with your actual task ID
metadata = {
"key1": "value1",
"key2": "value2"
}
# Set the metadata for the task
client.set_task_metadata(task_id, metadata)
# Print confirmation
print(f"Metadata for task '{task_id}' has been set to {metadata}.")
```
```json theme={null}
{
"task_id": "601ba74eec471ff9b01557cc",
"created_at": "2021-06-23T09:09:34.752Z",
"callback_url": "http://www.example.com/callback",
"type": "imageannotation",
"status": "canceled",
"instruction": "Label every object in this image",
"params": {
"attachment": "https://example.com/image.jpg",
"geometries": {
"box": {
"objects_to_annotate": [
"vehicle",
"pedestrian"
]
}
}
},
"metadata": {
"key": "value",
"key2": "value2"
}
}
```
# Update unique\_id
Easily enhance task management and data accuracy with the Scale Update Task Unique ID API endpoint. Seamlessly modify and optimize task identifiers, ensuring your task tracking and organization remain precise and efficient. This endpoint empowers you to maintain data integrity and adaptability, offering a streamlined way to manage unique IDs associated with tasks within your workflow. Explore this versatile endpoint to effortlessly tailor task identification according to your evolving needs.
ID of the Task to modify
***
```json Python theme={null}
import requests
# Replace with your actual API key
API_KEY = 'your_api_key_here'
# Define the URL for the API endpoint
url = "https://api.scale.com/v1/task/576ba74eec471ff9b01557cc/unique_id"
# Define the payload to set the unique ID for the task
payload = {
"unique_id": "56766ba764ee6c4761f6f9b6015657cc6" # Unique ID to be set
}
# Set up the headers for the request
headers = {
"accept": "application/json", # Specify that we want the response in JSON format
"content-type": "application/json" # Specify the content type of the request
}
# Adding authentication to the POST request
# The auth parameter requires a tuple with the API key and an empty string
response = requests.post(url, json=payload, headers=headers, auth=(API_KEY, ''))
# Print the response text to see the result
print(response.text)
```
```python Python SDK theme={null}
import scaleapi
# Initialize the ScaleClient with your API key
client = scaleapi.ScaleClient("YOUR_API_KEY_HERE")
# Define the task ID and the new unique ID
task_id = "601ba74eec471ff9b01557cc" # Replace with your actual task ID
new_unique_id = "new_unique_id_value" # Replace with the new unique ID
# Update the unique_id for the task
client.update_task_unique_id(task_id, new_unique_id)
# Print confirmation
print(f"Unique ID for task '{task_id}' has been updated to '{new_unique_id}'.")
```
```json theme={null}
{
"task_id": "601ba74e98762345bcbcaaaa",
"created_at": "2021-06-23T09:09:34.752Z",
"callback_url": "http://www.example.com/callback",
"type": "imageannotation",
"status": "completed",
"instruction": "Label every object in this image",
"params": {
"attachment": "https://example.com/image.jpg",
"geometries": {
"box": {
"objects_to_annotate": [
"vehicle",
"pedestrian"
]
}
}
},
"unique_id": "new_unique_id"
"metadata": {}
}
```
# Delete unique\_id
Enables the secure removal of task identifiers, providing you with enhanced control over your data management processes. You can confidently eliminate obsolete or redundant task unique IDs from your system, maintaining data accuracy and improving workflow organization. Seamlessly integrate this functionality into your task management workflow to ensure your records remain up-to-date and clutter-free. Explore the convenience and flexibility of the Scale Delete Task Unique ID endpoint to optimize your data management practices.
ID of the Task to modify
***
```python Python theme={null}
import requests
# Replace with your actual API key
API_KEY = 'your_api_key_here'
# Define the URL for the API endpoint
url = "https://api.scale.com/v1/task/576ba74eec471ff9b01557cc/unique_id"
# Set up the headers for the request
headers = {
"accept": "application/json" # Specify that we want the response in JSON format
}
# Adding authentication to the DELETE request
# The auth parameter requires a tuple with the API key and an empty string
response = requests.delete(url, headers=headers, auth=(API_KEY, ''))
# Print the response text to see the result
print(response.text)
```
```python Python SDK theme={null}
import scaleapi
# Initialize the ScaleClient with your API key
client = scaleapi.ScaleClient("YOUR_API_KEY_HERE")
# Define the task ID for which you want to clear the unique ID
task_id = "601ba74eec471ff9b01557cc" # Replace with your actual task ID
# Clear the unique_id for the task
client.clear_task_unique_id(task_id)
# Print confirmation
print(f"Unique ID for task '{task_id}' has been cleared.")
```
```json theme={null}
{
"task_id": "601ba74e98762345bcbcaaaa",
"created_at": "2021-06-23T09:09:34.752Z",
"callback_url": "http://www.example.com/callback",
"type": "imageannotation",
"status": "completed",
"instruction": "Label every object in this image",
"params": {
"attachment": "https://example.com/image.jpg",
"geometries": {
"box": {
"objects_to_annotate": [
"vehicle",
"pedestrian"
]
}
}
},
"metadata": {}
}
```
# Add Task Tag
With this endpoint, you can include a list of `**tags**` to be added to a task. If a `**tag**` is already associated with the task, it will be ignored to avoid duplication. Please note that setting an empty or null string as a `**tag**` is not allowed. Ensure to provide valid non-empty strings in the `**tags**` list to update the task's tags successfully.
ID of the Task to modify
***
List of tags to add to the task
***
```python Python theme={null}
import requests
# Replace with your actual API key
API_KEY = 'your_api_key_here'
# Define the URL for the API endpoint
url = "https://api.scale.com/v1/task/576ba74eec471ff9b01557cc/tags"
# Set up the headers for the request
headers = {
"accept": "application/json", # Specify that we want the response in JSON format
"content-type": "application/json" # Specify the content type of the request
}
# Define the payload to set the tags for the task
payload = [
"tag1",
"tag2",
"tag3"
]
# Adding authentication to the PUT request
# The auth parameter requires a tuple with the API key and an empty string
response = requests.put(url, headers=headers, json=payload, auth=(API_KEY, ''))
# Print the response text to see the result
print(response.text)
```
```python Python SDK theme={null}
import scaleapi
# Initialize the ScaleClient with your API key
client = scaleapi.ScaleClient("YOUR_API_KEY_HERE")
# Define the task ID and the tags to add
task_id = "601ba74eec471ff9b01557cc" # Replace with your actual task ID
tags_to_add = ["tag1", "tag2"] # Replace with the tags you want to add
# Add the tags to the task
client.add_task_tags(task_id, tags_to_add)
# Print confirmation
print(f"Tags {tags_to_add} have been added to task '{task_id}'.")
```
# Set Task Tag
This endpoint allows you to set a completely new list of `tags` on a task. This will replace all currently existing `tags` on it if the target exists.
ID of the Task to modify
***
List of tags to add to the task
***
```python Python theme={null}
import requests
# Replace with your actual API key and task ID
API_KEY = 'your_api_key_here'
TASK_ID = 'task_id_here'
# Define the URL for the API endpoint
url = f"https://api.scale.com/v1/task/{TASK_ID}/tags"
# Set up the headers for the request
headers = {
"accept": "application/json", # Specify that we want the response in JSON format
"content-type": "application/json" # Specify the content type of the request
}
# Define the payload to set the tags for the task
payload = [
"tag1",
"tag2",
"tag3"
]
# Adding authentication to the POST request
# The auth parameter requires a tuple with the API key and an empty string
response = requests.post(url, headers=headers, json=payload, auth=(API_KEY, ''))
# Print the response text to see the result
print(response.text)
```
```python Python SDK theme={null}
import scaleapi
# Initialize the ScaleClient with your API key
client = scaleapi.ScaleClient("YOUR_API_KEY_HERE")
# Define the task ID and the tags to add
task_id = "601ba74eec471ff9b01557cc" # Replace with your actual task ID
tags_to_add = ["tag1", "tag2"] # Replace with the tags you want to add
# Add the tags to the task
client.set_task_tags(task_id, tags_to_add)
# Print confirmation
print(f"Tags {tags_to_add} have been added to task '{task_id}'.")
```
# Delete Task Tag
With this endpoint, you can include a list of `**tags**` to be added to a task. If a `**tag**` is already associated with the task, it will be ignored to avoid duplication. Please note that setting an empty or null string as a `**tag**` is not allowed. Ensure to provide valid non-empty strings in the `**tags**` list to update the task's tags successfully.
ID of the Task to modify
***
List of tags to add to the task
***
```python Python theme={null}
import requests
# Replace with your actual API key and task ID
API_KEY = 'your_api_key_here'
TASK_ID = 'task_id_here'
# Define the URL for the API endpoint
url = f"https://api.scale.com/v1/task/{TASK_ID}/tags"
# Set up the headers for the request
headers = {
"accept": "application/json", # Specify that we want the response in JSON format
"content-type": "application/json" # Specify the content type of the request
}
# Adding authentication to the DELETE request
# The auth parameter requires a tuple with the API key and an empty string
response = requests.delete(url, headers=headers, auth=(API_KEY, ''))
# Print the response text to see the result
print(response.text)
```
```python Python SDK theme={null}
import scaleapi
# Initialize the ScaleClient with your API key
client = scaleapi.ScaleClient("YOUR_API_KEY_HERE")
# Define the task ID and the tags to add
task_id = "601ba74eec471ff9b01557cc" # Replace with your actual task ID
tags_to_add = ["tag1", "tag2"] # Replace with the tags you want to add
# Add the tags to the task
client.delete_task_tags(task_id, tags_to_add)
# Print confirmation
print(f"Tags {tags_to_add} have been added to task '{task_id}'.")
```
# Avoiding Duplicate Tasks
Creating duplicate tasks is an issue every team should be mindful to avoid.
Scale AI provides two different mechanisms to prevent duplicate tasks from being created in its task creation endpoints. This allows you to resubmit requests that may have failed in transit or otherwise need to be retried without the risk of creating duplicate tasks.
### Option 1: The `unique_id` field
The `unique_id` field is a field available on every task type Scale provides.
Once a `unique_id` has been submitted to Scale, any future task creation requests with the same `unique_id` will fail with a [409 error](/docs/api-reference/errors) that also conveniently points to the conflicting task.
Values passed into the `unique_id` field are permanently associated with the task and will always be returned to you when retrieving tasks from our platform.
You are able to query for tasks directly based on the `unique_id` field at any point with our [Task Retrieval endpoints](/docs/api-reference/tasks#retrieve-a-task)[.](/docs/api-reference/tasks#retrieve-a-task)
### Best Practices:
1. `unique_id` should be thought of as your own customizable id for a task. Ideally, this id can be easy to look up based on the data you have available on your side. A good `unique_id` might be the filename being submitted, or other types of metadata like a scene or run id that you use internally.
2. `unique_id` is set globally across all projects and task types. If you'd like to enforce uniqueness only within a project or task type, we recommend simply prepending or appending the project or task type to the unique id itself, problem solved!
### Option 2: The `Idempotency-Key` header
To use this feature, provide a header `Idempotency-Key: \`. You, the client, are responsible for ensuring the uniqueness of your chosen keys. We recommend using V4 UUIDs.
The results of requests specifying an idempotency key are saved. If we later receive a matching request with the same idempotency key, the saved response will be returned, and no additional task will be created. Note that this behavior holds even when the response is an error. Keys are removed after 24 hours.
If an incoming request has the same idempotency key as a saved request, but the two requests do not match in parameters or the users associated with the two requests are different, we will return a [409 error](/docs/api-reference/errors).
In rare situations, we may return a [429 error](/docs/api-reference/errors) if two matching requests with identical idempotency keys are made simultaneously. In this case, it is safe to retry.
**When would I use this instead of the **`**unique_id**`** field?** Using the header-based approach is useful in retry logic that catches network or other transient failure modes when you would be immediately retrying the exact same request. Specifically, the feature that allows you to seamlessly get the same task response back if the payload didn't change makes for easier code integrations.
You are able to use both options simultaneously as well.
### Workflow Support
Because Unique Ids are permanently tied to a task, this means if something unexpected happened, it can be hard to recover on your own. We have added two features to help support more robust workflows.
**Canceling Tasks** When canceling tasks, there is a `clear_unique_id` query parameter you can specify on the request. See the [Cancel Task endpoint](/docs/api-reference/tasks#cancel-task) for more details.
**Errored Tasks** Sometimes after a task is submitted, it can run into an error, especially in regards to processing attachments.
Everywhere you can specify a unique id, you can also specify `clear_unique_id_on_error: true`. As the param name suggests, if the task reaches an error status, the unique id will automatically be unset, such that you could submit a new task with the same new unique id.
```python theme={null}
import scaleapi
from scaleapi.tasks import TaskType
from scaleapi.exceptions import ScaleDuplicateResource
# Initialize the ScaleClient with your API key
client = scaleapi.ScaleClient("YOUR_API_KEY_HERE")
# Define the task payload
payload = {
"project": "your_project_name", # Replace with your project name
"callback_url": "http://www.example.com/callback",
"instruction": "Draw a box around each object.",
"attachment_type": "image",
"attachment": "http://i.imgur.com/v4cBreD.jpg",
"unique_id": "unique_task_id_12345", # Replace with a unique identifier for the task
"geometries": {
"box": {
"objects_to_annotate": ["Object"],
"min_height": 10,
"min_width": 10,
}
},
}
# Attempt to create the task, handling duplicates
try:
task = client.create_task(TaskType.ImageAnnotation, **payload)
print(f"Task created successfully: {task.as_dict()}")
except ScaleDuplicateResource as err:
print(f"Task creation failed: {err.message}")
```
```json theme={null}
{
"unique_id": "s3://bucket/file.png",
"instruction": "Do the thing",
"callback_url": "you@gmail.com",
...
}
```
```json theme={null}
{
"status_code": 409,
"error": 'The unique_id ("s3://bucket/file.png") is already used for a different task (602c399c6d092c00115aa3c9).'
}
```
```shell theme={null}
curl "https://api.scale.com/v1/task/comparison" \
-u "{{ApiKey}}:" \
-H "Idempotency-Key: UNIQUE_IDENTIFIER"
-d callback_url="http://www.example.com/callback" \
...
```
# Taxonomy Service
Source: https://api-reference.scale.com/docs/api-reference/taxonomy-service
Taxonomy Service is self-service tool to easily create, modify, and publish new taxonomy versions
# Overview
The **Taxonomy Service** is an intuitive, self-service tool designed to help teams efficiently create, update, and manage taxonomies. With its streamlined version management feature, users can easily track changes and ensure consistency across multiple versions of taxonomy-related tasks. By providing a unique identifier for each taxonomy, this tool guarantees that your data is always properly labeled and organized.
Whether you're managing simple annotations or complex tasks like Lidar segmentation, the Taxonomy Service simplifies the process, so you can focus more on accuracy and less on manual management. It ensures that every step in your workflow, from data labeling to final version tracking, is handled seamlessly, reducing errors and maintaining high standards across your operations.
### Supported Task Types
Currently, Taxonomy Service supports the following task types:
* **Segment Annotation**
* **LidarAnnotation**
* **Lidar TopDown**
* **LidarLinking**
* **LidarSegmentation**
* **VideoAnnotation**
* **ImageAnnotation**
* **Sensor Fusion**
### Taxonomy SRN
A **Taxonomy SRN** (Service Resource Name) is a unique identifier for a specific taxonomy version. It allows you to compare and track different versions of taxonomies across tasks ensuring consistency and accuracy for your labeling requirements.
Format
` srn:scale:avcv:taxonomy:taxonomy:\/\/\/\`
Example
`srn:scale:avcv:taxonomy:taxonomy:654dd0785yytd22829f342hs/user_myproject_lidar_test/v1/0ac7578c6b32107816e071aa11353bab61bb6h1f`
### Task Creation Process
When Taxonomy Service is enabled on your project, tasks that are created will automatically be updated with the latest taxonomy that was published using Taxonomy Service. Therefore, you can remove taxonomy JSON during the task creation process. Note, that if you keep the taxonomy JSON in the task creation API call, the\*\* taxonomy JSON will override\*\* any taxonomy published using Taxonomy Service.
### **Dependent Tasks**
Each dependent project must have its own taxonomy defined in Taxonomy Service. For example, if you have a LidarAnnotation project and dependent LidarLinking and Lidartopdown projects, each must have its taxonomy defined separately in Teddy.
### **Dependent projects**
Taxonomy Service supports dependent projects, managing their taxonomies automatically. Ensure each project is set up with its own taxonomy, and remove any taxonomy-related parameters from the scripts and task parameters.
### Tracking
Use the Taxonomy Metrics tool to review and monitor where the Taxonomies and policies were applied.
Once you are on the Taxonomy Metrics page, you can track the batches and the status for these Tasks understanding the progress as well as work done for them.
# Getting Started
### Access
You can access the Taxonomy Service from the Scale dashboard page on the left panel for each project that is enabled.
### **Set Up**
Create and publish a taxonomy using the Taxonomy Editor. Reference the Section “How to use the Taxonomy editor for further details”
Ensure to ask your account manager to enable the "Pull last published Taxonomy when creating Tasks" property in the project configurations to allow auto-pulling of the latest taxonomy.
### **Adding annotations**
1. Access the Taxonomy Editor from the project dashboard page.
1. Once on Taxonomy Service, you will see on the left sidebar a list of all current Taxonomies versions, only the current draft is editable, and all previously published versions are static and cannot be edited.
1. To work on changes we need first to select our current Taxonomy version that in on draft, for this click on the Taxonomy with a Draft status as next:
1. Once you have selected your draft, you will see the Taxonomy details will appear on the right side of the screen, on top being our main edit controls and on the below part our taxonomy details.
1. By default this view will not allow us to edit the taxonomy, in order for us to be able to edit first we need to click on the “Edit” button on our Editor tooling
1. Now we are on the editor view! Lets try now to add a new label:
2. First lets click on the + Label button, this will add a new base label that we can modify.
3. You will see a new Label appear under the current existing list with the name\*\* new-label-#\*\*
1. To change the new label name just click on it, and type the new name you will want. You can also modify other label names with the same process just click on the label you want to adjust
1. To delete a label just click on the Trash bin button on the right side of any label previously added.
1. Now let's review how we can add Leaf labels, this is very straightforward, to get started let's first click on the add child button on the right side of our label:
1. Once you have added a new leaf label, we can go ahead and rename them as needed. Note that you can also add multiple leaf labels as shown next:
1. Now that we have added a label lets review how to add an attribute for them.
### **Adding Attributes**
The process of adding attributes is very similar to adding an annotation, the next steps show how to add an attribute and some of their main setups we need to take into account:
1. First, we need to get into the Attribute view as next, clicking on the Attributes tab above our Taxonomy view as next:
1. Once we are on the Attribute view, you will see any previously added attribute listed, in addition, we can add a new Attribute by clicking on the button + Attribute.
1. Now that we have added a new attribute it is important to set its values on how it will work, the most common setup options we will have are as next:
1. Details: General Attribute information
2. Attribute Name: Display the name of the attribute to labelers.
3. Description: Description for attribute available for labelers.
4. Attribute Input type: This would vary depending on the task type but it sets how the attribute values would be entered:
5. Category: Multiple choice input
6. Angle: Heading in degrees. ( eg. Sign direction )
7. Number: Any numerical input
8. Text: Open text input
9. Linked: Link to another annotation present.
10. Label Condition: To which label in specific this attribute be applied, if left empty applies to all labels.
11. Attribute condition: Used when the current attribute depends on another attribute value, you need to reference the other attribute in addition to the value expected for it.
Once you are done working with both your Annotation and Attribute changes now it's time to submit this taxonomy for Scale review! Let's now click on **Validate & Save** in order to send this draft to Scale.
### Publishing
Create a draft taxonomy, review and make necessary changes, and publish it when ready. Published taxonomies are automatically used for new tasks.
Each time you publish a new version of the taxonomy, a new draft will be created where you can work on new changes, and the published version will be locked.
### Proposal and Reviews
You can review taxonomies and submit proposals for changes. The Scale team will review these proposals and implement approved changes, ensuring that taxonomies remain up-to-date and accurate.
* When you submit a taxonomy, Scale will be able to analyze the changes and reject this proposal.
* When Scale submits a taxonomy, you will be able to review, accept, or reject changes.
This draft should cover the key points for each section, tailored to the needs of the changes happening, once done you can submit and the Scale will review it and ensure to come back with questions if needed.
# AWS S3
Source: https://api-reference.scale.com/docs/aws-s3
AWS S3 S3 IAM Access If you use AWS S3 to store data, if you submit tasks with attachments as s3: protocol URIs, rather than http: or https:, we will use the S3 API to fetch your data. For example, instead of sending htt
# AWS S3
## **S3 IAM Access**
If you use AWS S3 to store data, if you submit tasks with attachments as `s3:`\*\* protocol URIs\*\*, rather than `http:` or `https:`, we will use the S3 API to fetch your data. For example, instead of sending `https://s3-us-west-2.amazonaws.com/bucket/key`, you would send `s3://bucket/key`.
We can either fetch your data using **IAM Delegated Access** (preferred, more secure) or **Cross-account Access**.
### **IAM Delegated Access**
To access S3 data in your AWS account, Scale can **[assume a role in your account](https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html)**, which has permission to access data in your S3 buckets. This role must be named `ScaleAI-Integration`.
To set up IAM Delegated Access:
1. As a team admin or manager, go to **[dashboard.scale.com/settings/integrations](https://dashboard.scale.com/settings/integrations)**.
2. In another window, create a new role in the **[AWS IAM Console](https://console.aws.amazon.com/iam/home?#/roles)**
* Select `Another AWS account` for the Role Type.
* Enter `307185671274` (Scale's Account ID) as the Account ID.
* Check `Require external ID`, and enter the external ID displayed in the AWS section of the Integrations Settings page.
* Do not check `Require MFA`.
1. For permissions, either attach a policy that grants appropriate access, or create a policy. A sample role policy is shown below.
2. Name the role `ScaleAI-Integration`.
3. Return to the Scale Dashboard and enter your AWS account ID.
> Sample Role Policy for IAM Delegated Access
```
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "scales3access",
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::YOUR_BUCKET_NAME/*",
"arn:aws:s3:::YOUR_BUCKET_NAME"
],
}
]
}
```
Note that if you enable the AWS integration for your account, we will not attempt to fetch attachments from our account (`307185671274`) directly; the policies described in **Cross-account Access** will not work.
### **Cross-account Access**
If IAM delegated access is not configured, we will directly fetch attachments from your S3 bucket, using AWS account ID `307185671274` (canonical ID `ae2259599e139df6cedb60b6300bcafa1c652aff129aa3d887477b6d4abf2e47`), which you can grant access to on a **[per-object basis using ACLs](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/set-object-permissions.html)** or using **[bucket policies](https://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html)**.
For most customers, we recommend setting a **Bucket Policy** that shares the bucket's contents with Scale's account.
A sample Bucket Policy below - please be sure to replace `YOUR_BUCKET_NAME` with the name of your bucket, leaving the `/*` as shown or replacing it with a more specific bucket path to further restrict access.
Please note that if using Access Control Lists (ACLs), each object must have its ACL individually updated to grant read access to our account, as Bucket ACLs cannot grant read permissions to the objects inside.
> Sample Bucket Policy for Cross-account Access
```
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "scale-s3-access",
"Action": [
"s3:GetObject"
],
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::307185671274:root"
]
},
"Resource": "arn:aws:s3:::YOUR_BUCKET_NAME/*"
}
]
}
```
Please note that this authentication mechanism suffers from the **[confused deputy problem](https://en.wikipedia.org/wiki/Confused_deputy_problem)** — a third party that can guess your S3 URLs will be able to submit tasks with your data.
# Azure Blob Storage
Source: https://api-reference.scale.com/docs/azure-blob-storage
Azure Blob Storage Azure Blob Storage Access If you use Azure Blob Storage, you can grant access to your Blob Storage resources by completing the Azure access delegation process. Scale has registered the Scale AI applica
# Azure Blob Storage
## **Azure Blob Storage Access**
If you use Azure Blob Storage, you can grant access to your Blob Storage resources by completing the Azure access delegation process. Scale has registered the Scale AI application as an Azure **[multi-tenant application](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-convert-app-to-be-multi-tenant)** that can access resources in your Azure subscription on your behalf.
After completing the access delegation process, blob storage resource URIs (i.e. of the form `https://\{storageaccount\}.blob.core.windows.net/\{container\}/\{key\}`) will be fetched using the Scale AI service principal, and you will be able to submit blob URIs to the API that are not publicly accessible.
The process involves the following steps:
* **Consenting to grant Scale AI the permissions it requires to access resources in your subscription.**
* **Assigning the Scale AI app an appropriate role.**
### **Application Consent**
As an administrator or manager of your Scale AI team, go to the integrations tab in the **[settings page](https://dashboard.scale.com/settings)**, click the **Connect to Azure** button. Azure displays the resource permissions requested by the application.
Click **Accept** to allow Azure to grant permission to Scale AI to access resources in your subscription. You will still need to grant the application a role to access Blob Storage data. Note that after providing application consent, the Scale AI app will stop using anonymous credentials to fetch attachments sent in by your team.
### **Role-Based Access**
As part of the access delegation process, you must assign a role to the `Scale AI` application service principal to read data from your storage accounts. We recommend assigning the *Storage Blob Data Reader* role for the particular storage accounts or containers to retrieve data from. Alternatively, you can create a custom role that provides only the minimum permissions necessary. See the **[Azure](https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-rbac-portal?toc=/azure/storage/blobs/toc.json#assign-rbac-roles-using-the-azure-portal)** **[docs](https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-rbac-portal?toc=/azure/storage/blobs/toc.json#assign-rbac-roles-using-the-azure-portal)** for instructions on how to assign the role.
### **Disconnecting from Azure**
To stop the Scale AI service principal from authenticating via Azure AD to access your user's storage accounts, use the **Unlink from Azure** button in the integrations tab in the settings page. Note that this does not revoke permissions from the Scale AI service principal in Azure, nor does it uninstall the Scale AI app from your subscription; those must done using the Azure portal or the Azure CLI.
# Customer Support
Source: https://api-reference.scale.com/docs/customer-support
Customer Support Availability Support is available during our normal business hours: 9:00 AM - 5:00 PM Pacific Time Monday-Friday with the exception of major US holidays. Contacting Support For Pro and Nucleus Customers:
# Customer Support
## **Availability**
Support is available during our normal business hours: 9:00 AM - 5:00 PM Pacific Time Monday-Friday with the exception of major US holidays.
## **Contacting Support**
**For Pro and Nucleus Customers:** You can contact your Engagement Manager or Field Engineer for support on any quality / technical issues and they will help triage / connect you to the appropriate team for help.
**For Customers interested in learning more about Scale:** Please contact **[sales@scale.com](mailto:sales@scale.com)** for more information!
# Downloading data
Source: https://api-reference.scale.com/docs/downloading-data
Downloading Data To retrieve your data from Scale, navigate to the "Deliveries" tab in the dashboard, or use the Scale API. Once you have the option you want to use and found the Scale tasks you want to download, you'll
# Downloading Data
To retrieve your data from Scale, navigate to the "Deliveries" tab in the dashboard, or use the Scale API.
Once you have the option you want to use and found the Scale tasks you want to download, you'll want to review the callback format to know what each of the response fields means. Each type of task has its own callback format. For example, this is how the response format for Image Annotation with Polygons looks (at the bottom): Polygons or for 2d segmentation: Callback Format.
### Option 1: Scale API - Listing Multiple Tasks
Our API docs have support for listing multiple tasks.
### Option 2: Python SDK
If you're working with Python, we highly recommend leveraging our Python SDK.
The specific function we'll be using with the Python SDK is the "List Tasks" function.
# GenAI Data Engine
Source: https://api-reference.scale.com/docs/genai-data-engine
Scale Generative AI Data Engine enables rapid creation of tailored, high-quality datasets curated by vetted subject matter experts to train the world’s most advanced models. Access customized data annotations, model eval
# Getting Started
Scale Generative AI Data Engine enables rapid creation of tailored, high-quality datasets curated by vetted subject matter experts to train the world’s most advanced models. Access customized data annotations, model evaluations, and RLHF data via API, SDK, or web frontend.
Explore Data Engine integrations in our [Gen AI Data Engine Documentation](https://docs.genai.scale.com).
# GenAI Platform
Source: https://api-reference.scale.com/docs/genai-platform
The Scale GenAI Platform empowers modern enterprises to rapidly develop, test and deploy Generative AI applications for custom use cases, using their proprietary data assets. It includes an API, SDK and web frontend whic
# Getting Started
The Scale GenAI Platform empowers modern enterprises to rapidly develop, test and deploy Generative AI applications for custom use cases, using their proprietary data assets. It includes an API, SDK and web frontend which abstract the flexible use of both open and closed-source resources, providing full-stack capabilities that meet enterprise security and scalability standards.
Explore more in our [GenAI Platform Documentation](https://docs.gp.scale.com/home).
# Google Cloud Storage
Source: https://api-reference.scale.com/docs/google-cloud-storage
Google Cloud Storage Google Cloud Storage Access If you use Google Cloud Storage to store data, if you submit tasks with attachments as gs: protocol URIs, rather than http: or https:, we will use the Google Cloud Storage
# Google Cloud Storage
## **Google Cloud Storage Access**
If you use Google Cloud Storage to store data, if you submit tasks with attachments as `gs:`\*\* protocol URIs\*\*, rather than `http:` or `https:`, we will use the Google Cloud Storage API to fetch your data. For example, instead of sending `https://storage.googleapis.com/bucket/key`, you would send `gs://bucket/key`.
We can either fetch your data using **Service Account Impersonation** (preferred, more secure) or **Cross-project Access**.
### **Service Account Impersonation**
To access Cloud Storage data in your GCP project, Scale can **[impersonate a service account within that project](https://cloud.google.com/iam/docs/creating-short-lived-service-account-credentials)**, which has permission to access data in Cloud Storage.
To set up Service Account Impersonation:
1. As a team admin or manager, go to **[dashboard.scale.com/settings/integrations](https://dashboard.scale.com/settings/integrations)**.
2. In another window, navigate to the **[GCP Service Accounts](https://console.cloud.google.com/iam-admin/serviceaccounts)** page for the appropriate project.
3. Create a service account.
* The service account ID must contain an 8-character user identifier as a substring, this identifier can be found in the Google Cloud Platform section of the Integrations Settings page.
* We suggest the ID `scaleai-integrations-\{uid\}`.
1. Grant Scale's service account the ability to impersonate the newly created service account
* In the Service Accounts page on GCP, check the box associated with the newly created service account.
* In the permissions pane on the right, click `Add Principal`, you may need to click "Show Info Panel" in the top right to see this option.
* Specify `backend-bucket-access@attachment-storage-243718.iam.gserviceaccount.com` as the member, and `Service Account Token Creator` as the role.
* Save the permissions.
1. In Google Cloud Storage, assign the `Storage Object Viewer` permissions for the requisite buckets to the newly created service account.
* If you use fine-grained access controls, add the service account email as a Reader for any objects you would like to upload (if not already granted by bucket-level access).
1. Return to the Scale Dashboard and enter the email of the service account.
Note that if you enable the GCP integration for your account, we will not attempt to fetch attachments from the default service account ( `backend-bucket-access@attachment-storage-243718.iam.gserviceaccount.com`) directly; the policies described in **[GCP IAM Access](/docs/google-cloud-storage#gcp-iam-access)** will not work.
### **Cross-project Access**
If Service Account Impersonation is not configured, we will directly fetch attachments from your GCS bucket, using the GCP service account `backend-bucket-access@attachment-storage-243718.iam.gserviceaccount.com`. You can grant access to this service account on a **[per-object basis with ACLs](https://cloud.google.com/storage/docs/access-control/create-manage-lists)**, or on a **[per-bucket basis with Cloud IAM](https://cloud.google.com/storage/docs/access-control/using-iam-permissions)** **[Permissions](https://cloud.google.com/storage/docs/access-control/using-iam-permissions)**.
Please note that this authentication mechanism suffers from the **[confused deputy problem](https://en.wikipedia.org/wiki/Confused_deputy_problem)** — a third party that can guess your Cloud Storage URLs will be able to submit tasks with your data.
# Getting Started with Scale
Source: https://api-reference.scale.com/docs/index
Scale AI's mission is to accelerate the development of AI applications. To enable teams to make faster progress, we began with data - the foundation of all AI applications. Scale AI turns raw data into high-quality training data by combining machine learning powered pre-labeling and active tooling with varying levels and types of human review.
# Getting Started with Scale
Scale AI's mission is to accelerate the development of AI applications. To enable teams to make faster progress, we began with data - the foundation of all AI applications. Scale AI turns raw data into high-quality training data by combining machine learning powered pre-labeling and active tooling with varying levels and types of human review.
Guides, workflows, and product documentation.
API concepts and endpoint reference documentation.
# IP Allowlisting
Source: https://api-reference.scale.com/docs/ip-allowlisting
IP Allowlisting IP Allowlisting For non-AWS customers*, Scale uses a consistent set of IP addresses to fetch data and send callbacks, allowing for IP allowlisting of attachments sent to us, as well as for callback endpoi
# IP Allowlisting
Expanded Public IP addresses - Action requested by April 1, 2023
We are expanding our public IPs from which we may fetch resources or issue callback requests.
Action is required to prevent disruption if you use Scale's API to send attachments or receive callbacks AND have implemented allow-listed public IP addresses for Scale AI services in your network firewall egress policies.
By April 1, 2023, please ensure your firewall allows access from the additional IPs highlighted below:
```
18.246.77.0/27
2602:fb33::/45
```
## **IP Allowlisting**
For non-AWS customers\*, Scale uses a consistent set of IP addresses to fetch data and send callbacks, allowing for IP allowlisting of attachments sent to us, as well as for callback endpoints, to increase data security.
If you are enabling IP allowlisting, we request that you whitelist access to your data to all 7 listed IP addresses below, and we will only fetch content using these IP addresses. In this way, you can secure your content from the public while still allowing Scale to access it.
> Scale static IP addresses
```
52.38.24.56
35.160.30.43
35.167.66.86
52.11.250.38
54.203.55.239
18.246.77.0/27 //needed by April 1, 2023
2602:fb33::/45 //needed by April 1, 2023
```
\*If you are using AWS S3, do **not** use IP allowlisting, use S3 IAM Access instead. Requests to S3 will not necessarily originate from our static IPs.
# Key Concepts & Definitions
Source: https://api-reference.scale.com/docs/key-concepts-and-definitions
Key Concepts & Definitions To get high quality ground truth data with Scale, your first step is to create a project . Within a project, you will upload data and create tasks , which are pieces of data to be labeled. The
# Key Concepts & Definitions
To get high quality ground truth data with Scale, your first step is to create a **[project](/docs/key-concepts-and-definitions#project)** . Within a project, you will upload data and create **[tasks](/docs/key-concepts-and-definitions#task)** , which are pieces of data to be labeled. The tasks can be grouped within different **[batches](/docs/key-concepts-and-definitions#batch)** to be launched for labeling. Every task will follow the same **[taxonomy](/docs/key-concepts-and-definitions#taxonomy)** defined at the project level.
Once your data is hosted in a way that Scale can access it, you can use our UI or submit an API call to create tasks. After you have launched a batch of tasks for labeling, the statuses of your tasks will be “pending.”
* **[Scale Pro](/docs/pro-or-manage-account)** customers should expect to receive these tasks back according to the delivery schedule that we have aligned on with you. We can support extremely high and dynamic volumes customized to your needs.
Once a task has been labeled, you'll see the task status move to be “completed.” The task will now have a JSON response associated with it that you can download via our platform.
Inside the web application, you can download a given task's response, or do a bulk export over a filterable range of tasks. We have APIs to support the programmatic **[retrieval of tasks given a task ID](/docs/api-reference/tasks#retrieve-a-task)**, or to **[list all tasks meeting customizable filter criteria](/docs/api-reference/tasks#retrieve-multiple-tasks)**. Lastly, we **[fully support callbacks](/docs/api-reference/callbacks)** as tasks are moved to a completed or error status or have other actions taken on them, allowing fully programmatic access to your labeled data.
## **Task**
A task represents an individual unit of work to be done. There's a one-to-one mapping between a task and the data to be labeled. For example, there is one task for each image, video, or piece of text to be labeled and each task will have a unique Scale-generated ID. To create a task using our API, please refer to our **[API reference ](/docs/api-reference/tasks)**.
## **Project**
Within a given project, you can organize similar tasks based on instructions and the use case. All tasks will share the same instructions and annotation rules.
A project is tied to one specific annotation use case, which is associated with a task type in our API reference. You can have multiple projects per use case.
As an example, you could have one project for categorizing scenes, and another for annotating images.
Every task is tied to an explicit project to keep things organized. To create a project using our API, please refer to our **[API reference](/docs/api-reference/projects#create-project)**.
## **Batch**
**On Scale Pro**: For high-volume projects, batches can optionally be used to further divide work inside a project. For example, batches can tie tasks to specific datasets you use internally, or can be used to note which tasks were part of a weekly submission.
To create and launch a batch, you can refer to our **[API reference](/docs/api-reference/batches#create-a-batch)**.
## **Taxonomy**
A taxonomy is a collection of labels and information associated with those labels, which is defined at the project level. We refer to each label as an **annotation**. Available annotations include box, polygon, point, ellipse, cuboid, event, text response, list selection, tree selection, date, linear scale, and ranking. Within a taxonomy, there can be \*\*classes of annotations \*\*(i.e. different types of an annotation), \*\*global attributes \*\*(i.e. information about the whole task) and **annotation attributes **(i.e. information associated with a specific annotation). We can also create** link attributes** (i.e. relationships between two annotations).
Example: One use case may involve drawing boxes around all cats and dogs in an image and indicating the total number of cats and dogs in the image. For each cat, we want to indicate if they are sleeping or not sleeping. For each dog, we want to indicate which cat they are looking at (if applicable).
We would create a taxonomy with two *classes* of box *annotations* (one for cat and one for dog). Within the cat class, we would define an *annotation attribute* of “sleeping or not sleeping” so that we can associate each box drawn around a cat with whether or not the cat is sleeping. Within the dog class, we would define a *link attribute* such that we can relate a dog box with a cat box and indicate a “looking at” relationship. Finally, we would create a *global attribute* that asks the labeler to indicate the total number of cats and dogs in the image.
# Nucleus
Source: https://api-reference.scale.com/docs/nucleus
Nucleus is a dataset management platform that helps ML teams build better datasets. Bring your data, labels, and model predictions together to debug your models and improve your datasets.
# Getting Started
Nucleus is a dataset management platform that helps ML teams build better datasets. Bring your data, labels, and model predictions together to debug your models and improve your datasets.
Check out the full documentation for Nucleus at **[nucleus.scale.com/docs](http://nucleus.scale.com/docs)**.
# Overview
Source: https://api-reference.scale.com/docs/overview
Overview Hello! Welcome to Scale AI documentation. We are excited to work with you to accelerate the development of AI. Explore our documentation to quickly deliver value from your AI investments with high quality data.
# Overview
Hello! Welcome to Scale AI documentation.
We are excited to work with you to accelerate the development of AI. Explore our documentation to quickly deliver value from your AI investments with high quality data.
## Scale Pro
Scale Pro is the high-leverage data platform for AI-enabled businesses. We provide best-in-class customer experience through labeling products and concepts that come from the future, clean powerful interfaces, flexible tools and platforms to enable fast iteration, and the highest model improvement/\$ ratio in the industry.
```
✅ Seamlessly initiate labeling with our API
✅ Work with dedicated Engagement Managers who will help you set up labeling projects fully customized for your use case
✅ Scalably label production volumes of data, including complex 3D and Sensor Fusion data formats
✅ Receive the highest quality labeled data, guaranteed (via SLAs)
```
## Nucleus
Nucleus is the ultimate dataset management tool for machine learning teams looking to go beyond surface-level model evaluations. Nucleus helps you dig deeper into your data, correct failure points, and improve overall model accuracy. With Nucleus you can:
```
✅ Visualize and analyze data at scale
✅ Curate and focus on the most important dataset segments
✅ Review and refine annotations
✅ Measure and enhance model performance
```
Nucleus brings your data and predictions together, empowering you to solve data quality issues, fix failure modes, and create better models faster.
## Gen AI Platform
The Scale GenAI Platform enables enterprises to quickly develop, test, and deploy Generative AI applications tailored to custom use cases using proprietary data. With tools for connecting data, fine-tuning models, and secure deployment, it provides a flexible and scalable solution for enterprise needs, allowing you to:
```
✅ Implement optimized RAG pipelines and fine-tuned models with the Scale Data Engine.
✅ Deploy open and closed-source models, including OpenAI, Cohere, and Meta.
✅ Securely launch AI applications in your VPC with support for AWS and Azure.
✅ Fine-tune models to enhance performance, reduce latency, and optimize token usage.
✅ Test, evaluate, and monitor AI systems with advanced metrics and tools to ensure quality and reliability.
```
# Pro | Manage Account
Source: https://api-reference.scale.com/docs/pro-or-manage-account
Manage your Account Overview Customers can add new users, change user roles, and manage their team defaults by clicking the Your Team link found on the top right corner of the dashboard. Adding New Users To have someone
# Manage your Account
## **Overview**
Customers can add new users, change user roles, and manage their team defaults by clicking the **Your Team** link found on the top right corner of the dashboard.
## **Adding New Users**
To have someone join your team, scroll down to the **Invite Member** button at the bottom of the page. Clicking this button will allow you to specify the email address of the user would like to invite.
Once this happens, Scale will send that user an invitation to join your team. You will see this person listed as **Invited** once you refresh the page:
To re-send an invitation to join your team, you may click **Remove**, and then re-invite them with the **Invite Member** button.
When a new user accepts an invitation to join your team, they will be asked for their name, company, and other details for their profile.
Good to Know!
Users **must** accept your invite to join your team. There is not a way to have a user join a different team once signed up.
If the user has already made a new account as opposed to clicking your invite, please email **[support@scale.com](mailto:support@scale.com)** and we will be able to move the user to your team on your behalf.
**Users can not be added to more than one team.** If a user needs access to multiple Scale organizations, a best practice is to add a `**+**` with the second team name to their email address.
As an example, you could have `**[joe.schmoe@scale.com](mailto:joe.schmoe@scale.com)**` be in the Scale account and `**[joe.schmoe+second@scale.com](mailto:joe.schmoe+second@scale.com)**` be in the second account. Most email providers ignore anything after the + so you could still receive emails as expected.
## **User Roles**
Users can be in 1 of 3 roles. These roles are:
**Member** - Members can see and audit all tasks submitted, and create and edit projects.
**Manager** - In addition to all things a Member can do, Managers can manage team members, retrieve API keys to submit new tasks, and view billing information.
**Admin** - A team can only have 1 admin. Admins manage every aspect of a Scale account. On the back-end, all projects, team members, and API keys roll up to this admin user. If you need to change the admin user for any reason, please reach out to **[support@scale.com](mailto:support@scale.com)** and we will be happy to assist.
## **Team Details**
In addition to managing your team, you can also change your **team name** and **callback URL** under the team details header.
* Your **team name** will be used as the prefix as your project names
* Your **callback URL** is the default callback URL if the callback\_url field is not provided. This is the URL where we will POST the results of your annotations
## **API Keys**
To retrieve your API keys, please navigate to \*\*API Keys \*\*link found by clicking on the user profile icon in the top right of the page.
Within this page, you can also **Add a new key** or **Delete your existing one.**
Scale offers live and test API keys.
* The \*\*live key \*\*is used to send tasks to our labeling team for completion.
* The **test key** will not be sent to our labeling team. The purpose of this functionality is to test out our API on our website
If you’d like to authenticate our callbacks, we set a scale-callback-auth HTTP header on each of our callbacks. The value will be equal to your **Live Callback Auth Key** shown on your dashboard. If this header is not set, or it is set incorrectly, the callback is not from Scale.
## **Billing**
For our enterprise customers, your account is billed via an invoice. Please contact your Scale account representative for more details on billing.
For our on-demand customers, we display the number of completed tasks, accrued balance, and last invoices in our billing tab. Your payment information can be updated in the **teams** button by clicking on the user profile on the top right of your page.
## **Integrations**
Within the customer dashboard, you are able to set up cloud storage integrations with AWS, Google Cloud Platform, and Azure.
For detailed instructions, please see the **[secure attachment access guide](/docs/upload-your-data)**.
# Pro | Manage projects
Source: https://api-reference.scale.com/docs/pro-or-manage-projects
Manage your Projects Overview Projects allow you to organize similar tasks based on instructions and task types. All projects will share the same instructions and annotation rules. Create New Projects To create a new pro
# Manage your Projects
## Overview
**Projects** allow you to organize similar tasks based on instructions and task types. All projects will share the same instructions and annotation rules.
## Create New Projects
To create a new project, you can click on the Project list dropdown positioned on the left side of the page and select “Create New Project Group“
Important things to know about projects
* **Projects cannot be deleted**
* \*\*A project is tied to one specific Task Type, such as “2D Box Annotation”. \*\*You cannot change a project’s task type after you have created the project.
* **All tasks in a project should have the same instructions.**
* \*\*Please do change instructions significantly within a project. \*\*If instructions need to be changed significantly, please create a new project.
* Examples of significant changes:
* Adding/removing labels
* Changing the intent of a project entirely
## Manage Your Projects
To manage all of the projects, click on the three **ellipsis** on the left hand navigation or on the **Manage** button on the right hand bar.
Within the "Manage Projects" view, you are able to edit:
1. Project name
2. Instructions
3. Pin projects to the top
In addition, you can sort projects by name, task type, and creation date!
# Pro | Overview Tab
Source: https://api-reference.scale.com/docs/pro-or-overview-tab
Overview Tab Overview The Overview tab allows you to view a summary of your submissions over time and the batch completion status Task Visualization After choosing a project, you can visualize the number of tasks over a
# Overview Tab
## **Overview**
The **Overview** tab allows you to view a summary of your submissions over time and the batch completion status
## **Task Visualization**
After choosing a project, you can visualize the number of tasks over a specific **task completion** or **task creation** date range.
Batches are a way for you to organize tasks within a project. You can assign custom names for your batches. Our customers use batches to organize their tasks based on location, date, or any other types of metadata.
The overview tab shows you all of your batches and the completion progress bar over a specific task creation or task completion time period.
# Pro | Quality
Source: https://api-reference.scale.com/docs/pro-or-quality
Quality Tab Overview The Quality tab allows you to audit a random sample of tasks for quality and view stats for tasks you’ve audited Auditing Workflow There are three ways to audit tasks: 1. Audit completed tasks indivi
# Quality Tab
## **Overview**
The **Quality** tab allows you to audit a random sample of tasks for quality and view stats for tasks you’ve audited
Why should you audit tasks?
Auditing tasks allow you to provide feedback directly to our taskers. This will help us identify common errors or clarify edge cases
## **Auditing Workflow**
There are three ways to audit tasks:
1. Audit completed tasks individually
2. Audit all completed tasks within a date range
3. Audit all tasks completed across a specific week
**How to audit completed tasks individually:**
1. In the **Tasks** tab, filter on the **Completed** tasks
2. Search for the task that you want to audit and click on "**Audit Task**"
**How to audit all completed tasks within a date range:**
1. In the **Quality** tab, select the specified task date range, audit or batch status
2. Click on the "**Audit Tasks**" button - this will allow you to audit a random sample of tasks
**Audit all tasks completed across a week:**
1. In the **Quality** tab go to the "Tasks completed last week" tab under "Report"
2. Select **Audit** for the desired week
Auditing Tasks
If you have multiple people on your team auditing at the same time, we will “lock” each task so that only one person can audit one task at a time. This was designed to prevent conflicted copies.
## **Audit Feedback**
Within the audit workflow, you are able to make the following selections:
* **Approve**: Mark that tasks that have been completed according to instructions
* **Make Changes**: Fix tasks that have minor or non-critical errors that you can use to update your callback response immediately
* **Reject**: Signal that tasks have completed incorrectly. Please provide detailed comments for how our team can fix these mistakes moving forward
Reject vs. Make Changes
* Both **Reject** and **Make** **Changes** will provide feedback to the Scale team about the quality of your tasks.
* For quick or minor changes, please **Make Changes** as you will get an updated callback response right away. In addition, not all rejected tasks will be resubmitted. Please contact your engagement manager if you want to discuss re-dos
* For critical and more structural misunderstanding of the instructions, please **Reject** the task and provide detailed feedback! This will help our taskers to improve.
## **Audit Reports**
* The quality assurance dashboard will display the total number of reviews, accepts, and rejects using the set of filtered tasks. In addition, we also provide the following reports:
* **Tasks completed last week** - lets you audit tasks that have been aggregated by completed date from Monday 00:00 to Sunday 23:59 PST for the last 4 weeks
* \*\*Task Level Report \*\*- provides you with a list of tasks with auditor and completion time
* **Auditor Tracker** - shows an audit report for tasks in batches
* Note: this is only available if you are use batches. To learn more about batches, please see this article
* **Auditor Quality** - displays a summary of reviews, accepts, and rejects per auditor
## **Callback Body**
* When an audit is submitted, several new field will be added to the callback body:
* `customer_review_status`: Reflects the result of the audit as “Accepted”, “Rejected” or “Fixed”
* `customer_review_comments`: Shows all comment left during the audit
* `prior_responses`: Response before the audit
* If a task is fixed through multiple audits:
* `response`: will be updated with the most recent audit
* `customer_review_status`: will reflect the most recent audit
* `customer_review_comments`: will append the comment to the array
* `prior_responses`: will append previous responses to the array
# Pro | Tasks Tab
Source: https://api-reference.scale.com/docs/pro-or-tasks-tab
Tasks Tab Overview The Tasks tab allows you to view a more granular list of your submitted tasks, filterable by status, project, and batch. Task Filters You can search through your tasks using the following filters: 1. C
# Tasks Tab
## **Overview**
The **Tasks** tab allows you to view a more granular list of your submitted tasks, filterable by status, project, and batch.
It is possible your task filters will hide all tasks. Please check your task filters carefully!
## **Task Filters**
You can search through your tasks using the following filters:
**1. Creation Date range** : Select the task completion or creation date range
* Note: You are able to select **UTC Time**, **Pacific Time,** or **Local Time**
**2. Task Status**
* You can also filter based on the task status. Here are the following status options:
* \*\*Completed - \*\*Tasks that have been completed by our taskers
* \*\*In Progress - \*\*Tasks that are being worked on by our taskers
* \*\*Queued - \*\*Tasks that have not been worked on yet
* \*\*Error - \*\*Tasks that have errors.
* \*\*Canceled - \*\*Tasks that have been cancelled by our customers
* \*\*Redo - \*\*Tasks that have been re-done by Scale
**3. Audit Status**
* Filter based on the customer audit status:
* \*\*Read to Audit \*\*(Tasks that have not been audited yet)
* **Accepted**
* **Fixed**
* **Rejected**
In Progress vs. Queued
You can only cancel tasks that are **Queued**.
Once the status of a task has moved into **In Progress** that means the task is already being worked on and can no longer be canceled.
For further information please reach out to **[support@scale.com](mailto:support@scale.com)**
## **Exporting JSON**
There are two ways for exporting JSON **1. Export JSON for all Tasks**
* Click on the **Export** button at the top of the **Tasks** page
Please note that there is a 1,000 task limit. If you want to download more, please use our API instead.
**2. Export JSON for each Task Individually**
* Click on each task and then click on the **Download JSON** button. You can also see a preview of the JSON directly from this view.
## **Task Details**
**1. Task Preview**
* To preview the media asset file, please click into task and hover over the image tile. Please note that you can only preview the original image for pending tasks and annotated images for completed tasks.
* For 3D tasks, you can preview the point cloud file by clicking on “Debug Task” or “Audit Task” (this is for completed tasks only)
**2. Documentation & Instructions**
* To view the documentation and instructions for the task, please click into the task and navigate to the “Instructions” or “documentation“ button
**3. JSON**
* To view the JSON file for each task individually, please click into the task and navigate to the “Download JSON" button. Please note that this will only return the JSON file for the original task upload
* You can also export all JSON for all tasks using the "Export JSON" button
**Note**: this will only export 1000 tasks. If you want to export more, you must use the **[task API endpoint](https://docs.scale.com/reference#list-multiple-tasks)**.
## **Audit Tasks**
To audit tasks, click on an individual task, and then click the "Audit tasks" button. In this view you can:
* **Accept** - This will signal that we have completed the task to your quality standards
* **Fix** - If you want to make an immediately update to your callback response, you can fix tasks directly
* previous responses will be saved in the `prior_reponse` field of the callback body and the `response` field will be updated with the updated response.
* **Reject** - This will signal that we have not met your quality standards.
To learn more about this workflow, please see our best practices **[here](/docs/pro-or-quality)**.
## **Cancel Tasks**
You can only cancel tasks via the API - you can learn more about this endpoint **[here](https://docs.scale.com/reference#cancel-task)**.
You can only cancel tasks that are **Queued**.
Please do not cancel tasks that are **In Progress.** These tasks are already being worked on. Cancelling these tasks may result in a charge to compensate for the existing work.
Note: **Completed** tasks cannot be cancelled.
# Team - Getting Started
Source: https://api-reference.scale.com/docs/team-getting-started
Team When you create an account on Scale, you will be the Admin on your account. You can then invite others to join your team as Managers, Members, or Labelers. Any projects created by your team will share the same payme
# Team
When you create an account on Scale, you will be the Admin on your account. You can then invite others to join your team as Managers, Members, or Labelers. Any projects created by your team will share the same payment information. The table below lists actions that can be performed by the various roles:
| | Admin | Manager | Member |
| ------------------------------------------------------------- | ----- | ------- | ------ |
| Manage and update billing information | ✓ | | |
| Invite others to the team | ✓ | ✓ | |
| Upload and archive data | ✓ | ✓ | ✓ |
| Create, rename and archive projects and batches | ✓ | ✓ | ✓ |
| Create and edit project taxonomy, settings, and quality tasks | ✓ | ✓ | ✓ |
| View project metrics | ✓ | ✓ | ✓ |
| Audit tasks | ✓ | ✓ | ✓ |
| Create, edit, and delete labels on a task | ✓ | ✓ | ✓ |
| Export labeled tasks | ✓ | ✓ | ✓ |
# Technical Limits & Recommendations
Source: https://api-reference.scale.com/docs/technical-limits-and-recommendations
Technical Limits & Recommendations Technical Limits * Please contact support if you would like the limits to be increased. Web Browser Compatibility The Scale experience is currently optimized for Google Chrome. Other br
# Technical Limits & Recommendations
## **Technical Limits**
| | Pro |
| ----------------------------------------- | ------------------ |
| Task Creation (req/s) | 30 requests/second |
| Max Attributes | 2500 |
| Max Annotations | Unlimited |
| Max Instances (for semantic segmentation) | Unlimited |
| Max Task Metadata JSON size | 10 kb\* |
| Max File Upload Metadata JSON Size | 8 kb |
| Max File Upload Attachment Size | 120 mb\* |
| Max Tasks per Batch | Unlimited |
* Please contact support if you would like the limits to be increased.
## **Web Browser Compatibility**
The Scale experience is currently optimized for Google Chrome. Other browsers (Firefox, Safari, Microsoft Edge, etc) have not been fully tested, and cannot be recommended for use.
While Scale may work with other web browsers, we cannot guarantee full compatibility at this time. Our support team may not be able to help resolve any bugs or issues you encounter when using other browsers.
Tip: If you are running into issues when using Chrome, be sure your browser is updated to the latest version.
| Web Browser | Support Policy |
| ----------- | --------------- |
| Chrome | Fully supported |
| Edge | Best effort |
| Safari | Best effort |
| Firefox | Best effort |