Skip to main content

Documentation Index

Fetch the complete documentation index at: https://api-reference.scale.com/llms.txt

Use this file to discover all available pages before exploring further.

​
Calibration Workflow

Before you can send off loads of data to the labelers, you’re required to calibrate your project by running a Calibration Batch. This will allow taskers to test your project, and they’ll give you feedback on the instructions and attempt to label a small subset of your data. Calibration Batches serve the purpose of making sure
  1. You have a properly functioning Project
  2. You have completed tasks that can be turned into Training tasks for the labelers
  3. You have completed tasks that can be turned in Evaluation tasks to track labeler performance
Note that this is an iterative process. You may have to create more than one Calibration Batch to properly fine-tune your instructions and create the necessary number of Quality Tasks.

​
Creating a Calibration batch

You can create a Calibration Batch from the Batches page of your project. 1288 Calibration batches are required before you can unlock a regular production batch You’ll select a subset of your data for the Calibration Batch. 2000 Select data for Calibration Batch. Ideally you’ll select around 20 items. Once you’ve created your Calibration Batch, it should be completed within a few hours.

​
Auditing a Calibration batch

The second part of calibrating your project is to view the results. It is important to take the time to review instruction feedback as well as the actual labelling attempt. Instruction feedback
Calibration Batches provide the inputs you’ll need to refine your instructions. The higher the clarity of your instructions, the better each Tasker will be able to perform on your project. You can view this feedback in the Calibration Batch view under Tasks. 1996 Any available feedback can also be seen in the Instruction Editor, so you can view the feedback as you refine your project’s instructions. 1978

​
Auditing the batch

At the top of the Calibration Batch view you should be able to open the Audit Tool to go through the tasks and grade the attempt. 1000 _Click _Open Audit Tool to launch the audit 2000 Go through each item of the Calibration to fully complete the Calibration Batch Once you’ve finished your audit, you’ll have a returned Calibration Score that is represents how accurately the taskers labeled. 1014 This score is only available after you complete the audit. It takes into account a confidence score reported by the Taskers as well as how accuracy their labels actually were. You might need to run more than one Calibration Batches until the calibration scores are high. Calibration Batches help to resolve initial confusions about the instructions and help you iterate. Note, for tasks that require a higher level of english understanding or other language requirements, you can use advanced controls to specify 1472 Advanced controls