Skip to main content

Documentation Index

Fetch the complete documentation index at: https://api-reference.scale.com/llms.txt

Use this file to discover all available pages before exploring further.

​
Writing Instructions: Best Practices

Below are some best practices to consider as you write instructions:
  1. Dive into your data: Before writing instructions, it’s important to review a representative sample of your data to understand what global and edge case rules need to be written. As you’re reviewing the data, put yourself in the labeler’s shoes; document areas you’d need guidance on, as well as any tricky cases that need to be addressed in the instructions.
  2. Writing instructions:
  • Structure: Following a clear structure in your instructions will help labelers better understand the rules, leading to higher quality annotations. The below structure is a good starting point:
  • Summary of Task: Provide a few pithy bullets describing the basics of your task.
  • Workflow: Provide the steps labelers must take to annotate your task from start to finish.
  • Annotation Rules: Start with the global rules (i.e. rules that apply to all tasks, then zoom in to more specific rules). Areas you’ll likely want to touch on:
    • Minimum pixel size
    • What to label and what NOT to label
    • How to manage occlusion, truncation, and low visibility cases
    • Geometry sizing if relevant
    • How to manage cases you’re not sure about (i.e. “err on the side of selecting XXX”)
    • Etc.
  • Label and Attribute Definitions and Examples
  • Common Errors
  • Edge Cases
  1. Communication:
  • Communicate succinctly and clearly (ie pithy bullets, use spaces to decrease cognitive load for the labeler)
  • Include at least 1 image for each point you’re looking to make. Make sure each image has a description of WHY it is correctly or incorrectly annotated.
  1. Pressure test your instructions: After writing the instructions, review your data again and make sure your instructions cover the majority (90+% of cases you see). Continue to iterate on instructions until you have hit the 90+% number.
  2. Final tips:
  • It can be helpful to include a short video walking through the instructions
  • If there are certain tools / features that you expect labelers to use when annotating your data, we’d suggest mentioning them in the instructions and providing a description of how to use the tool (e.g. interpolation)