Experiments

Introduction to Experiments

Experiments are the core feature for A/B testing and product experimentation within the product. They allow teams to test hypotheses, measure results, and make data-driven decisions about product changes. The experiment system provides a complete lifecycle from ideation to conclusion, with built-in collaboration, task management, and analytics integration.

Understanding the Experiment Framework

What is an Experiment?

An experiment represents a hypothesis you want to test. It tracks:

• Hypothesis - What you believe will happen and why
• Growth Area - Which part of the funnel this targets (acquisition, activation, retention, monetization)
• Status - Current stage in the experiment lifecycle
• Results - Measured outcomes including lift, significance, and conclusions

Growth Areas

Experiments are categorized by which part of the growth funnel they target:

Creating Experiments

To create a new experiment:

1. Navigate to the Experiments section from the sidebar
2. Click New Experiment
3. Fill in the experiment details:
   • Title - A clear, descriptive name for the experiment
   • Hypothesis - Your prediction of what will happen and why
   • Growth Area - Select the relevant funnel area
   • Owner - Assign a team member responsible for the experiment
   • Priority - Set urgency level (Low, Medium, High, Urgent)
   • Tags - Add labels for categorization and filtering
   • Start Date - When the experiment will begin
   • End Date - Expected completion date

Setting Priority

Experiments support multiple priority levels to help teams focus on what matters most:

Customizing Appearance

Each experiment can have custom visual settings:

• Icon - Choose an icon to represent the experiment
• Color - Set a color for easy visual identification in views

Experiment Lifecycle

Experiments progress through a defined lifecycle with seven distinct statuses:

Status Flow

IDEA → DESIGN → LIVE → ANALYSIS → WINNER/LOSER/INCONCLUSIVE

Lifecycle Tracking

The system automatically tracks when an experiment enters each status, creating a complete timeline of the experiment's journey. This data is valuable for:

• Understanding how long experiments spend in each phase
• Identifying bottlenecks in your experimentation process
• Reporting on team velocity and throughput

Managing Experiments

Kanban Board View

The Kanban view provides a visual board organized by status:

• Drag and drop experiments between status columns
• Quick edit experiment details directly from cards
• Filter by owner, area, priority, or tags
• Search across all experiment titles and content

List View

The list view shows experiments in a table format with:

• Sortable columns for all properties
• Bulk selection for batch operations
• Quick inline editing

Calendar View

View experiments on a calendar to:

• See experiment timelines at a glance
• Identify scheduling conflicts
• Plan around team capacity

Gantt View

The Gantt view provides timeline visualization:

• Drag to reschedule - Move experiments to new dates
• Resize to extend/shorten - Adjust experiment duration
• Dependencies - See relationships between experiments
• Auto-scheduling - Move related experiments together

Collaboration Features

Tasks

Break down experiments into actionable tasks:

1. Open an experiment
2. Navigate to the Tasks tab
3. Click Add Task
4. Configure the task:
   • Title - What needs to be done
   • Description - Additional details
   • Assignee - Who is responsible
   • Due Date - When it should be completed
   • Start Date - When work should begin

Tasks support:

• Completion tracking with checkboxes
• Assignment notifications
• Due date reminders

Comments

Collaborate with team members through comments:

• Rich text formatting - Format comments with markdown
• @Mentions - Tag team members for notifications
• Reactions - React to comments with emoji
• Threaded discussions - Keep conversations organized

Resources

Link external resources to experiments:

• Documentation links
• Design files
• Analytics dashboards
• Related documents
• External tools and references

Click Add Resource and provide a title and URL to attach resources.

Recording Results

When an experiment concludes, record the outcome:

Concluding an Experiment

1. Move the experiment to Winner, Loser, or Inconclusive status
2. Fill in the results form:
   • Outcome - Winner, Loser, Inconclusive, or Deploy Failure
   • Primary Metric - The main metric measured
   • Initial Value - Baseline measurement
   • Final Value - End measurement
   • Lift Percentage - The change observed
   • Statistical Significance - Confidence level
   • Conclusion Notes - Summary of learnings

Outcome Types

Sharing Results

Share experiment results publicly:

1. Open the concluded experiment
2. Click Share Results
3. Toggle Public Sharing on
4. Copy the generated shareable link

Shared results include:

• Experiment title and hypothesis
• Final outcome and metrics
• Conclusion notes

Templates

Experiment templates save time and ensure consistency by pre-configuring experiment settings, tasks, and checklists. Templates are organization-scoped and can only be created by Admins or Owners.

Understanding Templates

Templates define default values that are applied when creating new experiments. This includes:

• Basic properties - Title format, hypothesis structure, growth area, priority
• Scheduling - Start date logic and experiment duration
• Tasks - Pre-populated task lists with default assignees
• Checklists - Pre-flight verification items to ensure quality
• Visual settings - Icon and color for quick identification

Creating Templates

To create a new experiment template:

1. Navigate to Settings → Experiment Templates
2. Click New Template
3. Configure the template settings:

Basic Information

Default Experiment Values

Scheduling Settings

Experiment Configuration

Template Tasks

Pre-populate experiments with tasks that need to be completed:

1. In the template editor, navigate to the Tasks section
2. Click Add Task
3. Configure each task:
   • Title - Task description
   • Description - Additional details or instructions
   • Default Assignee - Team member automatically assigned
   • Order - Position in the task list

Example tasks for an A/B test template:

• "Define success metrics" (assigned to Product Manager)
• "Implement tracking code" (assigned to Engineering)
• "Set up experiment in analytics tool" (assigned to Data Analyst)
• "Review test configuration" (assigned to QA)

Template Checklists

Checklists ensure experiments meet quality standards before launch:

1. In the template editor, navigate to the Checklist section
2. Click Add Checklist Item
3. Configure each item:
   • Label - What needs to be verified
   • Check Type - Category of the check
   • Required - Whether this must be completed before launch
   • Order - Position in the checklist

Example checklist items:

• ✓ "Hypothesis is clearly defined" (Required)
• ✓ "Sample size is statistically valid" (Required)
• ✓ "Tracking is implemented and tested" (Required)
• ✓ "Stakeholders have been notified" (Optional)

Using Templates

To create an experiment from a template:

1. Click New Experiment from the experiments page
2. Select From Template in the creation dialog
3. Browse templates by category or search by name
4. Click on a template to preview its settings
5. Click Use Template to create the experiment
6. Review and customize the pre-filled values as needed
7. Click Create to finalize

The new experiment will have:

• All default values from the template applied
• Tasks pre-created with their default assignees
• Checklist items ready for verification
• Template icon and color applied

Template Categories

Organize templates into categories for easier discovery:

Template Usage Analytics

Track how templates are being used:

• Usage Count - Number of experiments created from the template
• Last Used - When the template was last used
• Created By - Who created the template
• Created At - When the template was created

This data helps identify:

• Most popular templates that should be maintained
• Unused templates that could be archived
• Patterns in team experimentation practices

Best Practices for Templates

Template Design

• Be specific but flexible - Set defaults that work 80% of the time, but allow customization
• Include clear instructions - Use the hypothesis field to guide proper hypothesis writing
• Pre-assign owners wisely - Default owners should be roles, not specific people when possible
• Keep task lists focused - Include essential tasks, not every possible task

Template Management

• Review templates quarterly - Update based on learnings and process changes
• Archive unused templates - Keep the template list clean and relevant
• Document template purposes - Use descriptions to explain when each template applies
• Standardize categories - Use consistent category names across the organization

Template Governance

• Limit template creators - Only Admins and Owners can create templates to ensure quality
• Version control changes - Document significant template updates
• Gather feedback - Ask team members which templates are most helpful

Linking to OKRs

Connect experiments to Key Results to show how experimentation drives goals:

1. Open an experiment
2. Click Link Key Result
3. Select relevant key results
4. Track impact on organizational objectives

This creates visibility into how experiments contribute to company goals.

Activity Logging

Every change to an experiment is tracked:

• Status changes
• Priority updates
• Owner reassignments
• Area modifications
• Hypothesis edits
• Date changes
• Tag additions/removals

View the complete history in the Activity tab to understand the experiment's evolution.

Workflow Automation

Experiments integrate with the workflow system to automate actions:

• Trigger workflows when experiment status changes
• Send notifications when experiments move to specific statuses
• Update external systems (Slack, Jira, etc.) automatically
• Generate reports when experiments conclude

See the Workflows documentation for details on setting up automations.

Exporting Data

Export experiment data for reporting:

CSV Export

Export a filtered list of experiments to CSV format including:

• All experiment properties
• Status history
• Result data

Excel Export

Export to Excel with additional formatting:

• Multiple worksheets for different views
• Formatted tables
• Summary statistics

Best Practices

Writing Good Hypotheses

• Use the format: "We believe [change] will result in [outcome] because [rationale]"
• Be specific about what you're testing
• Include measurable predictions
• State your assumptions clearly

Setting Up Experiments

• Define success metrics before starting
• Set appropriate sample sizes
• Plan for statistical significance
• Document your test design

Running Experiments

• Don't peek at results too early
• Maintain consistent conditions
• Document any anomalies
• Avoid making changes mid-experiment

Concluding Experiments

• Wait for statistical significance
• Document all learnings, not just outcomes
• Share results with stakeholders
• Plan follow-up experiments based on learnings

Team Collaboration

• Assign clear ownership
• Use tasks to break down work
• Keep comments and discussions in the product
• Link related resources for context