How AutoCI Works

AutoCI analyzes CI failures by intelligently interfacing with various CI systems through GitHub’s APIs. This page explains the technical details of how AutoCI detects and analyzes failures across different CI platforms.

CI Detection Strategy

AutoCI uses a cascading detection strategy to find CI runs, checking multiple GitHub APIs in sequence to ensure maximum compatibility:

1. GitHub Actions (Primary)

For repositories using GitHub Actions, AutoCI has the deepest integration:

API Used: GitHub Actions API (/repos/{owner}/{repo}/actions/runs)

What we can access:

Workflow run status and conclusions
Individual job failures within workflows
Step-by-step execution logs
Exact line numbers where failures occur
Full log content for detailed analysis

How it works:

1. Fetch all workflow runs for the PR's head SHA
2. Identify failed runs
3. For each failed run, fetch all jobs
4. For failed jobs, fetch step-by-step logs
5. Extract error messages and surrounding context
6. Send to AutoCI's analysis engine

This deep integration allows AutoCI to provide precise error analysis, including:

Exact error messages
Stack traces
Compilation errors
Test failure details
Environment-specific issues

2. Commit Status API (Prow, Jenkins, CircleCI legacy)

For CI systems that report via GitHub’s Commit Status API (like Kubernetes’ Prow):

API Used: GitHub Status API (/repos/{owner}/{repo}/commits/{sha}/statuses)

What we can access:

CI job names (contexts)
Pass/fail status
Brief descriptions
Links to external logs

How it works:

1. Fetch all commit statuses for the PR's head SHA
2. Identify failed statuses (state: "failure" or "error")
3. Extract available information:
   - Job name (e.g., "pull-kubernetes-verify")
   - Status description
   - Target URL for full logs
4. Send available data to AutoCI's analysis engine

Limitations:

Cannot access detailed logs directly (they’re hosted externally)
Analysis based on job names and descriptions
Users can visit provided URLs for full logs

Supported Systems:

Prow (Kubernetes, Istio, Knative)
Jenkins (when using GitHub integration)
CircleCI 1.0
Travis CI (older integrations)
Drone CI
TeamCity

3. GitHub Checks API (Modern CI Systems)

For newer CI systems that use GitHub’s Checks API:

API Used: GitHub Checks API (/repos/{owner}/{repo}/commits/{sha}/check-runs)

What we can access:

Check run names and status
Detailed output text and summaries
Annotations (inline code comments)
External URLs for full details

How it works:

1. Fetch all check runs for the PR's head SHA
2. Identify failed checks
3. Extract output text and summaries
4. Parse annotations for specific error locations
5. Send to AutoCI's analysis engine

Supported Systems:

CircleCI 2.0+
BuildKite
Azure Pipelines
Codecov
SonarCloud
Custom GitHub Apps

Detection Flow

Here’s the complete detection flow AutoCI follows:

┌─────────────────────┐
│ PR Analysis Request │
└──────────┬──────────┘
           ▼
    ┌──────────────────────┐
    │ GitHub Actions       │
    │ workflows exist?     │
    └──────┬───────┬───────┘
           │ Yes   │ No
           ▼       ▼
    ┌──────────┐  ┌──────────────────┐
    │ Use      │  │ Commit statuses  │
    │ Actions  │  │ exist?           │
    │ API      │  └──┬────────┬──────┘
    │          │     │ Yes    │ No
    │ Deep log │     ▼        ▼
    │ analysis │  ┌────────┐ ┌──────────────┐
    └─────┬────┘  │ Use    │ │ Check runs   │
          │       │ Status │ │ exist?       │
          │       │ API    │ └──┬─────┬──────┘
          │       └───┬────┘    │ Yes │ No
          │           │         ▼     ▼
          │           │    ┌────────┐ ┌──────────┐
          │           │    │ Use    │ │ No CI    │
          │           │    │ Checks │ │ detected │
          │           │    │ API    │ └──────────┘
          │           │    └───┬────┘
          │           │        │
          └───────────┴────────┴──────┐
                                      ▼
                          ┌───────────────────┐
                          │ AutoCI Analysis   │
                          │ Engine            │
                          └─────────┬─────────┘
                                    ▼
                          ┌───────────────────┐
                          │ Root Cause Report │
                          └───────────────────┘

API Rate Limits and Performance

AutoCI is designed to be efficient with API calls:

Caching: Results are cached for 5 minutes to avoid duplicate API calls
Parallel fetching: Multiple API calls are made concurrently where possible
Smart filtering: Only failed jobs are analyzed in detail
Progressive loading: Logs are fetched only for failed steps

Data Privacy and Security

What AutoCI accesses:

Public repository data (for public repos)
Repository data you have access to (for private repos)
CI logs and status information
No source code is analyzed unless it appears in CI logs

What AutoCI stores:

Anonymized failure patterns for improving analysis
No raw logs or sensitive data is permanently stored
Analysis results are retained for 30 days

Practical Examples

Example 1: GitHub Actions Failure

When analyzing a GitHub Actions failure:

# For a pull request
autoci failure-cause https://github.com/owner/repo/pull/123
 
# For a branch commit
autoci failure-cause --repo owner/repo --branch main

AutoCI will:

Fetch workflow runs via Actions API
Download detailed logs for failed jobs
Extract specific error messages and stack traces
Provide contextualized analysis with line numbers

Example 2: Kubernetes Prow Failure

When analyzing a Kubernetes PR with Prow:

autoci failure-cause https://github.com/kubernetes/kubernetes/pull/133444

AutoCI will:

Detect no GitHub Actions workflows
Fetch commit statuses via Status API
Find Prow jobs like pull-kubernetes-verify
Provide analysis based on job names and status descriptions
Include links to Prow’s external log viewer

Example 3: CircleCI via Checks API

For modern CircleCI integrations:

autoci failure-cause https://github.com/owner/repo/pull/456

AutoCI will:

Detect no GitHub Actions workflows
Check commit statuses (may be empty for newer setups)
Fetch check runs via Checks API
Parse CircleCI’s output text and annotations
Provide analysis of test failures and build errors

Limitations by CI System

CI System	Log Access	Error Detail	Analysis Quality
GitHub Actions	Full	Excellent	High - Line-level precision
Prow	External only	Limited	Medium - Based on job names
Jenkins	External only	Limited	Medium - Based on status
CircleCI 2.0+	Partial	Good	High - Output text available
BuildKite	Partial	Good	High - Check output available
Travis CI	External only	Limited	Medium - Status only
Azure Pipelines	Partial	Good	High - Annotations available

Future Enhancements

We’re working on:

Direct integration with Prow’s API for log access
Jenkins API integration for deeper analysis
Support for self-hosted CI systems
Custom webhook integrations for real-time analysis

Troubleshooting

“No CI runs found”

Ensure the PR has CI configured
Check if CI has started running (may be queued)
Verify your GitHub token has appropriate permissions

“Limited analysis available”

This appears for non-GitHub Actions CI
External logs are linked when available
Consider the CI system’s integration limitations

Need Help?

If your CI system isn’t properly detected, please open an issue with details about your setup.

Getting Started Overview