PR analysis methodology

Written to be read by a machine as much as by a person. The same text is served as markdown.

# PR Analysis methodology

Analyse the health of a software project using the pull request data in the supplied PR Analysis report.

Treat the data as a set of signals describing how effectively work flows through the development team. Produce a clear project health report suitable for a technical lead, CTO, engineering manager, development manager or company director.

Use the most recent 12 complete months as the main comparison period.

Do not draw strong conclusions from an incomplete month. It may be mentioned as an early indication, but say clearly that it is a part month.

## About the data

Every report has a page and a machine-readable twin at the same address with `.json` on the end. Prefer the JSON. Reading figures off rendered charts is guesswork, and the JSON carries every series the charts are drawn from.

The document is shaped like this:

* `snapshotAt` is the moment the data was exported. Everything is measured against that moment, not against today, so a report opened months later still describes the same picture.
* `months` is one object per month, oldest first, with every series the dashboard plots: `opened`, `merged`, `closed`, `netOpenChange`, `openCount` (open on the 1st), `openMoreThan7`, `openMoreThan30`, `openMoreThan90`, `medianDaysToMerge`, `meanDaysToMerge`, `p90DaysToMerge`, `mergeRate`, `meanBacklogAgeDays`, `medianBacklogAgeDays`, the merged size buckets (`mergedSizeSmall` through `mergedSizeExtraExtraLarge`), the time to merge and time to close bands, the percentage variants of each, and the merged line counts.
* `months[].complete` is false for the month the snapshot fell in. That month was still running when the data was taken, so its counts are partial. Leave it out of trend conclusions.
* `backlogAtSnapshot` describes the open backlog at the snapshot moment: `openCount`, `medianAgeDays`, how many are older than 7, 30 and 90 days, the age bands, and the size buckets.
* `hasLineData` is false when the import carried no line counts. When it is false, every size and line figure is a zero rather than a measurement, so say the size analysis is unavailable rather than reporting zeros.
* `statCards` are the six headline figures the page shows.
* `additionalNotes` holds anything the person who ran the report chose to say about the project. It is context, not evidence.

Size buckets are by total lines changed: S is under 50, M is under 200, L is under 500, XL is under 1,000, XXL is 1,000 or more.

## 1. Overall project health

Give the project an overall health score out of 10, and one of these classifications:

* Very healthy
* Healthy
* Needs attention
* Unhealthy

Open the report with a short executive summary stating the most important conclusion.

Never calculate health from a single metric. Evaluate the metrics together.

## 2. Throughput

Analyse pull requests opened, merged, and closed without merging.

Look at whether the team resolves work at roughly the rate new work arrives. Calculate the net backlog change over the most recent 12 complete months:

    net backlog change = opened - merged - closed

Say whether throughput looks stable, improving, deteriorating or unusually variable.

Do not assume a project with more pull requests is healthier than one with fewer. Volume and health are different things.

## 3. Backlog health

Analyse the current number of open pull requests, and where the data allows it: median backlog age, mean backlog age, and how many are older than 30, 60 and 90 days.

Decide whether old pull requests are isolated exceptions, a small long tail, a significant share of the backlog, or evidence of a systemic flow problem.

Analyse whether the backlog is growing, shrinking or broadly stable over time. A growing backlog matters much more when backlog age is rising with it.

## 4. Time to merge

Analyse median merge time, mean merge time, and the share merged within one day where it is available. Look at the trend, not just the latest value.

Say whether review and merge performance looks fast, slow, stable, inconsistent, improving or deteriorating.

Do not call a project healthy purely because its median merge time is low. A project can have very fast pull requests that get attention alongside a large population of stale ones. Identify that pattern explicitly when you see it.

## 5. Pull request size and complexity

Where size data is available, analyse typical size, median and mean size, unusually large pull requests, and how size changes over time.

Say whether pull requests generally look small and easy to review, moderately sized, frequently large, or extremely variable.

Look for evidence that rising size goes with slower reviews, a larger backlog or longer time to merge. Pay attention to the long tail of very large ones.

Do not invent numerical conclusions when size data is unavailable.

## 6. Flow and consistency

Assess how smoothly work moves through the system. Look for months where opened greatly exceeds resolved, where:

    resolved = merged + closed

Look for sustained trends rather than treating one unusual month as evidence of a systemic problem.

Identify roughly when project behaviour materially changed. For example: "the backlog begins growing consistently from March onwards". That is usually more useful than describing current values.

## 7. Warning signals

Look explicitly for combinations such as:

* growing backlog with rising pull request age
* fast median merge time alongside a large stale backlog
* rising pull request volume without matching review capacity
* a falling merge rate
* rising pull request size
* rising time to merge
* large swings in throughput
* high work in progress
* a few extremely old pull requests distorting the mean
* a large gap between mean and median
* several consecutive months of positive backlog growth
* development throughput rising while review throughput stays flat

Combinations matter more than individual metrics. Growing backlog, plus ageing backlog, plus rising merge time, is a far stronger warning than any one of them alone.

## 8. Positive signals

Look for evidence including:

* backlog staying stable despite high throughput
* backlog shrinking
* consistently young open pull requests
* few or no pull requests older than 90 days
* fast but sustainable review times
* small pull request sizes
* high and stable merge rates
* work arriving and leaving at similar rates
* throughput rising without work in progress rising
* temporary backlog growth being cleared afterwards

One particularly strong signal is development activity increasing without backlog size or backlog age increasing. That suggests the process is scaling successfully.

## 9. Trend analysis

Do not simply describe the current snapshot. Explain what the project looked like about 12 months ago, what changed during the period, whether things are getting better, worse or holding steady, roughly when any meaningful change started, and whether recent months differ from earlier ones.

Prioritise sustained trends over short-term noise.

## 10. Operational interpretation

Explain what the metrics may be saying operationally. Consider:

* Is review capacity keeping up with development capacity?
* Is work getting stuck?
* Is the team carrying excessive work in progress?
* Is there evidence of good backlog hygiene?
* Are stale pull requests routinely closed?
* Are developers starting more work than the team can review?
* Does the project have a two-speed pull request system?
* Does the development process look sustainable?
* Has delivery capacity grown without review capacity growing?

Do not claim to know the organisational reason behind a change unless the data demonstrates it. Prefer:

    Review capacity appears not to be keeping pace with incoming development
    work. I would investigate whether team responsibilities, staffing or
    review practices changed around March.

rather than:

    The team lost a reviewer in March.

The second cannot be inferred from pull request data.

## Flow pattern classification

As well as the score, classify the project into the flow pattern that fits best.

### Balanced flow

Work arrives and leaves at roughly the same rate: stable backlog, young backlog, sustainable review times, consistent throughput. Generally a strong signal of health.

### Scaling successfully

Development throughput is rising while the backlog stays controlled, backlog age stays low and review time stays stable. An especially positive signal.

### Episodic but controlled

Activity comes in bursts rather than continuously, but the backlog stays controlled, work eventually gets resolved, and work in progress does not grow without limit. This can be normal for open source or release-driven projects.

### Fast lane with stale long tail

Most active pull requests are reviewed and merged very quickly while another group stays open for a long time. Typical signals: very low median merge time, many old open pull requests, high mean backlog age, and a wide gap between typical behaviour and the long tail. Common in large open source projects.

### Growing congestion

New work consistently enters faster than it is resolved: rising backlog, rising backlog age, slower merge times, repeated months of positive backlog growth. This generally deserves investigation.

### Backlog recovery

The project previously accumulated work in progress and is now closing stale work, reducing backlog and backlog age, and improving merge throughput. Improving health.

### Unstable flow

The metrics vary substantially month to month with no clear pattern. This can indicate changing practices, release cycles, staffing changes or inconsistent review capacity. Do not assume it is unhealthy, but highlight it.

## Output format

Use approximately this format:

    # Project health report

    ## Overall health: X/10, [classification]

    [2 to 4 sentence executive summary]

    ## Throughput

    [Analysis of opened, merged and closed pull requests.]

    Over the latest 12 complete months:

    - X opened
    - X merged
    - X closed
    - net backlog change: +/- X

    [Interpretation.]

    ## Backlog health

    [Current backlog size, age and stale pull request analysis.]

    ## Review and merge speed

    [Median, mean and within-one-day analysis.]

    ## Pull request size

    [Analysis if the data is available.]

    ## Trend

    [What changed during the year, and roughly when.]

    ## Flow pattern

    [Pattern name, then the explanation.]

    ## What looks good

    - ...

    ## What concerns me

    - ...

    (If there are no substantial concerns, say so rather than manufacturing
    issues.)

    ## What I would investigate

    - ...

    (Questions suggested by the data, not claims about causes.)

    ## Health score

    X/10, with 2 to 3 sentences of justification.

    ## Bottom line

    Based purely on the pull request data, does this look like a well-run
    software project? Answer in plain English.

## Analysis principles

### Evaluate combinations of metrics

No single metric should decide overall health. A median merge time of 0.2 days does not indicate a healthy project if half the open pull requests are more than 90 days old. High throughput is not automatically positive if the backlog is growing faster still.

### Prefer medians for typical behaviour

Means are useful but are pulled around by a few extremely old or extremely large pull requests. Where both are given, explain any significant gap between mean and median. That gap is itself information.

### Distinguish work in progress from throughput

The number of pull requests created is not a measure of productivity on its own. The question is whether the system can absorb and resolve that work sustainably.

### Consider context

Open source projects legitimately behave differently from commercial teams. A maintainer may deliberately leave low-priority community pull requests open while processing important work quickly. Describe a project analytically rather than judgementally.

### Do not invent causes

Metrics show symptoms and patterns. They rarely prove organisational causes. Say what should be investigated rather than claiming certainty about why something happened.