How to Tell If Your Azure DevOps Organization Is Healthy: 12 Checks

Short answer: a healthy Azure DevOps organization is one you could rebuild from Git, where access is narrow enough to explain, and where every pipeline and agent pool has an owner. If any of those is shaky, the org is carrying risk that will show up during an audit, an outage, or a migration. The 12 checks below will tell you where you stand in an afternoon.

Most Azure DevOps orgs were not designed. They accumulated: a project per team, a pipeline per app, a service connection created on a deadline. None of that is a moral failing. It is what busy orgs look like. The trick is to measure it before it measures you.

You can do every check here by hand in the portal. You can also run the free ADO Health Check CLI, which automates a good part of it and writes the results to a JSON inventory and an HTML report.

1. How much of your delivery still runs on classic pipelines?

Count classic build and release definitions against YAML pipelines. A classic pipeline lives only in the UI, so a change to it cannot be reviewed in a pull request, and nobody can rebuild it from source control. A few left over are normal. If most of your production releases are classic, that is the single biggest reproducibility gap in the org.

2. Are your YAML pipelines built on shared templates?

YAML alone is not the goal. If every repository has its own hand-written pipeline, you have moved the copy-paste problem into Git. Look for shared templates that carry the build, test, and security steps, so a fix lands once instead of forty times.

3. What do your failed-run trends look like?

Pull the last 30 days of runs per pipeline. A pipeline that fails one run in five and gets rerun until it passes is teaching your team to ignore red. Flaky pipelines are a reliability problem first and a morale problem second.

4. Are retention settings deliberate?

Check run and artifact retention at the project and pipeline level. Keep too little and you lose the evidence for how a release reached production. Keep everything forever and you pay for it in storage and noise. Either way, it should be a choice someone made on purpose.

5. Who is a Project Collection Administrator, and why?

List the members of Project Collection Administrators and every project's administrators group. Each name should have a reason you can say out loud. Admin rights tend to grow one favor at a time and almost never shrink on their own.

6. Do you know every guest account?

Review guest and external users in the organization and the backing Microsoft Entra ID tenant. Contractors who left last year, partner accounts from a finished project, and test accounts are the usual finds.

7. How broad are your service connections?

Service connections are how pipelines reach Azure and everything else. Check what scope each one has, which projects and pipelines are allowed to use it, and how it authenticates. A connection with subscription-wide rights, shared across every project and using a secret that never rotates, is a high-value target. Narrow the scope, restrict which pipelines can use it, and prefer workload identity federation where Azure supports it so there is no secret to leak.

8. Where do your secrets actually live?

Open your variable groups. Secrets should be marked secret or linked to Azure Key Vault, not sitting in plain values that anyone with read access to the library can see. While you are there, look for secrets pasted straight into pipeline YAML.

9. Are branch policies consistent on the branches that ship?

Check the default and release branches in each repository for required reviewers, a build validation policy, and linked work items if you need them. The question is not whether policies exist somewhere. It is whether "required review" means the same thing in every repository that reaches production, and who is allowed to bypass it.

10. Can you trace a production release back to an approval?

Pick a recent production release and walk it backward: which commit, which pull request, who approved it, which pipeline run, which environment check. If that takes more than a few minutes, or ends in a chat thread, your promotion gates are not doing their job. Environments with approvals and checks make this trail automatic.

11. Is any agent pool a single point of failure?

Look at your self-hosted pools. One machine running every production deploy, or a pool nobody can rebuild because it was configured by hand, is an outage waiting for a reboot. Decide what really needs self-hosted agents, such as private network access or special tooling, and use Microsoft-hosted agents for the rest.

12. Are your agents current and defined as code?

Check agent versions across pools. Mismatched versions and unpatched hosts are common, and they turn into odd failures and security findings. The fix that lasts is building agent images and pools as infrastructure as code, so replacing an agent is routine instead of brave.

Do not try to fix all twelve at once. Fix the access and secrets findings first (checks 5 through 8), because they carry the most risk and are usually the quickest to close. Then work through pipelines and agents project by project, so delivery keeps moving while the org gets healthier.

Write the findings down as a backlog with owners, and re-run the same checks afterward so you can show what moved. Measuring the same way twice is what turns a cleanup into evidence.

If you want a head start, run the free ADO Health Check CLI against your org. If you want help with what it finds, our Azure DevOps health and modernization work starts with exactly this kind of evidence, and the 60–90 minute assessment will walk the report with your team and rank the next moves.

— The Mentat Collective