Azure App Testing: Fix Setup, Config & Load Test Errors

The portal wizard for creating an Azure Load Testing resource looks simple, but there are a few fields that will cause you pain later if you rush past them. Open the Azure portal and search for Azure Load Testing in the top search bar. Click Create. On the Basics tab, your choices here are permanent, you cannot move a Load Testing resource to a different region without recreating it from scratch, so pick the region closest to your application's hosting region.

Give the resource a name that includes the environment suffix, something like myapp-loadtest-prod or myapp-loadtest-staging. I've seen teams create a single shared resource for all environments and then wonder why test results from a staging run are contaminating their production metrics. Keep them separate.

On the Review + create tab, pay attention to the Tags section if your organization uses Azure Policy to enforce tagging. Missing a required tag here will cause the deployment to fail with an error like RequestDisallowedByPolicy, which looks scary but just means your IT department has a policy that requires certain metadata tags on all resources. Add them and redeploy.

For Playwright Workspaces, the creation path is under Azure App Testing > Playwright Workspaces. The workspace creation wizard is slightly different, you'll also see an option to configure regional affinity for browser workers. Leave this at the default (same region as workspace) unless your application has a specific geographic audience you need to simulate.

After the resource shows Deployment succeeded, click Go to resource. If the Overview blade shows any warning banners about configuration being incomplete, address those before running your first test, they're telling you something real.

The fastest way to validate that your Azure Load Testing resource is working correctly is to run a URL-based test, no JMeter script, no Locust file, just a URL. This also happens to be how Microsoft recommends getting started, and I agree with that call. It removes script errors as a variable entirely.

Inside your Azure Load Testing resource, click Tests in the left navigation, then click Create > Create a URL-based test. Enter your application's URL in the Add requests field. If your app is behind authentication, you can add HTTP headers (like an Authorization: Bearer <token> header) under Add request > Headers.

On the Load tab, set your virtual user count to something modest for a first run, 10 to 25 users. Don't immediately blast your production app with 500 virtual users. The Test duration and Ramp-up time settings control how quickly those virtual users spin up. A 60-second ramp-up with a 5-minute test duration is a reasonable starting point for validating connectivity.

Under the Test criteria tab, you can define fail criteria, for example, flagging the test as failed if average response time exceeds 2,000ms or the error rate goes above 5%. Set these now, even for your first test. Getting into the habit of defining pass/fail boundaries early is what separates teams that catch regressions from teams that don't.

Click Run test. Watch the real-time dashboard. If you see virtual users climbing and response data appearing, your Azure App Testing resource is healthy and connected. If the test immediately moves to a Failed state, jump to the Advanced Troubleshooting section.

Once the URL-based test works, you're ready for real-world Azure Load Testing with JMeter or Locust scripts. This is where most teams have more advanced requirements, multiple request types, think time simulation, parameterized user data, non-HTTP endpoints.

In your Azure Load Testing resource, go to Tests > Create > Upload a script. Select your test type: JMeter (.jmx file) or Locust (.py file). Upload the script file. If your JMeter test plan references external files, CSV data files, JMeter plugins, or referenced properties files, you must upload those under Additional files. Missing a referenced file is one of the most common reasons JMeter-based tests fail with an error like FileNotFoundException in the test log.

For JMeter users: Azure Load Testing runs your .jmx file on distributed worker engines in the cloud. Some JMeter plugins that work locally aren't available in the cloud environment by default. If your test plan uses a custom plugin JAR, you need to upload it as an additional file and confirm it's compatible with the version of JMeter that Azure Load Testing is running. Check the official documentation for the current supported JMeter version before assuming your local plugin version will work.

For Locust users: your .py file must define a standard Locust HttpUser class. Azure Load Testing spins up your Locust workers on managed cloud infrastructure, you don't manage that infrastructure at all, which is exactly the point. Upload your file, set the number of users and spawn rate on the Load tab, and run. If your Locust script has local imports or dependencies, package them as additional files and update your import paths to match the flat directory structure Azure Load Testing uses.

# Example Locust script minimum structure
from locust import HttpUser, task, between

class AppUser(HttpUser):
    wait_time = between(1, 3)

    @task
    def load_homepage(self):
        self.client.get("/")

    @task(3)
    def load_product_page(self):
        self.client.get("/products")

After the test run completes, the results dashboard will show you requests per second, response time percentiles (p50, p90, p95, p99), and error rates. If the run succeeded and data looks reasonable, you're in good shape.

Getting Playwright Workspaces configured for parallel testing is a different workflow from the load testing side. You're not uploading a script to the portal, you're connecting your existing local Playwright test project to the cloud workspace via the @playwright/test package and a workspace access token.

First, make sure you have the Playwright Workspaces package installed in your project:

npm install --save-dev @azure/microsoft-playwright-testing

Next, go to your Playwright Workspaces resource in the portal. Click Access tokens in the left navigation and generate a new token. Copy it immediately, you won't be able to view it again after you leave the page. Store it as an environment variable named PLAYWRIGHT_SERVICE_ACCESS_TOKEN in your local environment and your CI/CD secret store.

In your Playwright project, add or update your playwright.config.ts to include the Azure service configuration:

// playwright.config.ts
import { defineConfig } from '@playwright/test';
import { getServiceConfig, ServiceOS } from '@azure/microsoft-playwright-testing';

export default defineConfig(
  getServiceConfig({
    os: ServiceOS.LINUX,
    runId: process.env.PLAYWRIGHT_SERVICE_RUN_ID,
  }),
  {
    testDir: './tests',
    // your existing config here
  }
);

Set the PLAYWRIGHT_SERVICE_URL environment variable to the endpoint URL shown on your workspace's Overview blade in the portal. Now when you run npx playwright test, your tests execute on cloud-hosted browsers managed by Playwright Workspaces instead of your local machine. You'll see the parallel workers spinning up in the workspace's test runs dashboard in real time.

If the connection fails with an authentication error, double-check that the access token hasn't expired and that the PLAYWRIGHT_SERVICE_URL matches exactly what the portal shows, including the trailing path component. Token expiry during a long CI job is a common failure mode; generate tokens with the longest expiry your security policy allows for automated pipelines.

Running tests manually is fine for initial validation, but the real value of Azure App Testing comes when it's part of your automated deployment pipeline. Both Azure Load Testing and Playwright Workspaces support CI/CD integration, and the setup is more approachable than most teams expect.

For Azure Load Testing in Azure Pipelines, Microsoft provides the AzureLoadTest@1 task. Add it to your azure-pipelines.yml:

- task: AzureLoadTest@1
  inputs:
    azureSubscription: 'your-service-connection'
    loadTestConfigFile: 'load-test-config.yaml'
    resourceGroup: 'your-resource-group'
    loadTestResource: 'myapp-loadtest-prod'

The load-test-config.yaml file in your repo defines test parameters, fail criteria, and which test plan to run. Keeping this file version-controlled alongside your application code means your load test definition evolves with your application, critical for catching performance regressions early, which is exactly what the Azure App Testing documentation recommends.

For Playwright Workspaces in CI/CD, the setup is even simpler. In your GitHub Actions workflow or Azure Pipelines YAML, set the two required environment variables (PLAYWRIGHT_SERVICE_ACCESS_TOKEN and PLAYWRIGHT_SERVICE_URL) as pipeline secrets, then run your Playwright test command as normal. The service configuration in your playwright.config.ts handles the rest.

# GitHub Actions example
- name: Run Playwright Tests
  env:
    PLAYWRIGHT_SERVICE_ACCESS_TOKEN: ${{ secrets.PLAYWRIGHT_SERVICE_ACCESS_TOKEN }}
    PLAYWRIGHT_SERVICE_URL: ${{ secrets.PLAYWRIGHT_SERVICE_URL }}
    PLAYWRIGHT_SERVICE_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}
  run: npx playwright test

One thing that bites teams in CI: the PLAYWRIGHT_SERVICE_RUN_ID must be unique for each test run. Using the pipeline's run ID combined with the attempt number (as shown above) is a reliable way to guarantee uniqueness. Duplicate run IDs cause test result data to get merged in the Playwright Workspaces dashboard in ways that make results confusing and unreliable.

After your pipeline runs, the Azure Load Testing results dashboard and the Playwright Workspaces test results view will both show historical data you can compare across runs, which is where the real performance regression detection value lives.