318 Batch Jobs, 0 Work Orders, 1 Dead URL

Written with Claude after reading too many AsyncApexJob rows with the same ExtendedStatus.

This is a postmortem on a production integration between Salesforce and Installation Made Easy (IME / MIC): custom object WorkOrderChange__c, after-insert/update trigger, WorkOrderChangeBatch with Database.AllowsCallouts, Named Credentials for outbound HTTP, and @RestResource MIC_WebhookHandler on an Experience Cloud site for inbound status events.

The package looked complete in the org. In production it had never successfully called IME.

287 WorkOrderChange__c records. 0 with WorkOrderId__c populated. 0 with MIC_Response_Code__c. 318 WorkOrderChangeBatch jobs in the last 30 days, failing at ExternalCalloutHelper.getAuthToken() with Failed to get token. Code: 302 and an HTML redirect body.

The integration user credentials were valid. The Named Credential endpoints were not.

Architecture as deployed

Outbound path:

DML on WorkOrderChange__c (Flow, manual, or REST — the creator isn’t in source control).
WorkOrderChangeTrigger (after insert / after update) → WorkOrderChangeHandler.
Handler enqueues Database.executeBatch(new WorkOrderChangeBatch(ids, operationType), 50).
Batch execute() calls getAuthToken() via callout:MIC_Authentication, then sendToExternalSystem() via callout:MIC_WorkordersPost with Authorization: Bearer {token}.
Batch DML updates MIC_Response_Code__c, WorkOrderId__c, MIC_Sync_DateTime__c, MIC_Status_Tracker__c.

Inbound path:

IME POSTs to https://{site}.my.site.com/webhooksvforcesite/services/apexrest/micwebhook
MIC_WebhookHandler deserializes Type, WorkOrderId, Status, queries WorkOrderChange__c WHERE WorkOrderId__c = :workOrderId, updates Status__c

Triggers cannot perform callouts. Queueing async work from a trigger is normal. What’s abnormal is that step 4 never returned 200 in prod, so step 5 never stored an external id, so the inbound lookup always missed.

Metadata that wasn’t in git

MIC_WebhookHandler, ExternalCalloutHelper, WorkOrderChangeBatch, and related metadata were in the production org but missing from the repo until an explicit retrieve by Apex class name.

A sf project retrieve start --source-dir force-app only refreshes components already on disk. Net-new Apex in the org does not appear in a diff you never had. Operational risk: reviews and CI run against an incomplete picture of what’s deployed.

Named Credentials pointed at the wrong host

Production Named Credentials:

Named Credential	Endpoint in org	IME v2 API
`MIC_Authentication`	`https://apps.trustedhomeservices.com:443/Account/Login`	`https://extapi.installationmadeeasy.com/api/v2/auth/login`
`MIC_WorkordersPost`	`https://apps.trustedhomeservices.com:443/WorkOrders`	`https://extapi.installationmadeeasy.com/api/v2/WorkOrders`

ExternalCalloutHelper builds the auth body from System.label.MIC_Username and System.label.MIC_Password, POSTs JSON, expects HTTP 200 and accessToken in the response. On anything else it throws CalloutException, which fails the whole batch execute() — no partial row updates, no MIC_Response_Code__c written for that scope.

Repro outside the platform with the same payload:

# Configured in org → 302, HTML redirect
curl -s -o /dev/null -w "%{http_code}\n" -X POST \
  "https://apps.trustedhomeservices.com/Account/Login" \
  -H "Content-Type: application/json" -d '{"username":"…","password":"…"}'

# Current vendor API → 200, accessToken in JSON body
curl -s -X POST "https://extapi.installationmadeeasy.com/api/v2/auth/login" \
  -H "Content-Type: application/json" -d '{"username":"…","password":"…"}'

Http.send() does not behave like a browser following redirects for you. A 302 to a login page is not an auth error message — it’s the wrong integration surface. 318 batch failures are the platform doing what the metadata told it to do.

Secrets in Custom Labels

Username and password are not on the Named Credential principal. Both NCs use protocol: NoAuthentication with principalType: Anonymous. Secrets live in Custom Labels and deploy as plain text in CustomLabels.labels-meta.xml.

That pattern “works” for a basic login POST, but it’s the wrong abstraction: no External Credential, no protected custom setting, rotation tied to metadata deploys. Separate issue from the outage, same operational bucket.

Trigger → batch for per-record sync

WorkOrderChangeHandler.handleAfterInsert only adds ids when Status__c == 'APPOINTMENT_PENDING' and MIC_Response_Code__c is blank.

handleAfterUpdate enqueues batch on any Status__c change — no static Boolean reentrancy guard, no “skip if MIC_Sync_Source__c == 'Inbound'” field, no check for existing WorkOrderId__c before calling create again.

Database.executeBatch with batch size 50 is reasonable for bulk catch-up. For “this one row changed status, notify vendor now,” a Queueable (or Platform Event → subscriber) gives you faster turnaround, clearer failure isolation per record, and an easier place to implement idempotency. Here, every status flip schedules another batch that re-authenticates and re-POSTs.

Prod data also shows many rows created directly as APPOINTMENT_SET, which never satisfies the insert-time filter. Outbound on insert doesn’t run until a later update changes status — if the Flow always sets APPOINTMENT_SET on create, insert-time sync never fires.

Create endpoint used for updates

WorkOrderChangeBatch constructor takes operationType ('Insert' | 'Update'). That value is appended to MIC_Status_Tracker__c only.

ExternalCalloutHelper.sendToExternalSystem always POSTs to the work order create path. It does not pass IME’s workOrderId on update. Object fields AffiliateId__c, subCategoryId__c, businessModelId__c are commented out in the payload builder; hardcoded partnerId 19, subCategoryId 223, businessModelId 1577 instead.

Vendor API semantics: POST /WorkOrders creates. Updates require other operations with an existing id. After fixing Named Credentials, status-driven batch runs risk duplicate work orders in IME unless the callout layer branches on WorkOrderId__c != null and uses the correct verb.

Inbound webhook blocked on outbound key

MIC_WebhookHandler matches WorkOrderId__c (stringified from JSON numeric WorkOrderId). That field is set from json.get('workOrderId') only when the outbound POST returns 200.

With 0 / 287 rows holding an external id, inbound STATUS_CHANGED notifications return 404 — handler message: WorkOrderChange not found for WorkOrderId: {id}. The Experience Cloud URL can be correct, guest profile can expose the Apex class, and the integration still does nothing useful until outbound create succeeds at least once per row.

Issues visible after auth is fixed

These didn’t show up in row-level fields because auth never got past step 1:

Null overwrite: rec.WorkOrderId__c = response.externalId runs even when externalId is null after a failed callout — can clear a previously stored IME id on retry.
Re-entrancy: Webhook update on Status__c → after-update trigger → another batch → another create POST. Needs an inbound-origin flag or status-only webhook handler that doesn’t fire the outbound trigger path.
Observability: Failures surface in AsyncApexJob / debug logs; MIC_Response_Code__c stays blank when execute() throws before the update loop. No Log__c, no Platform Event for failed sync.
REST surface: GetSlotsWebhooks and WorkOrderChangeService lack the x-api-key pattern used on CHAU endpoints (Keys__c CHAU Webhooks API Key). Worth hardening separately.

What I’d check first next time

When AsyncApexJob says Failed to get token. Code: 302, don’t assume SSO or certificate issues until you curl the Named Credential URL and read the response body.

When WorkOrderId__c is null on every row in the object, don’t debug the webhook first — debug outbound.

When 287 records and 318 failed jobs agree nothing left the org, believe the data over the fact that Apex classes exist and tests pass in a sandbox with different endpoints.

Fixing prod starts with updating MIC_Authentication and MIC_WorkordersPost to extapi.installationmadeeasy.com /api/v2/..., validating one row end-to-end, then tackling update-vs-create, trigger guards, and moving secrets off Custom Labels.

Co-authored with Claude. If you’re maintaining a similar pattern — trigger, batch, Named Credential, REST callback — verify the endpoint outside Salesforce before you burn a month of job queue.