Silent API drift: the ATS changes under you

Silent API drift — the ATS changes under you

The failure that costs you most isn't the one that throws an exception. It's the one that doesn't. Studies of API evolution suggest roughly 15% of changes break backwards compatibility (one large-scale study of 317 Java libraries put it at 14.78%), and Bullhorn and JobAdder evolve on their own timeline, not yours. A field gets renamed. Pagination changes shape. A response that used to carry employmentType now calls it something else, or stops sending it.

Your tool doesn't crash. It does something worse: it carries on, scoring every candidate against a requirement field that is now silently, permanently empty. The shortlist still appears. It's just quietly wrong, and nobody knows until a placement goes sideways weeks later.

The defence is to deserialise defensively and fail loud: assert that the fields you depend on are actually present, and raise an alert the moment one goes missing, rather than treating absence as a polite empty string.

// Illustrative excerpt. Contract assertion on the ATS payload — fail loud, not silent.
static JobRequirements RequireFields(JobOrder job)
{
    // Bullhorn: a renamed/removed field comes back null, not as an error.
    if (string.IsNullOrWhiteSpace(job.Title) || job.Skills is null)
        throw new AtsContractException(
            $"Bullhorn JobOrder {job.Id}: expected fields missing — possible API drift");
    return new JobRequirements(job.Title, job.Skills);
}
// JobAdder's GET /jobs/{jobId} payload carries skills under skillTags.tags — same guard.

The other half of drift is authentication, which breaks on its own schedule. Bullhorn's REST BhRestToken session expires and starts returning 401; JobAdder's OAuth2 access_token expires roughly every hour (expires_in: 3600, about 60 minutes) and must be refreshed against https://id.jobadder.com/connect/token. JobAdder's refresh token rotates: each refresh returns a new access_token, a new refresh_token, and a fresh per-account api base URL, so you persist the new refresh token every time or the next refresh fails. Neither is an error in your code. It's expected behaviour you have to handle: catch the 401, re-authenticate, retry once, and only then escalate.

// Illustrative excerpt. Re-auth on 401, retry once, then give up loudly.
async Task<T> WithReauth<T>(Func<Task<T>> call)
{
    try { return await call(); }
    catch (AtsUnauthorizedException)          // BhRestToken expired / JobAdder access_token stale
    {
        await _ats.RefreshSessionAsync();     // Bullhorn login refresh / JobAdder refresh_token
                                              //   rotate — persist the NEW refresh token + api URL
        return await call();                  // one retry on a fresh session
    }
}

These aren't edge cases you might hit. They're certainties you will hit. The only question is whether your monitoring tells you, or a client does. (Catching drift before it bites is the canary-and-contract-test job that belongs to monitoring and observability.)

Idempotency — the retry that writes twice

Here's where two of the safeguards above collide. You retry a timed-out ATS write, which is sensible. But the original write may have succeeded; the timeout was on the response, not the request. Now you've written the same shortlist Note twice, or re-emailed the same candidate. "At least once" delivery has just met your system of record, and the system of record believed you both times.

The fix is an idempotency key: derive a deterministic key from what the action is (candidate, job, and a hash of the content), check whether that exact action already happened, and skip the write if it did.

// Illustrative excerpt. Idempotent decision write keyed on (candidate, job, content).
var key = Idempotency.Key(s.CandidateId, jobId, actionHash: s.DecisionHash());

if (await _store.AlreadyApplied(key))         // dedupe check before any write
    return WriteResult.Skipped(key);

await _bullhorn.PutAsync("entity/Note", noteBody);   // the actual side-effect
await _store.MarkApplied(key);                        // record it so a retry is a no-op

Read-back verification is the belt to that braces: after a write that matters, read it back and confirm it landed once. It's a few extra milliseconds against the alternative: a candidate who gets two "you've been shortlisted" notes, or a client who sees the same name twice on the same list.