Adding Custom Job Board Providers

This section guides you through extending Envoy's capabilities to search and apply for jobs on new job boards. Integrating a new provider primarily involves adding browser automation logic within the Tools service, which the Agent service then orchestrates.

How Provider Integration Works

Envoy's architecture separates core logic (Agent) from browser interactions (Tools). When you integrate a new job board:

The Agent sends requests to the Tools service, specifying the job board's unique provider identifier (e.g., seek, linkedin).
The Tools service dispatches these requests to provider-specific modules (e.g., tools/src/providers/seek/).
These modules use Playwright to interact with the job board's website, performing actions like searching for jobs, parsing job details, and filling out application forms.

This design means most of your development work will be within the tools/ directory, following existing patterns.

Extending the `Tools` Service

To add a new provider, let's use "MyJobBoard" as an example. You'll typically create a new directory tools/src/providers/myjobboard/ and implement specific functionalities.

1. Create the Provider Module Structure

Start by creating a dedicated directory and files for your new provider:

tools/src/providers/myjobboard/
├── apply.ts
├── search.ts
└── details.ts

2. Implement Job Search Logic (`search.ts`)

This module is responsible for navigating to the job board's search page, performing a search based on keywords and location, and extracting basic job listing information.

Key Responsibilities:

Navigate to the job board's search interface.
Fill in search criteria (keywords, location).
Submit the search form.
Parse the search results from the listing page, extracting essential fields like title, company, location, url, and a unique provider_job_id.

Example (tools/src/providers/myjobboard/search.ts):

import type { Page } from 'playwright-core';
import { z } from 'zod';

// Define the schema for a job listing extracted during search
const SearchJobSchema = z.object({
  provider_job_id: z.string(),
  title: z.string(),
  company: z.string(),
  location: z.string(),
  url: z.string().url(),
  // Add other relevant fields found on listing pages (e.g., salary, work_type)
});
export type SearchJob = z.infer<typeof SearchJobSchema>;

export async function searchJobs(
  page: Page,
  keywords: string,
  location: string,
): Promise<SearchJob[]> {
  // 1. Navigate to MyJobBoard's job search page
  await page.goto('https://www.myjobboard.com/jobs');

  // 2. Fill in the search fields
  await page.getByPlaceholder('Keywords').fill(keywords);
  await page.getByPlaceholder('Location').fill(location);
  await page.locator('button[type="submit"]').click();

  // 3. Wait for the search results to load
  await page.waitForSelector('.job-listing-card'); // Adjust selector for your site

  // 4. Extract job details from the listing page
  const jobs: SearchJob[] = await page.evaluate(() => {
    const results: SearchJob[] = [];
    document.querySelectorAll('.job-listing-card').forEach(card => {
      const title = card.querySelector('.job-title')?.textContent?.trim();
      const company = card.querySelector('.company-name')?.textContent?.trim();
      const location = card.querySelector('.job-location')?.textContent?.trim();
      const url = card.querySelector('a.job-link')?.href;
      const provider_job_id = url ? new URL(url).pathname.split('/').pop() : 'unknown'; // Extract ID from URL

      if (title && company && location && url && provider_job_id) {
        results.push({ title, company, location, url, provider_job_id });
      }
    });
    return results;
  });

  return jobs;
}

3. Implement Job Details Fetching Logic (`details.ts`)

After a job is discovered in the search phase, Envoy needs to fetch the full job description and other detailed information to generate cover letters and assess suitability.

Key Responsibilities:

Navigate to a specific job detail page using its URL.
Extract the full job description text and any other valuable structured data (e.g., required skills, benefits, application instructions).

Example (tools/src/providers/myjobboard/details.ts):

import type { Page } from 'playwright-core';
import { z } from 'zod';

// Define the schema for detailed job information
const JobDetailsSchema = z.object({
  full_description: z.string(),
  // Add other detailed fields if available on the job page
});
export type JobDetails = z.infer<typeof JobDetailsSchema>;

export async function fetchJobDetails(page: Page, url: string): Promise<JobDetails> {
  // 1. Navigate to the job's detail page
  await page.goto(url, { waitUntil: 'domcontentloaded' });
  await page.waitForSelector('.job-description-content'); // Adjust selector

  // 2. Extract the full job description text
  const full_description = await page.$eval(
    '.job-description-content',
    el => el.textContent?.trim() || ''
  );

  return { full_description };
}

4. Implement Application Workflow Logic (`apply.ts`)

This module is crucial for initiating the application process and providing provider-specific logic for detecting different stages of an application workflow.

Key Functions:

startApply(page: Page, jobUrl: string): Navigates to the job listing and clicks the initial "Apply" button to begin the application.
isConfirmationPage(text: string): Analyzes the current page's visible text to determine if it's a final application confirmation page.
isExternalPortalUrl(url: string): Checks if the current URL indicates that the application has redirected to an external career portal (e.g., Workday, Greenhouse).
detectPortalType(url: string): (Optional) Provides more granular classification of external portals if different handling strategies are required for them.

Example (tools/src/providers/myjobboard/apply.ts):

import type { Page } from 'playwright-core';

export async function startApply(page: Page, jobUrl: string): Promise<void> {
  await page.goto(jobUrl, { waitUntil: 'domcontentloaded' });
  // Wait for and click the primary "Apply Now" button on the job page
  await page.locator('button.apply-now').click();
  await page.waitForLoadState('networkidle', { timeout: 20_000 }).catch(() => {});
}

export function isConfirmationPage(text: string): boolean {
  // Look for common phrases indicating successful submission
  return text.includes('Application submitted') || text.includes('Thank you for your application');
}

export function isExternalPortalUrl(url: string): boolean {
  // Determine if the URL has left the primary job board domain
  return !url.includes('myjobboard.com');
}

export function detectPortalType(url: string): string | null {
  // Optionally, provide specific identifiers for known external systems
  if (url.includes('workday.com')) return 'workday';
  if (url.includes('greenhouse.io')) return 'greenhouse';
  return null;
}

5. Integrate with the `Tools` Service API

The Tools service exposes an API for the Agent. You'll need to update relevant files within tools/src/ to integrate your new provider.

a. Update tools/src/browser/routes.ts

This file handles generic browser operations and session management.

Import your provider's apply.ts functions:

// tools/src/browser/routes.ts
// ... other imports ...
import {
  isConfirmationPage as myjobboardIsConfirmation,
  isExternalPortalUrl as myjobboardIsExternalPortal,
  detectPortalType as myjobboardDetectPortalType,
  // startApply is typically called via a dedicated API route, not here directly
} from '../providers/myjobboard/apply.js';

Add your provider's login URL to PROVIDER_LOGIN_URLS: This allows Envoy to detect when a login is required for your job board.

// tools/src/browser/routes.ts
const PROVIDER_LOGIN_URLS: Record<string, string> = {
  seek: 'https://www.seek.com.au/oauth/login/',
  linkedin: 'https://www.linkedin.com/login',
  myjobboard: 'https://www.myjobboard.com/login', // Add your provider's login URL
};

Update inspectOptsFor(provider: string): This function provides provider-specific page detection logic to the generic inspectStep function.

// tools/src/browser/routes.ts
function inspectOptsFor(provider: string): InspectOptions {
  if (provider === 'seek') {
    return {
      isConfirmation: (text) => seekIsConfirmation(text),
      isExternalPortal: (url) => seekIsExternalPortal(url),
      detectPortalType: (url) => seekDetectPortalType(url),
    };
  }
  // Add your new provider's logic
  if (provider === 'myjobboard') {
    return {
      isConfirmation: (text) => myjobboardIsConfirmation(text),
      isExternalPortal: (url) => myjobboardIsExternalPortal(url),
      detectPortalType: (url) => myjobboardDetectPortalType(url),
    };
  }
  return {};
}

b. Register Provider-Specific API Routes (in tools/src/main.ts or a dedicated routes file)

The Agent service interacts with the Tools service through dedicated HTTP endpoints for provider-specific actions. You'll need to add (or extend existing) routes to dispatch calls to your search.ts, details.ts, and apply.ts functions.

// Hypothetical: tools/src/main.ts or tools/src/routes/providers.ts
// Assuming this file registers routes like /tools/providers/:provider/*

import type { FastifyInstance } from 'fastify';
import { ok, error } from '../envelope.js'; // Assuming envelope.js for responses
import { getSession, createSession, sessionError } from '../browser/sessions.js';
import { z } from 'zod';

// Import your provider's functions
import { searchJobs as myjobboardSearchJobs, type SearchJob } from '../providers/myjobboard/search.js';
import { fetchJobDetails as myjobboardFetchJobDetails, type JobDetails } from '../providers/myjobboard/details.js';
import { startApply as myjobboardStartApply } from '../providers/myjobboard/apply.js';

// Also import existing providers like seek
import { searchJobs as seekSearchJobs } from '../providers/seek/search.js';
import { fetchJobDetails as seekFetchJobDetails } from '../providers/seek/details.js';
import { startApply as seekStartApply } from '../providers/seek/apply.js';


export function registerProviderRoutes(app: FastifyInstance): void {

  // Route for job searching: POST /tools/providers/:provider/search
  app.post('/tools/providers/:provider/search', async (request) => {
    const paramsParsed = z.object({ provider: z.string().min(1) }).safeParse(request.params);
    const bodyParsed = z.object({ keywords: z.string(), location: z.string() }).safeParse(request.body);
    if (!paramsParsed.success) return error('bad_request', paramsParsed.error.message);
    if (!bodyParsed.success) return error('bad_request', bodyParsed.error.message);

    const { provider } = paramsParsed.data;
    const { keywords, location } = bodyParsed.data;

    // A session is required to perform browser actions
    const session_key = await createSession(provider); // Or retrieve an existing session
    const session = getSession(session_key);
    if (!session) return sessionError(session_key);

    let jobs: SearchJob[] = [];
    if (provider === 'seek') {
      jobs = await seekSearchJobs(session.page, keywords, location);
    } else if (provider === 'myjobboard') { // Add dispatch for your provider
      jobs = await myjobboardSearchJobs(session.page, keywords, location);
    } else {
      return error('not_found', `unknown provider: ${provider}`);
    }
    return ok({ jobs });
  });

  // Route for fetching job details: POST /tools/providers/:provider/details
  app.post('/tools/providers/:provider/details', async (request) => {
    const paramsParsed = z.object({ provider: z.string().min(1) }).safeParse(request.params);
    const bodyParsed = z.object({ url: z.string().url(), session_key: z.string() }).safeParse(request.body);
    if (!paramsParsed.success) return error('bad_request', paramsParsed.error.message);
    if (!bodyParsed.success) return error('bad_request', bodyParsed.error.message);

    const { provider } = paramsParsed.data;
    const { url, session_key } = bodyParsed.data;

    const session = getSession(session_key);
    if (!session) return sessionError(session_key);

    let details: JobDetails;
    if (provider === 'seek') {
      details = await seekFetchJobDetails(session.page, url);
    } else if (provider === 'myjobboard') { // Add dispatch for your provider
      details = await myjobboardFetchJobDetails(session.page, url);
    } else {
      return error('not_found', `unknown provider: ${provider}`);
    }
    return ok({ details });
  });

  // Route for starting the application: POST /tools/providers/:provider/start_apply
  app.post('/tools/providers/:provider/start_apply', async (request) => {
    const paramsParsed = z.object({ provider: z.string().min(1) }).safeParse(request.params);
    const bodyParsed = z.object({ jobUrl: z.string().url(), session_key: z.string() }).safeParse(request.body);
    if (!paramsParsed.success) return error('bad_request', paramsParsed.error.message);
    if (!bodyParsed.success) return error('bad_request', bodyParsed.error.message);

    const { provider } = paramsParsed.data;
    const { jobUrl, session_key } = bodyParsed.data;

    const session = getSession(session_key);
    if (!session) return sessionError(session_key);

    if (provider === 'seek') {
      await seekStartApply(session.page, jobUrl);
    } else if (provider === 'myjobboard') { // Add dispatch for your provider
      await myjobboardStartApply(session.page, jobUrl);
    } else {
      return error('not_found', `unknown provider: ${provider}`);
    }
    return ok({});
  });

  // Other browser-agnostic routes like inspect_apply_step would remain in browser/routes.ts
  // and use the inspectOptsFor function which now includes your provider's logic.
}

6. Playwright Selectors Best Practices

When writing Playwright code for your provider, aim for robust selectors that are less likely to break with minor UI changes:

Prioritize user-facing attributes: Use page.getByRole(), page.getByLabel(), or page.getByText(). These mimic how a human interacts with the page and are often more stable.
Utilize data-testid or data-automation: If the job board uses custom data attributes for testing or automation, these are excellent and stable targets.
Avoid fragile CSS selectors: Steer clear of deeply nested or index-based CSS selectors (e.g., .container > div:nth-child(2) > span) as they are highly susceptible to UI changes.
Use firstVisible(locator): The tools/src/browser/routes.ts file provides a firstVisible helper function. This is useful for interacting with elements that might have multiple hidden instances, ensuring you target the interactable one.

Agent Service Interaction

The Agent service communicates with the Tools service using HTTP endpoints, passing the provider name as part of the request. For example, to initiate a job search, the Agent makes a request similar to:

POST http://127.0.0.1:4320/tools/providers/myjobboard/search
Body: { "keywords": "Python", "location": "Remote" }

Generally, you do not need to modify the Agent service to add a new provider, as long as the Tools service adheres to the existing API contract. The Agent dynamically picks up the provider name and passes it down to the Tools service.

Testing Your New Provider

Unit Tests for Tools Modules: Write unit tests for your search.ts, details.ts, and apply.ts functions. You can use Playwright's Page object within these tests, potentially mocking network requests or running headless browser instances.
Integration Tests with Agent: Use mocking libraries like respx.mock (as seen in agent/tests/test_api_jobs.py) to simulate HTTP responses from the Tools service. This verifies that the Agent correctly calls your new provider and processes its responses throughout the job lifecycle.
Manual Testing: Run Envoy locally and configure it to target your new job board. Carefully observe its behavior during search, review, and application phases to ensure everything works as expected.

By following these guidelines, you can effectively extend Envoy to support a wider range of job boards, enhancing its autonomous application capabilities.

Adding Custom Job Board Providers