{"resourceId":"acm-dgov-genai-job-posting-extraction-validation-naswa-2025","versions":[{"version":"external-79b9e8ca94317b4ad52dcd3fa3f8c837a0b711a59863d91d5d0e677a712b32c9","resource":{"id":"acm-dgov-genai-job-posting-extraction-validation-naswa-2025","title":"Peer-reviewed ACM study by Amazon and NASWA authors finds Bedrock-extracted education requirements match Census survey data on a 6,800-posting sample","organization":"Amazon and NASWA authors (Howison et al.), ACM Digital Government: Research and Practice","sector":"State workforce agencies; labor market information research","geography":"United States (National Labor Exchange postings, Mar-Nov 2023)","publishedAt":"2025-02-12","publicationDate":"2025-02-12","eventDate":null,"sourceName":"Amazon Science (ACM Digital Government: Research and Practice, vol. 6, issue 1)","sourceLabel":"Extracting structured labor market information from job postings with generative AI","sourceUrl":"https://www.amazon.science/publications/extracting-structured-labor-market-information-from-job-postings-with-generative-ai","evidenceClass":"academic-research","outcomeClass":"emerging","topics":["operating-model","governance-procurement"],"finding":"Why now: the Nov 1 AWS post on NASWA's production pipeline gives no accuracy figures, and this paper is the method's only published validation. Authors from Amazon and NASWA (including NLx Research Hub manager Paul Daniels) sent a stratified sample of 6,800 National Labor Exchange postings (100 for each of 68 occupation groups, March to November 2023) through foundation models on Amazon Bedrock with a single JSON-schema prompt at temperature 0.2. Education requirement frequencies by occupation did not differ significantly from Census CPS ASEC survey data (Wilcoxon test, p=0.13). The authors name cost and runtime of scaling to millions of postings as the next challenge and propose distillation, which the 2026 pipeline redesign appears to address with a smaller model instead.","sledRelevance":"Cross-tagged to State Government because the Research Hub serves state workforce agencies. It gives state labor market units a published method and one validation check they can examine before relying on the parsed fields NASWA plans to release in 2027. Applicability is limited to public job postings; the paper validates only one field.","evidence":"Peer-reviewed journal article (DOI 10.1145/3674847; Crossref lists online publication Feb 12, 2025). The authors are Amazon and NASWA staff, so it is not independent of the parties in the NASWA deployment. Validation covers education requirements only; remote work, full-time status and benefits are described but not checked against external data. The specific Bedrock model is not named in the text reviewed. The paper's wording 'tested and rejected the hypothesis that the samples come from different distributions' at p=0.13 describes a non-significant result, not proof of equivalence.","architectureImplications":"The study used one schema prompt per posting with a JSON template, while the 2026 production pipeline uses separate prompts per field on Nova Lite. Results from the paper therefore do not carry over automatically to the production design.","governanceImplications":"The authors argue that labor market information used in policy must be open to inspection and audit, unlike closed commercial data, and describe a stakeholder user group with NASWA. Agencies can hold the production release to that standard.","securityPrivacyImplications":"Postings are public with no personal data or employer identifiers analyzed, and inputs come from verified NLx employers, which the authors say lowers manipulation risk.","caveats":"Not independent (Amazon and NASWA authors); single validated field; 2023 data; unnamed model; non-significance is not proof of equivalence; production pipeline differs from the method tested.","streamIds":["aws","state-government"],"roles":{"sales":"Sales framing: use this paper as the evidence behind the NASWA story, and be clear about how far it goes. It shows one field matched survey data on a modest sample, which helps a credible conversation with state labor market analysts who will ask about method. It does not prove accuracy for pay, benefits or remote status, and its authors are from Amazon and NASWA. Interpretation: position any proposal as replicating and extending this validation on the customer's own data.","engineering":"Engineering framing: the paper documents a reproducible approach: stratified sampling, a JSON schema prompt, low temperature and comparison with an external survey. Engineers can reuse that validation design. Note that production moved to per-field prompts on a different model, so earlier results are not inherited. A proof of value should re-run the external comparison for each released field and add a human-labeled sample, since the study used none.","delivery":"Delivery framing: build validation into the plan from the start. Use the paper's survey comparison as a minimum and add labeled samples for fields that lack an external benchmark. Record model versions and prompts so results can be audited, as the authors themselves recommend. Proposed acceptance criteria below are targets for a new deployment, not results from this study. Keep a sample of human-coded postings as a standing check."},"roleDetails":{"sales":{"customerProblem":"Labor market analysts need evidence that AI-extracted posting fields are trustworthy before using them in policy work.","stakeholders":["State labor market information director","Workforce board research staff","Agency chief data officer"],"discoveryQuestions":["Which extracted fields would inform decisions, and what benchmark exists for each?","Do you require published methods before adopting derived data?","Would you join or consult NASWA's user group on these fields?"],"valueHypothesis":"A published, survey-checked method may shorten the trust-building phase for agencies adopting AI-derived posting data, if they replicate the check for fields they use.","potentialEngagement":"A short validation design review that maps each field an agency needs to an external benchmark or labeled-sample test.","unsupportedClaims":["Accuracy for pay, benefits, remote or full-time fields","Independence from Amazon or NASWA","That the 2026 production pipeline inherits these results"]},"engineering":{"fit":"Good template for validating structured extraction from public text against external statistics.","architectureIntegration":"Sampling and comparison steps sit beside the extraction pipeline as a recurring evaluation job.","prerequisites":["Access to an external benchmark such as CPS ASEC","Stratified sample design","Versioned prompts and model IDs"],"constraints":["Only one field had an external benchmark","2023 data may not reflect current postings","Model not named, limiting replication"],"security":"Public postings only; keep the evaluation datasets free of personal data if extended to other sources.","proofOfValue":["Repeat the survey comparison on current postings with the production model","Add a human-labeled sample for fields without a benchmark","Report confidence intervals, not only p-values"]},"delivery":{"work":"Stand up a recurring evaluation that checks each released field against a benchmark or labeled sample.","dependencies":["Benchmark data access","Analyst time for labeling","Stable model versions"],"ownership":"The agency's labor market research unit owns validation; the data platform team runs the evaluation jobs.","skillsAdoption":"Analysts learn sampling, labeling and basic statistical comparison for model outputs.","governanceCheckpoints":["Validation report before each field release","Re-validation after model or prompt change","Publication of method notes"],"acceptanceCriteria":["Each field has a documented validation result before release","Re-validation completed within an agreed time after any model change","Method notes published alongside the data"],"risks":["Treating non-significance as proof of accuracy","Unvalidated fields used in policy","Author affiliation questions reduce stakeholder trust"]}},"retrievedAt":"2026-11-02T14:33:07Z","enrichedAt":"2026-11-02T14:40:06Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"No workforce or accessibility effects measured; the paper's aim is broader access to labor market data for agencies without commercial subscriptions.","procurementImplications":"Supports asking vendors for published methods and validation, not only cost claims, when buying or co-developing AI-derived statistics.","operatingModelImplications":"The study needed no human annotation, but that also means no human-labeled accuracy check beyond the survey comparison.","peerReview":"peer-reviewed","sourceVerification":{"openedUrl":"https://www.amazon.science/publications/extracting-structured-labor-market-information-from-job-postings-with-generative-ai","referenceExcerpt":"As a validation, we compare frequencies of educational requirements by occupation from our sample to survey data and find no statistically significant difference.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}