FHIR TX Benchmark

Tests

Overview

This benchmark covers 20 test cases across FHIR terminology operations, exercising some of the most important capabilities a FHIR terminology server should support.

Tests are grouped by operation family. Within each family, cases vary the terminology system, input complexity, and feature set — from simple code lookups to multi-filter intensional ValueSet expansions. This spread is intentional: a server that performs well only on simple lookups might not be a good general-purpose terminology server.

Each test has a preflight phase that verifies the server returns a semantically correct response before any load is applied. Tests that fail preflight are excluded from that server’s benchmark run and marked in the capability matrix.

GroupTestsFHIR operation
FSFS01Generic FHIR search
LKLK01–LK05CodeSystem/$lookup
VCVC01–VC03ValueSet/$validate-code
EXEX01–EX08ValueSet/$expand
SSSS01CodeSystem/$subsumes
CMCM01–CM02ConceptMap/$translate

Bias

Each test carries a bias coefficient b∈(0,1]b \in (0, 1] that scales its contribution to the composite score. Bias is used to down-weight operations that are less representative of real production use-cases. We tend to give higher values to SNOMED tests than their counterparts since SNOMED is arguably the most used and important terminology in healthcare. We also prioritize hot-path operations over rarely used ones (e.g.: ValueSet/$validate-code over CodeSystem/$subsumes)

See a breakdown of the per-test bias values here

FS — FHIR Search

FS01 — FHIR Search (CodeSystem, ValueSet)

Operation: GET /ValueSet or GET /CodeSystem with query parameters

Description: Tests standard FHIR search across both ValueSet and CodeSystem resource types. Inputs are drawn from a pool of search parameter combinations including url, version, and _count.

Preflight: Searches for the administrative-gender ValueSet by URL. Passes if the response is a Bundle with at least one entry.

Preflight request
GET /ValueSet?url=http%3A%2F%2Fhl7.org%2Ffhir%2FValueSet%2Fadministrative-gender
Accept: application/fhir+json

LK — Code Lookup

LK01 — SNOMED CT lookup

Operation: GET /CodeSystem/$lookup?system=http://snomed.info/sct&code=<code>

Description: Looks up SNOMED CT codes drawn from a pool of 2,000+ codes spanning clinical findings, procedures, organisms, and more. This test serves as the normalising baseline for the composite score.

Preflight: Looks up code 73211009 (Diabetes mellitus). Passes if the response echoes the code and system.

Preflight request
GET /CodeSystem/$lookup?system=http%3A%2F%2Fsnomed.info%2Fsct&code=73211009
Accept: application/fhir+json

LK02 — LOINC lookup

Operation: GET /CodeSystem/$lookup?system=http://loinc.org&code=<code>

Description: Lookup on LOINC observation and laboratory codes. Pool covers 2,000+ codes.

Preflight: Looks up code 8867-4 (Heart rate). Passes if the response echoes the code and system.

Preflight request
GET /CodeSystem/$lookup?system=http%3A%2F%2Floinc.org&code=8867-4
Accept: application/fhir+json

LK03 — Mixed FHIR package codes

Operation: GET /CodeSystem/$lookup?system=<system>&code=<code>&version=<version>

Description: Looks up codes from a diverse set of smaller terminologies bundled in FHIR R4 core and extension packages — DICOM, ActCode, MaritalStatus, v2 tables, NCI, and others.

Preflight: Looks up DICOM code 121054 (Observer context). Passes if the response echoes the code and system.

Preflight request
GET /CodeSystem/$lookup?system=http%3A%2F%2Fdicom.nema.org%2Fresources%2Fontology%2FDCM&code=121054&version=01
Accept: application/fhir+json

LK04 — RxNorm lookup

Operation: GET /CodeSystem/$lookup?system=http://www.nlm.nih.gov/research/umls/rxnorm&code=<code>

Description: Lookup on RxNorm drug codes. Pool covers 2,000+ ingredient and product codes.

Preflight: Looks up code 161 (Acetaminophen). Passes if the response echoes the code and system.

Preflight request
GET /CodeSystem/$lookup?system=http%3A%2F%2Fwww.nlm.nih.gov%2Fresearch%2Fumls%2Frxnorm&code=161
Accept: application/fhir+json

LK05 — SNOMED negative lookup

Operation: GET /CodeSystem/$lookup?system=http://snomed.info/sct&code=<code>

Description: Inputs are fabricated SNOMED-like codes that do not exist. Verifies that the server correctly handles the not-found case and returns an appropriate OperationOutcome.

Preflight: Looks up code 999973211009 (a real code prefixed to guarantee non-existence). Passes if the response is an OperationOutcome.

Preflight request
GET /CodeSystem/$lookup?system=http%3A%2F%2Fsnomed.info%2Fsct&code=999973211009
Accept: application/fhir+json

VC — Validate Code

VC01 — SNOMED implicit ValueSet, code only

Operation: GET /ValueSet/$validate-code?url=http://snomed.info/sct?fhir_vs&system=...&code=<code>

Description: Validates SNOMED codes against the implicit "all SNOMED codes" ValueSet.

Preflight: Validates code 73211009. Passes if result = true.

Preflight request
GET /ValueSet/$validate-code?url=http%3A%2F%2Fsnomed.info%2Fsct%3Ffhir_vs&system=http%3A%2F%2Fsnomed.info%2Fsct&code=73211009
Accept: application/fhir+json

VC02 — SNOMED implicit ValueSet, code + display

Operation: GET /ValueSet/$validate-code?...&code=<code>&display=<display>

Description: Like VC01, but also validates the display string against the code's preferred term.

Preflight: Validates code 73211009 with display "Diabetes mellitus". Passes if result = true.

Preflight request
GET /ValueSet/$validate-code?url=http%3A%2F%2Fsnomed.info%2Fsct%3Ffhir_vs&system=http%3A%2F%2Fsnomed.info%2Fsct&code=73211009&display=Diabetes%20mellitus
Accept: application/fhir+json

VC03 — SNOMED is-a membership

Operation: GET /ValueSet/$validate-code?url=http://snomed.info/sct?fhir_vs=isa/<root>&code=<code>

Description: Validates a code against a hierarchy-constrained implicit ValueSet. Pool entries cover a range of root concepts from large hierarchies to small ones, mixing positive and negative cases.

Preflight: Validates that 22298006 (Myocardial infarction) is a descendant of 404684003 (Clinical finding). Passes if result = true.

Preflight request
GET /ValueSet/$validate-code?url=http%3A%2F%2Fsnomed.info%2Fsct%3Ffhir_vs%3Disa%2F404684003&system=http%3A%2F%2Fsnomed.info%2Fsct&code=22298006
Accept: application/fhir+json

EX — Expand

EX01 — SNOMED hierarchy expand with count and offset

Operation: GET /ValueSet/$expand?url=http://snomed.info/sct?fhir_vs=isa/<concept>&count=<n>&offset=<k>

Description: Expands a SNOMED is-a hierarchy slice with pagination. Pool covers 100 SNOMED hierarchies across three tiers — large, mid, and small — with count/offset combinations sized to each tier.

Preflight: Expands isa/404684003 (Clinical finding) with count=10, offset=0. Passes if exactly 10 results are returned.

Preflight request
GET /ValueSet/$expand?url=http%3A%2F%2Fsnomed.info%2Fsct%3Ffhir_vs%3Disa%2F404684003&count=10&offset=0
Accept: application/fhir+json

EX02 — SNOMED hierarchy filters

Operation: POST /ValueSet/$expand with hierarchy filters

Description: Expands an ad-hoc ValueSet constructed inline using SNOMED hierarchy filter operators (descendent-of, generalizes).

Preflight: Expands descendants of 64572001 (Disease) with count=10. Passes if exactly 10 results are returned.

Preflight request
POST /ValueSet/$expand
Content-Type: application/fhir+json
Accept: application/fhir+json

{
  "resourceType": "Parameters",
  "parameter": [
    {
      "name": "valueSet",
      "resource": {
        "resourceType": "ValueSet",
        "compose": {
          "include": [
            {
              "system": "http://snomed.info/sct",
              "filter": [
                { "property": "concept", "op": "descendent-of", "value": "64572001" }
              ]
            }
          ]
        }
      }
    },
    { "name": "count", "valueInteger": 10 }
  ]
}

EX03 — SNOMED text filter

Operation: GET /ValueSet/$expand?url=http://snomed.info/sct?fhir_vs&filter=<term>&count=<n>

Description: Searches within SNOMED using a text filter. Pool mixes high-frequency terms (many matches), selective terms (few matches), and prefix fragments.

Preflight: Filters for "diabetes". Passes if at least one result is returned.

Preflight request
GET /ValueSet/$expand?url=http%3A%2F%2Fsnomed.info%2Fsct%3Ffhir_vs&filter=diabetes&count=20
Accept: application/fhir+json

EX04 — VSAC extensional ValueSets

Operation: GET /ValueSet/$expand?url=<vsac-url>&count=<n>[&filter=<term>]

Description: Expands extensional ValueSets from the VSAC (Value Set Authority Center). Pool covers 10 curated clinical value sets with counts of 10, 100, and 1,000, plus filtered variants.

Preflight: Expands the SNOMED CT CORE Problem List (2.16.840.1.113762.1.4.1018.240) with count=10. Passes if results are returned.

Preflight request
GET /ValueSet/$expand?url=http%3A%2F%2Fcts.nlm.nih.gov%2Ffhir%2FValueSet%2F2.16.840.1.113762.1.4.1018.240&count=10
Accept: application/fhir+json

EX05 — SNOMED intensional filters

Operation: POST /ValueSet/$expand with composed ValueSet

Description: Expands intensional SNOMED ValueSets using complex filter combinations — is-a hierarchy, single property equals, AND combinations, and OR multi-include.

Preflight: Expands descendants of 195967001 (Asthma) with property filter 363698007 (Finding site) = 89187006 (Airway structure) and count=10. Passes if exactly 10 results are returned.

Preflight request
POST /ValueSet/$expand
Content-Type: application/fhir+json
Accept: application/fhir+json

{
  "resourceType": "Parameters",
  "parameter": [
    {
      "name": "valueSet",
      "resource": {
        "resourceType": "ValueSet",
        "compose": {
          "include": [
            {
              "system": "http://snomed.info/sct",
              "filter": [
                { "property": "concept",   "op": "is-a", "value": "195967001" },
                { "property": "363698007", "op": "=",    "value": "89187006"  }
              ]
            }
          ]
        }
      }
    },
    { "name": "count", "valueInteger": 10 }
  ]
}

EX06 — RxNorm intensional filters

Operation: POST /ValueSet/$expand with composed ValueSet

Description: Intensional expansions using RxNorm-specific attributes — term type (TTY), relationship properties (tradename_of, has_ingredient), and combinations.

Preflight: Expands brand-name drugs for Acetaminophen (tradename_of = 161, TTY = BN) with count=10. Passes if exactly 10 results are returned.

Preflight request
POST /ValueSet/$expand
Content-Type: application/fhir+json
Accept: application/fhir+json

{
  "resourceType": "Parameters",
  "parameter": [
    {
      "name": "valueSet",
      "resource": {
        "resourceType": "ValueSet",
        "compose": {
          "include": [
            {
              "system": "http://www.nlm.nih.gov/research/umls/rxnorm",
              "filter": [
                { "property": "TTY",         "op": "=", "value": "BN"  },
                { "property": "tradename_of", "op": "=", "value": "161" }
              ]
            }
          ]
        }
      }
    },
    { "name": "count", "valueInteger": 10 }
  ]
}

EX07 — Multi-system text filter

Operation: POST /ValueSet/$expand with an ad-hoc composed ValueSet spanning multiple code systems

Description: Expands an ad-hoc ValueSet combining SNOMED CT, LOINC, and RxNorm with a free-text filter. Small counts test server short-circuiting; larger counts force multi-system result merging.

Preflight: Expands an ad-hoc ValueSet including SNOMED CT, LOINC, and RxNorm with filter=amphetamine and count=200. Passes if results are returned.

Preflight request
POST /ValueSet/$expand
Content-Type: application/fhir+json
Accept: application/fhir+json

{
  "resourceType": "Parameters",
  "parameter": [
    {
      "name": "valueSet",
      "resource": {
        "resourceType": "ValueSet",
        "compose": {
          "include": [
            { "system": "http://snomed.info/sct" },
            { "system": "http://loinc.org" },
            { "system": "http://www.nlm.nih.gov/research/umls/rxnorm" }
          ]
        }
      }
    },
    { "name": "filter", "valueString": "amphetamine" },
    { "name": "count",  "valueInteger": 200 }
  ]
}

EX08 — SNOMED combined filters

Operation: POST /ValueSet/$expand with composed ValueSet

Description: The most complex expansion test. Each request combines three filter types simultaneously — is-a hierarchy, a SNOMED property equals filter, and a free-text display filter.

Preflight: Expands Clinical findings (isa/404684003) with associated morphology = fracture (116676008 = 72704001) and text filter "fracture", count=10. Passes if exactly 10 results are returned.

Preflight request
POST /ValueSet/$expand
Content-Type: application/fhir+json
Accept: application/fhir+json

{
  "resourceType": "Parameters",
  "parameter": [
    {
      "name": "valueSet",
      "resource": {
        "resourceType": "ValueSet",
        "compose": {
          "include": [
            {
              "system": "http://snomed.info/sct",
              "filter": [
                { "property": "concept",   "op": "is-a", "value": "404684003" },
                { "property": "116676008", "op": "=",    "value": "72704001"  }
              ]
            }
          ]
        }
      }
    },
    { "name": "filter", "valueString": "fracture" },
    { "name": "count",  "valueInteger": 10 }
  ]
}

SS — Subsumes

SS01 — SNOMED hierarchy subsumption

Operation: GET /CodeSystem/$subsumes?system=http://snomed.info/sct&codeA=<a>&codeB=<b>

Description: Tests hierarchy introspection via the $subsumes operation. Pool covers all four FHIR subsumption outcomes — subsumes, subsumed-by, equivalent, and not-subsumed — across concepts at different hierarchy depths.

Preflight: Tests that 64572001 (Disease) subsumes 6142004 (Influenza). Passes if outcome = subsumes.

Preflight request
GET /CodeSystem/$subsumes?system=http%3A%2F%2Fsnomed.info%2Fsct&codeA=64572001&codeB=6142004
Accept: application/fhir+json

CM — Translate

CM01 — SNOMED implicit REPLACED BY map

Operation: GET /ConceptMap/$translate?url=http://snomed.info/sct?fhir_cm=900000000000526001&...

Description: Translates retired SNOMED codes to their replacement codes using SNOMED's implicit REPLACED BY ConceptMap.

Preflight: Translates code 225983005 (Duodenal ulcer NOS → replaced by Duodenal ulcer). Passes if result = true.

Preflight request
GET /ConceptMap/$translate?url=http%3A%2F%2Fsnomed.info%2Fsct%3Ffhir_cm%3D900000000000526001&system=http%3A%2F%2Fsnomed.info%2Fsct&code=225983005&target=http%3A%2F%2Fsnomed.info%2Fsct
Accept: application/fhir+json

CM02 — FHIR package ConceptMaps

Operation: GET /ConceptMap/$translate?url=<map-url>&system=<src>&code=<code>&target=<tgt>

Description: Translates codes using ConceptMaps bundled in FHIR R4 core and IPS packages — FHIR-to-v2/v3 crosswalks, status maps, LOINC↔SNOMED mappings, and more.

Preflight: Translates FHIR male to v3 Administrative Gender. Passes if result = true.

Preflight request
GET /ConceptMap/$translate?url=http%3A%2F%2Fhl7.org%2Ffhir%2FConceptMap%2Fcm-administrative-gender-v3&system=http%3A%2F%2Fhl7.org%2Ffhir%2Fadministrative-gender&code=male&target=http%3A%2F%2Fterminology.hl7.org%2FCodeSystem%2Fv3-AdministrativeGender
Accept: application/fhir+json