Round 0

2026-03-25

Apple M3, 8-core · 24 GB · macOS Tahoe 26.3 · Docker Desktop 4.64.0 (8 CPU / 20 GB)

	FS01 FHIR Search (CodeSystem, ValueSet) Tests standard FHIR search across both ValueSet and CodeSystem resource types. Inputs are drawn from a pool of search parameter combinations including url, version, and _count.	LK01 SNOMED CT lookup Looks up SNOMED CT codes drawn from a pool of 2,000+ codes spanning clinical findings, procedures, organisms, and more. This test serves as the normalising baseline for the composite score.	LK02 LOINC lookup Lookup on LOINC observation and laboratory codes. Pool covers 2,000+ codes.	LK03 Mixed FHIR package codes Looks up codes from a diverse set of smaller terminologies bundled in FHIR R4 core and extension packages — DICOM, ActCode, MaritalStatus, v2 tables, NCI, and others.	LK04 RxNorm lookup Lookup on RxNorm drug codes. Pool covers 2,000+ ingredient and product codes.	LK05 SNOMED negative lookup Inputs are fabricated SNOMED-like codes that do not exist. Verifies that the server correctly handles the not-found case and returns an appropriate OperationOutcome.	VC01 SNOMED implicit ValueSet, code only Validates SNOMED codes against the implicit "all SNOMED codes" ValueSet.	VC02 SNOMED implicit ValueSet, code + display Like VC01, but also validates the display string against the code's preferred term.	VC03 SNOMED is-a membership Validates a code against a hierarchy-constrained implicit ValueSet. Pool entries cover a range of root concepts from large hierarchies to small ones, mixing positive and negative cases.	EX01 SNOMED hierarchy expand with count and offset Expands a SNOMED is-a hierarchy slice with pagination. Pool covers 100 SNOMED hierarchies across three tiers — large, mid, and small — with count/offset combinations sized to each tier.	EX02 SNOMED hierarchy filters Expands an ad-hoc ValueSet constructed inline using SNOMED hierarchy filter operators (descendent-of, generalizes).	EX03 SNOMED text filter Searches within SNOMED using a text filter. Pool mixes high-frequency terms (many matches), selective terms (few matches), and prefix fragments.	EX04 VSAC extensional ValueSets Expands extensional ValueSets from the VSAC (Value Set Authority Center). Pool covers 10 curated clinical value sets with counts of 10, 100, and 1,000, plus filtered variants.	EX05 SNOMED intensional filters Expands intensional SNOMED ValueSets using complex filter combinations — is-a hierarchy, single property equals, AND combinations, and OR multi-include.	EX06 RxNorm intensional filters Intensional expansions using RxNorm-specific attributes — term type (TTY), relationship properties (tradename_of, has_ingredient), and combinations.	EX07 Multi-system text filter Expands an ad-hoc ValueSet combining SNOMED CT, LOINC, and RxNorm with a free-text filter. Small counts test server short-circuiting; larger counts force multi-system result merging.	EX08 SNOMED combined filters The most complex expansion test. Each request combines three filter types simultaneously — is-a hierarchy, a SNOMED property equals filter, and a free-text display filter.	SS01 SNOMED hierarchy subsumption Tests hierarchy introspection via the $subsumes operation. Pool covers all four FHIR subsumption outcomes — subsumes, subsumed-by, equivalent, and not-subsumed — across concepts at different hierarchy depths.	CM01 SNOMED implicit REPLACED BY map Translates retired SNOMED codes to their replacement codes using SNOMED's implicit REPLACED BY ConceptMap.	CM02 FHIR package ConceptMaps Translates codes using ConceptMaps bundled in FHIR R4 core and IPS packages — FHIR-to-v2/v3 crosswalks, status maps, LOINC↔SNOMED mappings, and more.
termbox	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	~	~
fhirsmith	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✗	✗	✗	✗	✓	✗	✗
ontoserver	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
pascal	✗	✓	✓	✓	✓	✗	✓	✓	✓	✓	✗	✓	✓	✗	✗	✗	✗	✓	✗	✗
hermes	~	✓	✗	✗	✗	✗	~	~	~	✓	~	~	~	~	~	~	~	~	~	~
snowstorm	✗	✓	~	~	~	✓	✓	✓	✓	✓	✗	✓	~	✗	~	~	✗	✓	~	~

	FS01 FHIR Search (CodeSystem, ValueSet) Tests standard FHIR search across both ValueSet and CodeSystem resource types. Inputs are drawn from a pool of search parameter combinations including url, version, and _count.	LK01 SNOMED CT lookup Looks up SNOMED CT codes drawn from a pool of 2,000+ codes spanning clinical findings, procedures, organisms, and more. This test serves as the normalising baseline for the composite score.	LK02 LOINC lookup Lookup on LOINC observation and laboratory codes. Pool covers 2,000+ codes.	LK03 Mixed FHIR package codes Looks up codes from a diverse set of smaller terminologies bundled in FHIR R4 core and extension packages — DICOM, ActCode, MaritalStatus, v2 tables, NCI, and others.	LK04 RxNorm lookup Lookup on RxNorm drug codes. Pool covers 2,000+ ingredient and product codes.	LK05 SNOMED negative lookup Inputs are fabricated SNOMED-like codes that do not exist. Verifies that the server correctly handles the not-found case and returns an appropriate OperationOutcome.	VC01 SNOMED implicit ValueSet, code only Validates SNOMED codes against the implicit "all SNOMED codes" ValueSet.	VC02 SNOMED implicit ValueSet, code + display Like VC01, but also validates the display string against the code's preferred term.	VC03 SNOMED is-a membership Validates a code against a hierarchy-constrained implicit ValueSet. Pool entries cover a range of root concepts from large hierarchies to small ones, mixing positive and negative cases.	EX01 SNOMED hierarchy expand with count and offset Expands a SNOMED is-a hierarchy slice with pagination. Pool covers 100 SNOMED hierarchies across three tiers — large, mid, and small — with count/offset combinations sized to each tier.	EX02 SNOMED hierarchy filters Expands an ad-hoc ValueSet constructed inline using SNOMED hierarchy filter operators (descendent-of, generalizes).	EX03 SNOMED text filter Searches within SNOMED using a text filter. Pool mixes high-frequency terms (many matches), selective terms (few matches), and prefix fragments.	EX04 VSAC extensional ValueSets Expands extensional ValueSets from the VSAC (Value Set Authority Center). Pool covers 10 curated clinical value sets with counts of 10, 100, and 1,000, plus filtered variants.	EX05 SNOMED intensional filters Expands intensional SNOMED ValueSets using complex filter combinations — is-a hierarchy, single property equals, AND combinations, and OR multi-include.	EX06 RxNorm intensional filters Intensional expansions using RxNorm-specific attributes — term type (TTY), relationship properties (tradename_of, has_ingredient), and combinations.	EX07 Multi-system text filter Expands an ad-hoc ValueSet combining SNOMED CT, LOINC, and RxNorm with a free-text filter. Small counts test server short-circuiting; larger counts force multi-system result merging.	EX08 SNOMED combined filters The most complex expansion test. Each request combines three filter types simultaneously — is-a hierarchy, a SNOMED property equals filter, and a free-text display filter.	SS01 SNOMED hierarchy subsumption Tests hierarchy introspection via the $subsumes operation. Pool covers all four FHIR subsumption outcomes — subsumes, subsumed-by, equivalent, and not-subsumed — across concepts at different hierarchy depths.	CM01 SNOMED implicit REPLACED BY map Translates retired SNOMED codes to their replacement codes using SNOMED's implicit REPLACED BY ConceptMap.	CM02 FHIR package ConceptMaps Translates codes using ConceptMaps bundled in FHIR R4 core and IPS packages — FHIR-to-v2/v3 crosswalks, status maps, LOINC↔SNOMED mappings, and more.
termbox	25ms	11ms	11ms	8.2ms	8.0ms	9.4ms	11ms	11ms	12ms	327ms	90ms	290ms	366ms	69ms	68ms	111ms	51ms	9.4ms	—	—
fhirsmith	361ms	17ms	50ms	11ms	32ms	12ms	14ms	16ms	55ms	424ms	193ms	44.96s	1.58s	—	—	—	—	11ms	—	—
ontoserver	51.43s	298ms	118ms	73ms	116ms	274ms	300ms	292ms	290ms	825ms	377ms	9.88s	29.14s	1.49s	789ms	4.49s	13.84s	1.52s	1.94s	109ms
pascal	—	170ms	1.06s	77ms	73ms	—	73ms	87ms	84ms	31.93s	—	30.55s	14.56s	—	—	—	—	44ms	—	—
hermes	—	62ms	—	—	—	—	—	—	—	55.01s	—	—	—	—	—	—	—	—	—	—
snowstorm	—	4.76s	—	—	—	378ms	730ms	785ms	703ms	6.35s	—	5.38s	—	—	—	—	—	4.18s	—	—