Enriching Contextualized Representations with Biomedical Ontologies: Extending KnowBert to UMLS

Guilhem Piat (1) , Nasredine Semmar (1) , Alexandre Allauzen (2) , Hassane Essafi (3) , Julien Tourille (1)

1 LASTI - Laboratoire Analyse Sémantique Textes et Images
2 LAMSADE - Laboratoire d'analyse et modélisation de systèmes pour l'aide à la décision
3 DIASI (CEA, LIST) - Département Intelligence Ambiante et Systèmes Interactifs

Currently, biomedical document processing is mostly human work. Software solutions which attempt to alleviate this burden exist but generally do not perform well enough to be helpful in many applications. Concurrently, there exist projects which organize concepts in the biomedical field. Therefore, we seek to leverage existing structured knowledge resources to improve biomedical language modeling. In this paper, we provide an implementation integrating the UMLS knowledgebase into a BERT-based language model, aiming to improve its performance in biomedical Named Entity Recognition. To achieve this, we extend KnowBert, a recently developed technique for integrating knowledge into language models. Preliminary results reveal the challenges of applying KnowBert to the biomedical domain given the number and subtlety of different concepts in UMLS. Going forward, addressing these challenges and combining this with other approaches such as BioBERT may help expand the range of usefully automatable biomedical language processing tasks.

Keywords: Artificial neural networks; Knowledge based systems; Knowledge representation; Machine learning; Biomedical informatics; Information Extraction

Domains: Computation and Language [cs.CL]

Guilhem Piat, Nasredine Semmar, Alexandre Allauzen, Hassane Essafi, Julien Tourille. Enriching Contextualized Representations with Biomedical Ontologies: Extending KnowBert to UMLS. Intelligent Computing Proceedings of the 2022 Computing Conference, Jul 2022, Londres, United Kingdom. pp.760-773, ⟨10.1007/978-3-031-10464-0_52⟩. ⟨cea-04563039⟩