Florian Kurpicz
Presentation at ALENEX 2017

Engineering a Distributed Full-Text Index

Today, I presented our distributed full-text index at ALENEX 2017. It is based on Patricia tries, and we have implemented different (succinct) trie representations. The slides are available as handout and heavily animated. The paper is available here (DOI) and here (arXiv).

Slides in Handout Mode

Slide image alenex_2017_slides_handout-01.jpg
Slide image alenex_2017_slides_handout-02.jpg
Slide image alenex_2017_slides_handout-03.jpg
Slide image alenex_2017_slides_handout-04.jpg
Slide image alenex_2017_slides_handout-05.jpg
Slide image alenex_2017_slides_handout-06.jpg
Slide image alenex_2017_slides_handout-07.jpg
Slide image alenex_2017_slides_handout-08.jpg
Slide image alenex_2017_slides_handout-09.jpg
Slide image alenex_2017_slides_handout-10.jpg
Slide image alenex_2017_slides_handout-11.jpg
Slide image alenex_2017_slides_handout-12.jpg
Slide image alenex_2017_slides_handout-13.jpg

Abstract

We present a distributed full-text index for big data applications in a distributed environment. Our index can answer different types of pattern matching queries (existential, counting and enumeration). We perform experiments on inputs up to 100 GiB using up to 512 processors, and compare our index with the distributed suffix array by Arroyuelo et al. [Parall. Comput. 40(9): 471–495, 2014]. The result is that our index answers counting queries up to 5:5 times faster than the distributed suffix array, while using about the same space. We also provide a succinct variant of our index that uses only one third of the memory compared with our non-succinct variant, at the expense of only 20% slower query times.