The review presents and compares three open source and web-based text collation tools, namely Juxta Web Service, LERA, and Variance Viewer, and investigates their suitability especially for philological work with TEI. While all tools adequately fulfill the general requirements of text collation, each of them supports individual workflow concepts and visualization methods. It is recommended that future developments consider supporting TEI standoff markup and tool modularization.
Have you ever needed to collate two copies or editions of a historical document in order to quickly and reliably identify the differences between them? In digital scholarly editions, variant readings are often encoded manually by human editors. Tools for automatic text collation do exist, but most of them are either difficult to use, or they address specific use cases. As such, there is a need for easy-to-use, accessible, and adaptable tools which can handle textual variance and text alignment, including support for TEI-XML markup. In other words, some tools exist, but do they really fit the philological and technical requirements of researchers?
This article starts with a quick overview of text collation in
philology and computer science. It goes on to develop a suitable review method
and analyses three TEI-interoperable, web based collation tools: Juxta Web Service
This review focuses on
Text comparison means
Often used in authoring environments, text comparison generally
follows functional purposes, such as when collaborative writing teams
utilize it to revert unwanted or ill-intended modifications, in teaching and
academic writing in order to check for plagiarism, or in code development to
avoid conflicting changes. However, another use case of text comparison
exists that aims at a better understanding of cultural processes. Scholars
from all historical disciplines have an interest in retracing the creation
and transmission of texts in order to establish connections or divergences
between them. In textual criticism, the epistemic process of text comparison
is called
The following two sections engage with divergent understandings
of
Scheme of descent of the manuscripts of
Pseudo-Apuleius Herbarius, in: Ernst Howald and Henry E.
Sigerist: Antonii Musae De herba vettonica liber, Leipzig
1927.
Example of a critical apparatus, as in Die
Metaphysik des Aristoteles, ed. by Albert Schwegler, Tübingen
1847, p. 1;
Editors need to establish clear guidelines to define the phenomena of interest for future users of an edition and those that will be explicitly disregarded during the collation. This is also an economic decision: in many cases, tracking of graphemic variance (e.g. æ/ä) or regular orthography (e.g. color/colour) requires immense additional resources. Even if such variants are potentially interesting for later studies, the required human effort is often not justifiable. Furthermore, the more details have to be tracked, the more human errors might increase.
There are various conventional methods to present the
results of a given text collation in printed editions, while digitally
based presentations are still in development. Independently from the
media, classical scholarly editions prefer a
Digital tools can help. First, automatic collation can be significantly faster, if all witnesses are already fully transcribed. Philologists could focus their attention on semantics and leave the tedious letter-by-letter-collation to the machine. Or, argued from a positive perspective, the time and energy saved by automatic collation can be fully used for interpreting and refining the results.
Second, an automatic collation process would always produce the same result, thus is more reliable than a human produced collation. Furthermore, during the process of collating, humans naturally tend to produce individual interpretations and need to make assumptions on a writer’s or scribe’s possible intentions (Plachta 2013: 115–121). Instead, computers do not interpret, unless instructed to, but focus exclusively on comparing the texts. These two very different competences – precision on the one hand and interpretation on the other – could be utilized to divide the process into an automated collation layer and a human interpretation layer.
English Bible Variants of Genesis 1,1,
visualized by BibleViz.
Analyzing differences between texts implies awareness about
which type of variation can actually occur. Given two sequences A and B,
the two basic cases of textual difference can be described as:
Two more complex categories of variation are:
Identifying substitution and transposition is less trivial than additions and deletions (for an analysis on the transposition problem, see Schöch 2016). Both substitution and transposition can also be interpreted as successive additions and deletions, and this ambiguity often makes it difficult to decide what the original intention actually was. It is a philologist’s task to produce a well-founded and weighted interpretation of a scribe’s possible intentions or error’s causes. Here lies a watershed between the different procedures for identifying substitutions or transpositions in philology and computer science: a textual scholar decides by personal knowledge and experience, an algorithm by formal models.
The first algorithms for text comparison were invented in the 1960s. The most popular example is called Levenshtein Distance (Levenshtein 1966) and expresses textual difference as ‘edit distance’, which is the calculus of the minimal number of characters that need to be added, deleted or substituted to transform sequence A into sequence B. In the 1970s, it was the Hunt-McIlroy-Algorithm which solved the longest common subsequence problem, and revolutionized file comparison when it was implemented for the Unix command ‘diff’ (Hunt and Mcllroy 1976), capable of identifying and describing differences line wise (not only character wise) in human and machine readable format. Larry Walls invented ‘patch’ in the 1980s, a Unix program capable of reverting changes by saving the difference in a separate file (“Patch” 2019). These algorithms or optimized variations of them have been in use until today.
Programming linguists and philologists have implemented
collation tools since the 1960s. Collating long texts turned out to be a
complex scenario, with some major developments around 1990 (Spadini
2016: 112–116). In 2009, a working group presented an abstract framework
for complex text collation procedures, which was later called
‘Gothenburg model’
The following questions should be considered before starting
the collation procedure:
@rend.
On the one hand, the review aims at pointing out the individual strengths and specific qualities of each of the three presented tools. On the other hand, the review should also cover general aspects systematically. To this aim, a short list of basic features that helps to conduct this task, obligatory and non-obligatory, is presented first. All tools were tested with the same examples.
The list of requirements contains both
Although the selected tools were all intended to work as generic solutions, they have been developed from individual starting points or specific methodological perspectives on text analysis. Furthermore, there is a great variety of possible use cases – diversified by languages, epochs, genres, individual styles – which are too manifold to be adequately represented within the scope of this review. For this review, examples were chosen which are capable of demonstrating the general functionalities of the tools, while the suitability for specific use cases needs to be tested by the individual users themselves.
Two sets of texts have been used to test the tools:
Juxta (http://www.juxtacommons.org/, tested version: 1.8.3-BETA) is a
text collation software produced by the University of Virginia in 2009. It
was developed into the web application ‘Juxta Web Service’ by the Networked
Infrastructure for Nineteenth-Century Electronic Scholarship (NINES)
Juxta Web Service requires an account which can be created online free of charge with a few clicks. The collation procedure follows three steps: First, the user uploads two or more files that he/she wishes to collate and that will appear as ‘sources’ in the dashboard. Juxta WS accepts a broad range of input formats, such as TXT, XML, HTML, DOC, RTF, PDF and EPUB. The documents can also be retrieved from a URL or pasted into a text field (and, as a special feature, it is even possible to refer to Wikipedia articles and their revisions). If desired, source files can be reformatted (there is also an XML indent function) and edited directly in the browser, and saved as a new source. Secondly, each source needs to be prepared as ‘witness’. This means that a distinct name needs to be assigned to each source, while the text is being tokenized automatically in the background. The whole process becomes transparent and can also be modified in the ‘XML View’, which displays the XSLT transformation templates. For example, Juxta WS omits all elements which are not on the TEI block level (e.g. highlights) by default, unless this behavior is changed. Finally, the user selects two or more witnesses to form a ‘comparison set’. For the collation process, the user can define how punctuation, character case, hyphenation and line breaks should be handled.
Example for Juxta Web Service’s Heat Map.
Example for Juxta Web Service’s Side-by-side
View’.
Example for Juxta Web Service’s TEI view and
export.
Juxta Web Service’s in-depth and thoughtfully developed features
– although some of them remained in the experimental stage – make it a
powerful tool. It offers a well-written documentation
LERA (https://lera.uzi.uni-halle.de/, tested version: 1.1.0.21) is an
acronym for: ‘Locate, Explore, Retrace and Apprehend complex text variants’.
It is a text collation software which was developed into a web based tool by
Marcus Pöckelmann. It is part of the project SADA
LERA can be tried online with a few sample texts. An individual instance with full functionality is available on personal request. After login, the user can upload two or more ‘documents’ for collation. During this procedure, the user assigns a siglum and a title to each document, and optionally sets language, segmentation method, hyphenation handling, and linguistic annotation. LERA works smoothly with XML, and all settings can be changed at different stages of the process. In a second phase, the user selects a set of documents for collation, which will then be displayed as ‘edition’.
LERA, synoptic view with CATview (upper section) and
highlighted differences.
LERA, synoptic view with identic variants
highlighted.
LERA is an impressively coherent suite of tools for text alignment and collation which allows the user to flexibly combine tools and parameters for individual use cases. Two things are still on the wish list: TEI export (or another structured format like JSON) and a public code repository. The project is likely to be maintained, as it is essential for at least one ongoing large research project.
Variance Viewer (http://variance-viewer.informatik.uni-wuerzburg.de/Variance-Viewer/,
tested version as of June 2019) is a web application for text collation
which was developed by Nico Balbach at the Department for Artificial
Intelligence and Applied Computer Science of the University of Würzburg
(Germany) in 2018. The idea for its development was initiated by a group of
researchers from an academic working group for digital editions
Variance Viewer can be used without an account. The user can
directly upload (exactly) two files to collate on a simple one page dialogue
website. Accepted formats are only XML (by default) and TXT (as fallback).
The user can upload a configuration file with customized settings (the
GitHub documentation page<p> and
<head>; all contents outside of these elements will
be excluded. The configuration file also allows the user to define
normalization rules for graphemic variance, for punctuation and
abbreviations.
Variance Viewer, synoptic output.
hi/@rend), and separation (whitespace), and visualizes
these with a color code (see Fig.
9). Color shades make clear whether there is only one different
character in a token (light) or more than one (dark). Each feature can be
switched on and off by using respective buttons, and it is possible to adapt
the visualization of the results by using CSS properties. This makes
Variance Viewer an excellent tool for visual analysis.
Variance Viewer, collation of typographic features
(other deactivated).
@rend-attributes. While other collation tools only work on
plain text level, Variance Viewer analyzes typographic aspects even if the
underlying text is identical (see Fig.
10). Currently, this is implemented for the TEI attribute
@rend only, but could potentially be extended to other
attributes.
The result is downloadable in TEI/XML format, with variants
encoded in parallel segmentation, using elements <lem>
and <rdg> for variants in the first and second text
respectively, while the variant classification is recorded into the
attribute @type of the <app> element. The
TEI apparatus needs a manual check for some minor whitespace issues. Other
available formats are JSON (according to a data scheme for another tool
called Athen
Variance Viewer does an excellent job in handling the TEI input.
The configuration options are powerful and make Variance Viewer an excellent
generic tool. On the output side, it must be mentioned that the downloadable
TEI document is not perfectly schema valid in case a variant occurs within
an element that does not allow <app> as child, e.g.
within <locus>. This is probably not problematic for most
use cases, but still needs to be taken into account. The tool is well-suited
for inclusion in a workflow at a relatively early stage, e.g. after
transcription and base encoding are completed, but before deeper semantic
tagging takes place.
According to the requirements, all tools provided a web interface for document upload (feature 1) and starting a collation procedure (feature 2), and all of them offer options for individual configurations. The tools’ approaches are very different in this respect: While LERA and Juxta Web Service both offer extremely granular interfaces, Variance Viewer offers high flexibility through an uploadable configuration file. Performance with large portions of text is adequate, but the tools cause heavy load on the client side, as they all load the complete collation result into the browser (instead of smaller portions).
All tools were able to identify additions, deletions and substitutions correctly (feature 3), while transposition is obviously an interpretative issue and needs further analysis or manual editing, as supported by LERA.
Furthermore, all tools offer a parallel synopsis view with variants highlighted (feature 4). Juxta Web Service and LERA both offer a helpful exploration tool for easy navigation through longer texts with Histogram and CATview. Concerning analysis, Variance Viewer has not only developed an interesting approach to classify variants automatically, but it can also detect presentational variants, which is most useful for collating texts with complex typography.
Concerning output formats (feature 5), there is still much to be achieved. Although a schema valid TEI output is available in Juxta Web Service and Variance Viewer, the methods used to structure collation results in XML are very diverse. In each, case it will be necessary to revise the TEI code and adopt it to one’s own practice. The same applies to other output formats, especially presentational formats, but none of the tools offers options to configure PDF and HTML, so that the usefulness of these routines is questionable.
It is positive that the source code of all tools is available (or planned to be) on public repositories (feature 6), so projects have a chance to review or reuse the code and to customize it for their own purposes. Usage of the tools is relatively easy and free of charge, as long as no special implementations are required (feature 7). Concerning accessibility, Variance Viewer follows an interesting lightweight concept, as it does not require any user management nor authentication, while LERA and Juxta Web Service require individual accounts and bind users to their web service.
The most debatable aspect of the TEI output routines is that all
tools offer only @xml:id on each token, or
corresponding milestones. A TEI apparatus based on one of these methods
would create a much more reliable encoding and make further presentation
routines easier to process (as discussed by Cayless et al. 2019). An
advantage of parallel encoding is that it is easier to manually encode it in
standard XML editors.
Concerning visualization and analysis, it should be mentioned
that there are other tools which could cover this function independently. To
give an example, TEICat
For the workflow of an edition, it is not only important to
decide which tool suits the individual requirements, but also to decide the