# Blog de Frédéric

## Tag - mathml

Saturday, August 15 2015

## MathML Accessibility (part II)

As announced in a previous blog post, I was invited to two Mozilla Work Weeks in Toronto and Whistler during the month of June. Before these work weeks, the only assistive technology able to read MathML in Gecko-based browsers was NVDA, via the help of the third-party MathPlayer plugin developed by Design Science, as shown in the following video:

Thanks to the effort done during these work weeks plus some additional days, we have made good progress to expose MathML via accessibility APIs on other platforms: Mac OS X, Linux, Android and Firefox OS. Note that Firefox for iOS uses WebKit, so MathML should already be exposed and handled via Apple's WebKit/VoiceOver. If you are not familiar with accessibility APIs (and actually even if you are), I recommend you to read Marco Zehe's excellent blog post about why accessibility APIs matter.

Apple was the first company to rely on accessibility APIs to make MathML accessible: WebKit exposes MathML via its NSAccessibility protocol and it can then be handled by the VoiceOver assistive technology. One of the obvious consequence of working with open standards and open accessibility APIs is that it was then relatively easy for us to make MathML accessible on Mac OS X: We basically just read the WebKit source code to verify how MathML is exposed and did the same for Gecko. The following video shows VoiceOver reading a Wikipedia page with MathML mode enabled in Gecko 41:

Of course, one of the disadvantage is that VoiceOver is proprietary and so we are dependent on what Apple actually implements for MathML and we can not easily propose patches to fix bugs or add support for new languages. This is however still more convenient for users than the proprietary MathPlayer plugin used by NVDA: at least VoiceOver is installed by default on Apple's products and well-integrated into their user & accessibility interfaces. For instance, I was able to use the standard user interface to select the French language in VoiceOver and it worked immediately. For NVDA+MathPlayer, there are several configuration menus (one for the Windows system, one for NVDA and one for MathPlayer) and even after selecting French everywhere and rebooting, the math formulas were still read in English...

The next desktop platform we worked on was Linux. We continued to improve how Gecko expose the MathML via the ATK interface but the most important work was done by Joanmarie Diggs: making Orca able to handle the exposed MathML accessibility tree. Compared to the previous solutions, this one is 100% open and I was happy to be able to submit a couple of patches to Orca and to work with the Gnome Translation Team to keep the French translation up-to-date. By the way, if you are willing to contribute to the localization of Orca into your language feel free to join the Gnome Translation Project, help will be much appreciated! The following video shows how Orca reads the previous Wikipedia page in Nightly builds:

On mobile platforms (Android and Firefox OS) we use a common Javascript layer called AccessFu to handle Gecko's internal accessibility tree. So all of this is handled by Mozilla and hence is also 100% open. As I said in my previous blog post, I was not really aware of the internal details before the Work Weeks so it was good to get more explanations and help from Yura Zenevich. Although we were able to do some preliminary work to add MathML support to AccessFu in bug 1163374, this will definitely need further improvements. So I will not provide any demo for now :-)

To conclude this overview, you can check the status of accessibility on the Mozilla MathML Project page. This page also contains a table of MathML tests and how they are handled on the various platforms. At the end of September, I will travel to Toronto to participate to the Mozilla and FOSS Assistive Technology Meetup. In particular, I hope to continue improvements to MathML Accessibility in Mozilla products... Stay tuned!

Wednesday, May 6 2015

## Overview

In a previous blog post about MathML in Wikipedia, I mentioned that, despite ongoing efforts there was still no accessibility support for MathML in Gecko. The situation changed two months ago: Design Science and NV Access released new versions of MathPlayer and NVDA, which in particular add MathML accessibility support on Windows, as shown in the demos below. This is exciting news and I am really willing to see this support extended to other platforms...

Last December, I also met Joanmarie Diggs at the Web Engines Hackfest and we have been able to start some preliminary work for Linux (WebKit/Gecko/Orca). I had the opportunity to refresh some of the patches written by Jonathan Wei during a Mozilla internship and to get part of his work landed into trunk. I have also made basic improvements to how we expose the accessible tree for ATK in order to prepare future support in Orca. It is certainly too early to announce anything. Just as a comparison, I also provide how Orca currently (badly) reads the MathML formulas below.

MathML accessibility support is also available in the latest versions of Safari+VoiceOver. So in theory, we "only" need to make Gecko expose the same accessible tree as WebKit in order to support the Mac platform. Jonathan Wei had a work-in-progress patch for that, see bug 1001641. Since it is far from being ready, I will cheat a bit and just show how VoiceOver reads the MathML examples in Safari.

Finally, the mobile platforms (Firefox OS and Android) are also very important. So far, I have only submitted some patches to make the GeckoView accessible and to fix some other small accessibility bugs. So I am interested in hearing more from Mozilla developers about the AccessFu stuff and how we could make MathML accessible on these platforms.

## Audio Demos

The table below contains some concrete examples taken from Wikipedia (in MathML mode), Mozilla Developer Network, KaTeX and MathJax. Note that at the moment, MathJax MathML formulas are not exposed to all assistive technologies. I recommend to force native MathML using an add-on for Gecko browsers or Safari ; or to use this GreaseMonkey script.

MathML Example NVDA VoiceOver Orca (no support)
Pythagorean theorem (Wikipedia)
Real roots of a quadratic equation (Wikipedia)
Math.cosh from ECMAScript 6 (MDN)
Polyfill for Math.atanh from ECMAScript 6 (MDN)
Matrix of SVG's rotate transform (MDN)

For the record, the detailed software versions used are:

• NVDA: Windows 7, Firefox 37, NVDA try-t4673-10720,5202e97, MathPlayer 4.0 beta 1
• VoiceOver: OS X Yosemite, Safari 8, VoiceOver 4
• Orca: Debian GNU/Linux 8 (jessie), Gnome 3.14.1, Firefox 31, Orca 3.14.0

## June Schedule

All of these developments are still in progress and there are certainly many bugs to fix and improvements to do. Next month, I expect to have several opportunities to meet people and make progress on MathML. For people interested to help, here is my schedule:

• On the 8th, I'll attend the 9th European e-Accessibility Forum in Paris where Neil Soiffer (Design Science) will present "The State of Accessible Math".
• From the 11th to 22th, I'll stay in the Montréal area. I hope I'll have the chance to meet Joanmarie and the accessibility team from Mozilla Toronto during that time frame (to be confirmed).
• Last but not least, I'm one of the lucky Mozilla volunteer invited to Whistler's work week from the 22th to 26th!

Friday, October 24 2014

## A quick note for Mozillians regarding MathML on Wikipedia

As mentioned some time ago and as recently announced on the MathML and MediaWiki mailing lists, a MathML mode with SVG/PNG fallback is now available on Wikipedia. In order to test it, you need to log in with a Wikipedia account and select the mode in the "Math" section of your preferences.

Some quick notes for Mozillians:

• Although Mozilla intern Jonathan Wei has done some work on MathML accessibility and that there are reports about work in progress to make Firefox work with NVDA / Orca / VoiceOver, we unfortunately still don't have something ready for Gecko browsers. You can instead try the existing solutions for Safari or Internet Explorer (ChromeVox and JAWS 16 beta are supposed to be MathML-aware but fail to read the MathML on Wikipedia at the moment).

• By default, the following MATH fonts are tried: Cambria Math, Latin Modern Math, STIX Math, Latin Modern Math (Web font). In my opinion, our support for Cambria Math (installed by default on Windows) is still not very good, so I'd recommend to use Latin Modern Math instead, which has the same "Computer Modern" style as the current PNG mode. To do that, go to the "Skin" section of your preferences and just add the rule math { font-family: Latin Modern Math; } to your "Custom CSS". Latin Modern Math is installed with most LaTeX distributions, available from the GUST website and provided by the MathML font add-on.

• You can actually install various fonts and try to make the size and style of the math font consistent with the surrounding text. Here are some examples:

/* Asana Math (Palatino style) */
.mw-body, mtext {
font-family: Palatino Linotype, URW Palladio L, Asana Math;
}
math {
font-family: Asana Math;
}

/* Cambria (Microsoft Office style) */
.mw-body, mtext {
font-family: Cambria;
}
math {
font-family: Cambria Math;
}

/* Latin Modern (Computer Modern style) */
.mw-body, mtext {
font-family: Latin Modern Roman;
}
math {
font-family: Latin Modern Math;
}

/* STIX/XITS (Times New Roman style) */
.mw-body, mtext {
font-family: XITS, STIX;
}
math {
font-family: XITS Math, STIX Math;
}

/* TeX Gyre Bonum (Bookman style) */
.mw-body, mtext {
font-family: TeX Gyre Bonum;
}
math {
font-family: TeX Gyre Bonum Math;
}

/* TeX Gyre Pagella (Palatino style) */
.mw-body, mtext {
font-family: TeX Gyre Pagella;
}
math {
font-family: TeX Gyre Pagella Math;
}

/* TeX Gyre Schola (Century Schoolbook style) */
.mw-body, mtext {
font-family: TeX Gyre Schola;
}
math {
font-family: TeX Gyre Schola Math;
}

/* TeX Gyre Termes (Times New Roman style) */
.mw-body, mtext {
font-family: TeX Gyre Termes;
}
math {
font-family: TeX Gyre Termes Math;
}

• We still have bugs with missing fonts and font inflation on mobile devices. If you are affected by these bugs, you can force the SVG fallback instead:

span.mwe-math-mathml-inline, div.mwe-math-mathml-display {
display: none !important;
}
span.mwe-math-mathml-inline + .mwe-math-fallback-image-inline {
display: inline !important;
}
div.mwe-math-mathml-display + .mwe-math-fallback-image-display {
display: block !important;
}

• You might want to install some Firefox add-ons for copying MathML/LaTeX, zooming formulas or configuring the math font.

• Finally, don't forget to report bugs to Bugzilla so that volunteers can continue to improve our MathML support. Thank you!

Wednesday, June 18 2014

Four years ago I started to write some MathML add-ons using Jetpack 0.8, now called Add-on SDK. I've recently made progress on this project, so that all the initial features are now available as Firefox add-ons (my initial hope was that the Add-on SDK would eventually be compatible with all Gecko browsers but unfortunately that still does not seem to be the case at the moment). The Mathzilla collection is available on AMO but some of the add-ons are still undergoing review. Here is an overview:

• The math editor feature is now provided by the TeXZilla add-on. The Arabic math support I experimented a bit later is also available.

• The conversion of content MathML using David Carlisle's XSLT stylesheet is now in its own MathML-ctop add-on. There is another similar add-on to add MathML3 features missing in Gecko called MathML-mml3ff. Note that these add-ons do not rely on the Add-on SDK and will work in any Gecko browsers. However, they should probably be improved.

• Another add-on that does not rely on the Add-on SDK is the one adding mathematical fonts called MathML-fonts. I uploaded version 2.0 to use the new OpenType MATH fonts supported in Gecko 31, but I hope that it will no longer be necessary in the future (more on this later).

• The conversion of PNG images into MathML is now provided by the Image to MathML add-on. At the moment, it is still experimental, see the details on mozilla.dev.tech.mathml if you want to help. It only works for some Web sites using LaTeX in alt text but I wish I can find a solution for Wolfram Websites.

• Since many Web sites are using MathJax and because in the meantime MathJax moved to its slow HTML-CSS output by default I had to write an add-on to force MathJax to use native MathML, which is available here. Actually, it's even better since it disables the mml2jax preprocessor to avoid useless work by MathJax for Web sites that already use MathML in the source code. It also prevents the MathJax menu to override the browser user interface (note that the three add-ons below provide some UI features similar to what one can find in MathJax).

• The feature to copy a MathML formula is now provided by the MathML Copy add-on. Note that it actually copies two flavors (text and html). It is also possible to copy the original TeX source when it is provided (e.g. on MDN).

• A new MathML Zoom add-on provides a zooming feature similar to what MathJax does.

• A new MathML Font Settings add-on allows to configure font-family and font-size of mathematics similar to what MathJax provides. Note however, that the list of font-family choices in the context menu is based on the OpenType MATH fonts that will only be supported in Gecko 31.

I believe splitting the original Mathzilla add-on into many add-ons gives more flexibility to let people choose the desired features. As usual, help to localize the add-ons is very welcome.

Tuesday, June 3 2014

## TeXZilla 0.9.7 Released

Today the Mozilla MathML team released a new version of TeXZilla. You can download a release package or install it with npm. We fixed a few bugs, but there are known issues due to errors in the unicode.xml file of XML Entity Definitions for Characters or inherited from the itex2MML grammar that does not make it ready for version 1.0. The main improvements in this new release are enhancements to the public API and to the command line interface.

TeXZilla can now be used as a stream filter. Each TeX expressions delimited by the classical $...$, $$...$$, $...$ and $$...$$ will be converted into inline or display MathML. Outside these delimiters, you can use \$ and \\ as escaped characters. We offer three ways to apply that stream filter: • From the command line, in a UNIX pipeline: cat foo-tex.html | phantomjs TeXZilla.js streamfilter > foo-mathml.html echo "This is a **Markdown** document with a *math formula*:$ z = \\sqrt{x^2 + y^2} $" | markdown | nodejs TeXZilla.js streamfilter | sed '1s/^/\n<!-- HTML5 document -->\n/' (note: this is not yet supported by slimerjs) • Using the TeXZilla.filterString(aString) function, for example TeXZilla.filterString("blah$x^2$blah") will return the filtered string. • Using the TeXZilla.filterElement(aElement) function. This one will browse recursively the descendants of the DOM element aElement and the stream filter will be applied to the text leaves. By introducting these TeXZilla.filter* function, it becomes tempting to use TeXZilla the same way as MathJax, that is to process all the text nodes in your Web pages and to filter the TeX strings. This is not the intended goal of TeXZilla and it is strongly discouraged: not only the MathML content won't appear in crawlers (e.g. search engines or feed readers) but also browsing all the DOM elements and appending new ones can be very slow for large documents. Instead, it is recommend to filter your static Web page with commonJS TeXZilla.js streamfilter before publishing it or to use a server-side conversion for example using the Web server mode. There are situations where you do not have other choice, though. In that case try to reduce as much as possible the number of elements being processed (see the example in the next section). Of course, if you do not care about performance and MathML availibility outside your web site, you can just use MathJax. ## New Safe and Itex-Identifier parsing modes The most notable difference between TeXZilla and itex2MML is the handling of some expressions like $xy$ or $Func\$. By default, TeXZilla interprets this as individual MathML identifiers <mi>x</mi><mi>y</mi> (so that as in LaTeX, they will render in italic) while itex2MML interprets this as a single indentifier <mi>Func</mi>. It is now possible to configure TeXZilla to align with itex2MML's behavior. To do that, use TeXZilla.setItexIdentifierMode or pass the appropriate boolean to the command line. Consecutive non-basic letters (like Greek or Arabic) are still treated as individual tokens. With that change, we hope that TeXZilla could be used to parse all the commands supported by itex2MML into an equivalent output. Together with the command line stream filter, this should allow to recover all the nice itex2MML features.

Similarly, a safe mode is now available and can be enabled with TeXZilla.setSafeMode or by passing the appropriate boolean to the command line. This mode will forbid commands that could be used for XSS injections like \href. With that mode and the new TeXZilla.filterElement function, I'm now able to remove MathJax's use from my blog (users of browsers without good MathML support can still enable it or choose the lighter mathml.css stylesheet). MathJax was a bit overkill for my blog since I'm only parsing visitor comments. To illustrate how the setSafeMode and filterString functions can be used, I now just have to do

// Process TeX fragments in blog comments and comment preview.
TeXZilla.setSafeMode(true);
var toProcess =
document.querySelectorAll("#comments > dl > dd, #comment-form dd.comment-preview");
for (var i = 0; i < toProcess.length; i++) {
TeXZilla.filterElement(toProcess[i]);
}
});


## Inserting equations in a 2D/WebGL canvas

The new function TeXZilla.toImage has been introduced to convert a TeX fragment into a math HTML image with a base64-encoded src attribute. Contrary to other functions of the API, this one needs to do some work to determine the image size and perform the conversion, so it is unlikely to work as expected in a non-browser context. The goal is really only to have a convenient function to generate image of mathematical formulas and insert them into a canvas context to draw 2D or 3D scientific schemas. At the moment, this works well only in Gecko. For instance,

var image =
TeXZilla.toImage("\\vec{F} = G \\frac{m_1 m_2}{r^2} \\mathbf{u}");
canvas.getContext("2d").drawImage(image,
(canvas.width - image.width) / 2,
(canvas.height - image.height) / 2);
}


will insert a mathematical formula in the middle of a 2D canvas. Similarly, you can insert a mathematical formula as a texture in a WebGL canvas. It is recommended to pass aRoundToPowerOfTwo=true to TeXZilla.toImage, so that the image will have dimensions that are power of two. Note that the mathematical formula will be automatically centered in the middle of the generated image. See this example for how to setup the formulas with three.js and make them always oriented in the direction of the camera.

## Integration in Mozilla products

• The CKeditor editor plugin is now integrated in MDN, so you can click on the square root logo in the editor toolbar to insert mathematical formulas. By the way, the mathml.css is now used for browsers without MathML support. See for example the pages for acosh, atanh or CSS transform.

• The editor/ in comm-central now integrates a small input box to insert mathematical formulas, accessible from the Insert menu. This will be available in Thunderbird 31 and Seamonkey 2.28, so that you can write mathematics in your emails and in the WYSIWYG editors.

• Various FirefoxOS Web math apps have been written and use TeXZilla. Raniere is also working on a math keyboard for FirefoxOS as a GSoC project, which will allow to type mathematics faster on mobile devices.

- page 1 of 8