Blog de Frédéric

To content | To menu | To search

Tag - mozilla

Entries feed - Comments feed

Sunday, December 20 2015

MathML at the Web Engines Hackfest 2015

Hackfest

Two weeks ago, I travelled to Spain to participate to the second Web Engines Hackfest which was sponsored by Igalia and Collabora. Such an event has been organized by Igalia since 2009 and used to be focused on WebkitGTK+. It is great to see that it has now been extended to any Web engines & platforms and that a large percentage of non-igalian developers has been involved this year. If you did not get the opportunity to attend this event or if you are curious about what happened there, take a look at the wiki page or flickr album.

Last day of the hackfest
Photo from @webengineshackfest licensed under Creative Commons Attribution-ShareAlike

I really like this kind of hacking-oriented and participant-driven event where developers can meet face to face, organize themselves in small collaboration groups to efficiently make progress on a task or give passionate talk about what they have recently been working on. The only small bemol I have is that it is still mainly focused on WebKit/Blink developments. Probably, the lack of Mozilla/Microsoft participants is probably due to Mozilla Coincidental Work Weeks happening at the same period and to the proprietary nature of EdgeHTML (although this is changing?). However, I am confident that Igalia will try and address this issue and I am looking forward to coming back next year!

MathML developments

This year, Igalia developer Alejandro G. Castro wanted to work with me on WebKit's MathML layout code and more specifically on his MathML refactoring branch. Indeed, as many people (including Google developers) who have tried to work on WebKit's code in the past, he arrived to the conclusion that the WebKit's MathML layout code has many design issues that make it a burden for the rest of the layout team and too complex to allow future improvements. I was quite excited about the work he has done with Javier Fernández to try and move to a simple box model closer to what exists in Gecko and thus I actually extended my stay to work one whole week with them. We already submitted our proposal to the webkit-dev mailing list and received positive feedback, so we will now start merging what is ready. At the end of the week, we were quite satisfied about the new approach and confident it will facilitate future maintenance and developements :-)

Main room
Photo from @webengineshackfest licensed under Creative Commons Attribution-ShareAlike

While reading a recent thread on the Math WG mailing list, I realized that many MathML people have only vague understanding of why Google (or to be more accurate, the 3 or 4 engineers who really spent some time reviewing and testing the WebKit code) considered the implementation to be unsafe and not ready for release. Even worse, Michael Kholhase pointed out that for someone totally ignorant of the technical implementation details, the communication made some years ago around the "flexbox-based approach" gave the impression that it was "the right way" (indeed, it somewhat did improve the initial implementation) and the rationale to change that approach was not obvious. So let's just try and give a quick overview of the main problems, even if I doubt someone can have good understanding of the situation without diving into the C++ code:

  1. WebKit's code to stretch operator was not efficient at all and was limited to some basic fences buildable via Unicode characters.
  2. WebKit's MathML code violated many layout invariants, making the code unreliable.
  3. WebKit's MathML code relied heavily on the C++ renderer classes for flexboxes and has to manage too many anonymous renderers.

The main security concerns were addressed a long time ago by Martin Robinson and me. Glyph assembly for stretchy operators are now drawn using low-level font painting primitive instead of creating one renderer object for each piece and the preferred width for them no longer depends on vertical metrics (although we still need some work to obtain Gecko's good operator spacing). Also, during my crowdfunding project, I implemented partial support for the OpenType MATH table in WebKit and more specifically the MathVariant subtable, which allows to directly use construction of stretchy operators specified by the font designer and not only the few Unicode constructions.

However, the MathML layout code still modifies the renderer tree to force the presence of anonymous renderers and still applies specific CSS rules to them. It is also spending too much time trying to adapt the parent flexbox renderer class which has at the same time too much features for what is needed for MathML (essentially automatic box alignment) and not enough to get exact placement and measuring needed for high-quality rendering (the TeXBook rules are more complex, taking into account various parameters for box shifts, drops, gaps etc).

During the hackfest, we started to rewrite a clean implementation of some MathML renderer classes similar to Gecko's one and based on the MathML in HTML5 implementation note. The code now becomes very simple and understandable. It can be summarized into four main functions. For instance, to draw a fraction we need:

  • computePreferredLogicalWidths which sets the preferred width of the fraction during the first layout pass, by considering the widest between numerator and denominator.
  • layoutBlock and firstLineBaseline which calculate the final width/height/ascent of the fraction element and position the numerator and denominator.
  • paint which draws the fraction bar.

Perhaps, the best example to illustrate how the complexity has been reduced is the case of the renderer of mmultiscripts/msub/msup/msubsup elements (attaching an arbitrary number of subscripts and superscripts before or after a base). In the current WebKit implementation, we have to create three anonymous wrappers (a first one for the base, a second one for prescripts and a third one for postscripts) and an anonymous wrapper for each subscript/superscript pair, add alignment styles for these wrappers and spend a lot of time maintaining the integrity of the renderer tree when dynamic changes happen. With the new code, we just need to do arithmetic calculations to position the base and script boxes. This is somewhat more complex than the fraction example above but still, it remains arithmetic calculations and we can not reduce any further if we wish quality comparable to TeXBook / MATH rules. We actually take into account many parameters from the OpenType MATH table to get much better placement of scripts. We were able to fix bug 130325 in about twenty minutes instead of fighting with a CSS "negative margin" hack on anonymous renderers.

MathML dicussions

The first day of the hackfest we also had an interesting "breakout session" to define the tasks to work on during the hackfest. Alejandro briefly presented the status of his refactoring branch and his plan for the hackfest. As said in the previous section, we have been quite successful in following this plan: Although it is not fully complete yet, we expect to merge the current changes soon. Dominik Röttsches who works on Blink's font and layout modules was present at the MathML session and it was a good opportunity to discuss the future of MathML in Chrome. I gave him some references regarding the OpenType MATH table, math fonts and the MathML in HTML5 implementation note. Dominik said he will forward the references to his colleagues working on layout so that we can have deeper technical dicussion about MathML in Blink in the long term. Also, I mentioned noto-fonts issue 330, which will be important for math rendering in Android and actually does not depend on the MathML issue, so that's certainly something we could focus on in the short term.

Álex and Fred during the MathML breakout session
Photo from @webengineshackfest licensed under Creative Commons Attribution-ShareAlike

Alejandro also proposed to me to prepare a talk about MathML in Web Engines and exciting stuff happening with the MathML Association. I thus gave a brief overview of MathML and presented some demos of native support in Gecko. I also explained how we are trying to start a constructive approach to solve miscommunication between users, spec authors and implementers ; and gather technical and financial resources to obtain a proper solution. In a slightly more technical part, I presented Microsoft's OpenType MATH table and how it can be used for math rendering (and MathML in particular). Finally, I proposed my personal roadmap for MathML in Web engines. Although I do not believe I am a really great speaker, I received positive feedback from attendees. One of the thing I realized is that I do not know anything about the status and plan in EdgeHTML and so overlooked to mention it in my presentation. Its proprietary nature makes hard for external people to contribute to a MathML implementation and the fact that Microsoft is moving away from ActiveX de facto excludes third-party plugin for good and fast math rendering in the future. After I went back to Paris, I thus wrote to Microsoft employee Christian Heilmann (previously working for Mozilla), mentioning at least the MathML in HTML5 Implementation Note and its test suite as a starting point. MathML is currently on the first page of the most voted feature requested for Microsoft Edge and given the new direction taken with Microsoft Edge, I hope we could start a discussion on MathML in EdgeHTML...

Conclusion

This was a great hackfest and I'd like to thank again all the participants and sponsors for making it possible! As Alejandro wrote to me, "I think we have started a very interesting work regarding MathML and web engines in the future.". The short term plan is now to land the WebKit MathML refactoring started during the hackfest and to finish the work. I hope people now understand the importance of fonts with an OpenType MATH table for good mathematical rendering and we will continue to encourage browser vendors and font designers to make such fonts wide spread.

The new approach for WebKit MathML support gives good reason to be optmimistic in the long term and we hope we will be able to get high-quality rendering. The fact that the new approach addresses all the issues formulated by Google and that we started a dialogue on math rendering, gives several options for MathML in Blink. It remains to get Microsoft involved in implementing its own OpenType table in EdgeHTML. Overall, I believe that we can now have a very constructive discussion with the individuals/companies who really want native math support, with the Math WG members willing to have their specification implemented in browsers and with the browser vendors who want a math implementation which is good, clean and compatible with the rest of their code base. Hopefully, the MathML Association will be instrumental in achieving that. If everybody get involved, 2016 will definitely be an exciting year for native MathML in Web engines!

Friday, October 16 2015

Open Font Format 3 released: Are browser vendors good at math?

Version 3 of the Open Font Format was officially published as ISO standard early this month. One of the interesting new feature is that Microsoft's MATH table has been integrated into this official specification. Hopefully, this will encourage type designers to create more math fonts and OS vendors to integrate them into their systems. But are browser vendors ready to use Open Font Format for native MathML rendering? Here is a table of important Open Font Format features for math rendering and (to my knowledge) the current status in Apple, Google, Microsoft and Mozilla products.

FeatureRationaleAppleGoogleMicrosoftMozilla
Pre-installed math fontsMake mathematical rendering possible with the default system fonts.OSX: Obsolete STIX
iOS: no
Android: no
Chrome OS: no
Windows: Cambria Math
Windows phone: no?
Firefox OS: no
MATH table allowed in Web fontsWorkaround the lack of pre-installed math fonts or let authors provide custom math style.WebKit: yes (no font sanitizer?)Blink: yes (OTS)Trident: yes (no font sanitizer?)Gecko: yes (OTS)
USE_TYPO_METRICS OS/2 fsSelection flag taken into accountMath fonts contain tall glyphs (e.g. integrals in display style) and so using the "typo" metrics avoids excessive line spacing for the math text.WebKit: noBlink: yesTrident: yesGecko: yes (gfx/)
Open Font Format FeaturesGood mathematical rendering requires some glyph substitutions (e.g. ssty, flac and dtls).WebKit: yesBlink: yesTrident: yesGecko: yes
Ability to parse the MATH tableGood mathematical rendering requires many font data.WebKit: yes (WebCore/platform/graphics/)Blink: noTrident: yes (LineServices)Gecko (gfx/)
Using the MATH table for native MathML renderingThe MathML specification does not provide detailed rules for mathematical rendering.WebKit: for operator stretching (WebCore/rendering/mathml/)Blink: noTrident: noGecko: yes (layout/mathml/)
Total Score:4/63/64.5/65/6

update: Daniel Cater provided a list of pre-installed fonts on Chrome OS stable, confirming that no fonts with a MATH table are available.

Saturday, August 15 2015

MathML Accessibility (part II)

As announced in a previous blog post, I was invited to two Mozilla Work Weeks in Toronto and Whistler during the month of June. Before these work weeks, the only assistive technology able to read MathML in Gecko-based browsers was NVDA, via the help of the third-party MathPlayer plugin developed by Design Science, as shown in the following video:

Thanks to the effort done during these work weeks plus some additional days, we have made good progress to expose MathML via accessibility APIs on other platforms: Mac OS X, Linux, Android and Firefox OS. Note that Firefox for iOS uses WebKit, so MathML should already be exposed and handled via Apple's WebKit/VoiceOver. If you are not familiar with accessibility APIs (and actually even if you are), I recommend you to read Marco Zehe's excellent blog post about why accessibility APIs matter.

Apple was the first company to rely on accessibility APIs to make MathML accessible: WebKit exposes MathML via its NSAccessibility protocol and it can then be handled by the VoiceOver assistive technology. One of the obvious consequence of working with open standards and open accessibility APIs is that it was then relatively easy for us to make MathML accessible on Mac OS X: We basically just read the WebKit source code to verify how MathML is exposed and did the same for Gecko. The following video shows VoiceOver reading a Wikipedia page with MathML mode enabled in Gecko 41:

Of course, one of the disadvantage is that VoiceOver is proprietary and so we are dependent on what Apple actually implements for MathML and we can not easily propose patches to fix bugs or add support for new languages. This is however still more convenient for users than the proprietary MathPlayer plugin used by NVDA: at least VoiceOver is installed by default on Apple's products and well-integrated into their user & accessibility interfaces. For instance, I was able to use the standard user interface to select the French language in VoiceOver and it worked immediately. For NVDA+MathPlayer, there are several configuration menus (one for the Windows system, one for NVDA and one for MathPlayer) and even after selecting French everywhere and rebooting, the math formulas were still read in English...

The next desktop platform we worked on was Linux. We continued to improve how Gecko expose the MathML via the ATK interface but the most important work was done by Joanmarie Diggs: making Orca able to handle the exposed MathML accessibility tree. Compared to the previous solutions, this one is 100% open and I was happy to be able to submit a couple of patches to Orca and to work with the Gnome Translation Team to keep the French translation up-to-date. By the way, if you are willing to contribute to the localization of Orca into your language feel free to join the Gnome Translation Project, help will be much appreciated! The following video shows how Orca reads the previous Wikipedia page in Nightly builds:

On mobile platforms (Android and Firefox OS) we use a common Javascript layer called AccessFu to handle Gecko's internal accessibility tree. So all of this is handled by Mozilla and hence is also 100% open. As I said in my previous blog post, I was not really aware of the internal details before the Work Weeks so it was good to get more explanations and help from Yura Zenevich. Although we were able to do some preliminary work to add MathML support to AccessFu in bug 1163374, this will definitely need further improvements. So I will not provide any demo for now :-)

To conclude this overview, you can check the status of accessibility on the Mozilla MathML Project page. This page also contains a table of MathML tests and how they are handled on the various platforms. At the end of September, I will travel to Toronto to participate to the Mozilla and FOSS Assistive Technology Meetup. In particular, I hope to continue improvements to MathML Accessibility in Mozilla products... Stay tuned!

Wednesday, May 6 2015

MathML Accessibility

Overview

In a previous blog post about MathML in Wikipedia, I mentioned that, despite ongoing efforts there was still no accessibility support for MathML in Gecko. The situation changed two months ago: Design Science and NV Access released new versions of MathPlayer and NVDA, which in particular add MathML accessibility support on Windows, as shown in the demos below. This is exciting news and I am really willing to see this support extended to other platforms...

Last December, I also met Joanmarie Diggs at the Web Engines Hackfest and we have been able to start some preliminary work for Linux (WebKit/Gecko/Orca). I had the opportunity to refresh some of the patches written by Jonathan Wei during a Mozilla internship and to get part of his work landed into trunk. I have also made basic improvements to how we expose the accessible tree for ATK in order to prepare future support in Orca. It is certainly too early to announce anything. Just as a comparison, I also provide how Orca currently (badly) reads the MathML formulas below.

MathML accessibility support is also available in the latest versions of Safari+VoiceOver. So in theory, we "only" need to make Gecko expose the same accessible tree as WebKit in order to support the Mac platform. Jonathan Wei had a work-in-progress patch for that, see bug 1001641. Since it is far from being ready, I will cheat a bit and just show how VoiceOver reads the MathML examples in Safari.

Finally, the mobile platforms (Firefox OS and Android) are also very important. So far, I have only submitted some patches to make the GeckoView accessible and to fix some other small accessibility bugs. So I am interested in hearing more from Mozilla developers about the AccessFu stuff and how we could make MathML accessible on these platforms.

Audio Demos

The table below contains some concrete examples taken from Wikipedia (in MathML mode), Mozilla Developer Network, KaTeX and MathJax. Note that at the moment, MathJax MathML formulas are not exposed to all assistive technologies. I recommend to force native MathML using an add-on for Gecko browsers or Safari ; or to use this GreaseMonkey script.

MathML Example NVDA VoiceOver Orca (no support)
Pythagorean theorem (Wikipedia)
Real roots of a quadratic equation (Wikipedia)
Math.cosh from ECMAScript 6 (MDN)
Polyfill for Math.atanh from ECMAScript 6 (MDN)  
Matrix of SVG's rotate transform (MDN)
Two other formulas (KaTeX home page)
Another Formula (MathJax home page)

For the record, the detailed software versions used are:

  • NVDA: Windows 7, Firefox 37, NVDA try-t4673-10720,5202e97, MathPlayer 4.0 beta 1
  • VoiceOver: OS X Yosemite, Safari 8, VoiceOver 4
  • Orca: Debian GNU/Linux 8 (jessie), Gnome 3.14.1, Firefox 31, Orca 3.14.0

Note: all these audio files are distributed under the Creative Commons Attribution-Share Alike 4.0 International license.

June Schedule

All of these developments are still in progress and there are certainly many bugs to fix and improvements to do. Next month, I expect to have several opportunities to meet people and make progress on MathML. For people interested to help, here is my schedule:

  • On the 8th, I'll attend the 9th European e-Accessibility Forum in Paris where Neil Soiffer (Design Science) will present "The State of Accessible Math".
  • From the 11th to 22th, I'll stay in the Montréal area. I hope I'll have the chance to meet Joanmarie and the accessibility team from Mozilla Toronto during that time frame (to be confirmed).
  • Last but not least, I'm one of the lucky Mozilla volunteer invited to Whistler's work week from the 22th to 26th!

Friday, October 24 2014

A quick note for Mozillians regarding MathML on Wikipedia

As mentioned some time ago and as recently announced on the MathML and MediaWiki mailing lists, a MathML mode with SVG/PNG fallback is now available on Wikipedia. In order to test it, you need to log in with a Wikipedia account and select the mode in the "Math" section of your preferences.

Zoomed-Windows8-Firefox32-MathML-LatinModern

Some quick notes for Mozillians:

  • Although Mozilla intern Jonathan Wei has done some work on MathML accessibility and that there are reports about work in progress to make Firefox work with NVDA / Orca / VoiceOver, we unfortunately still don't have something ready for Gecko browsers. You can instead try the existing solutions for Safari or Internet Explorer (ChromeVox and JAWS 16 beta are supposed to be MathML-aware but fail to read the MathML on Wikipedia at the moment).

  • By default, the following MATH fonts are tried: Cambria Math, Latin Modern Math, STIX Math, Latin Modern Math (Web font). In my opinion, our support for Cambria Math (installed by default on Windows) is still not very good, so I'd recommend to use Latin Modern Math instead, which has the same "Computer Modern" style as the current PNG mode. To do that, go to the "Skin" section of your preferences and just add the rule math { font-family: Latin Modern Math; } to your "Custom CSS". Latin Modern Math is installed with most LaTeX distributions, available from the GUST website and provided by the MathML font add-on.

  • You can actually install various fonts and try to make the size and style of the math font consistent with the surrounding text. Here are some examples:

    /* Asana Math (Palatino style) */
    .mw-body, mtext {
        font-family: Palatino Linotype, URW Palladio L, Asana Math;
    }
    math {
        font-family: Asana Math;
    }
    
    
    /* Cambria (Microsoft Office style) */
    .mw-body, mtext {
        font-family: Cambria;
    }
    math {
        font-family: Cambria Math;
    }
    
    
    /* Latin Modern (Computer Modern style) */
    .mw-body, mtext {
        font-family: Latin Modern Roman;
    }
    math {
        font-family: Latin Modern Math;
    }
    
    
    /* STIX/XITS (Times New Roman style) */
    .mw-body, mtext {
        font-family: XITS, STIX;
    }
    math {
        font-family: XITS Math, STIX Math;
    }
    
    
    /* TeX Gyre Bonum (Bookman style) */
    .mw-body, mtext {
        font-family: TeX Gyre Bonum;
    }
    math {
        font-family: TeX Gyre Bonum Math;
    }
    
    
    /* TeX Gyre Pagella (Palatino style) */
    .mw-body, mtext {
        font-family: TeX Gyre Pagella;
    }
    math {
        font-family: TeX Gyre Pagella Math;
    }
    
    
    /* TeX Gyre Schola (Century Schoolbook style) */
    .mw-body, mtext {
        font-family: TeX Gyre Schola;
    }
    math {
        font-family: TeX Gyre Schola Math;
    }
    
    
    /* TeX Gyre Termes (Times New Roman style) */
    .mw-body, mtext {
        font-family: TeX Gyre Termes;
    }
    math {
        font-family: TeX Gyre Termes Math;
    }
    
  • We still have bugs with missing fonts and font inflation on mobile devices. If you are affected by these bugs, you can force the SVG fallback instead:

    span.mwe-math-mathml-inline, div.mwe-math-mathml-display {
      display: none !important;
    }
    span.mwe-math-mathml-inline + .mwe-math-fallback-image-inline {
      display: inline !important;
    }
    div.mwe-math-mathml-display + .mwe-math-fallback-image-display {
      display: block !important;
    }
    
  • You might want to install some Firefox add-ons for copying MathML/LaTeX, zooming formulas or configuring the math font.

  • Finally, don't forget to report bugs to Bugzilla so that volunteers can continue to improve our MathML support. Thank you!

- page 1 of 9