Blog de Frédéric

To content | To menu | To search

Tag - webkit

Entries feed - Comments feed

Sunday, December 20 2015

MathML at the Web Engines Hackfest 2015

Hackfest

Two weeks ago, I travelled to Spain to participate to the second Web Engines Hackfest which was sponsored by Igalia and Collabora. Such an event has been organized by Igalia since 2009 and used to be focused on WebkitGTK+. It is great to see that it has now been extended to any Web engines & platforms and that a large percentage of non-igalian developers has been involved this year. If you did not get the opportunity to attend this event or if you are curious about what happened there, take a look at the wiki page or flickr album.

Last day of the hackfest
Photo from @webengineshackfest licensed under Creative Commons Attribution-ShareAlike

I really like this kind of hacking-oriented and participant-driven event where developers can meet face to face, organize themselves in small collaboration groups to efficiently make progress on a task or give passionate talk about what they have recently been working on. The only small bemol I have is that it is still mainly focused on WebKit/Blink developments. Probably, the lack of Mozilla/Microsoft participants is probably due to Mozilla Coincidental Work Weeks happening at the same period and to the proprietary nature of EdgeHTML (although this is changing?). However, I am confident that Igalia will try and address this issue and I am looking forward to coming back next year!

MathML developments

This year, Igalia developer Alejandro G. Castro wanted to work with me on WebKit's MathML layout code and more specifically on his MathML refactoring branch. Indeed, as many people (including Google developers) who have tried to work on WebKit's code in the past, he arrived to the conclusion that the WebKit's MathML layout code has many design issues that make it a burden for the rest of the layout team and too complex to allow future improvements. I was quite excited about the work he has done with Javier Fernández to try and move to a simple box model closer to what exists in Gecko and thus I actually extended my stay to work one whole week with them. We already submitted our proposal to the webkit-dev mailing list and received positive feedback, so we will now start merging what is ready. At the end of the week, we were quite satisfied about the new approach and confident it will facilitate future maintenance and developements :-)

Main room
Photo from @webengineshackfest licensed under Creative Commons Attribution-ShareAlike

While reading a recent thread on the Math WG mailing list, I realized that many MathML people have only vague understanding of why Google (or to be more accurate, the 3 or 4 engineers who really spent some time reviewing and testing the WebKit code) considered the implementation to be unsafe and not ready for release. Even worse, Michael Kholhase pointed out that for someone totally ignorant of the technical implementation details, the communication made some years ago around the "flexbox-based approach" gave the impression that it was "the right way" (indeed, it somewhat did improve the initial implementation) and the rationale to change that approach was not obvious. So let's just try and give a quick overview of the main problems, even if I doubt someone can have good understanding of the situation without diving into the C++ code:

  1. WebKit's code to stretch operator was not efficient at all and was limited to some basic fences buildable via Unicode characters.
  2. WebKit's MathML code violated many layout invariants, making the code unreliable.
  3. WebKit's MathML code relied heavily on the C++ renderer classes for flexboxes and has to manage too many anonymous renderers.

The main security concerns were addressed a long time ago by Martin Robinson and me. Glyph assembly for stretchy operators are now drawn using low-level font painting primitive instead of creating one renderer object for each piece and the preferred width for them no longer depends on vertical metrics (although we still need some work to obtain Gecko's good operator spacing). Also, during my crowdfunding project, I implemented partial support for the OpenType MATH table in WebKit and more specifically the MathVariant subtable, which allows to directly use construction of stretchy operators specified by the font designer and not only the few Unicode constructions.

However, the MathML layout code still modifies the renderer tree to force the presence of anonymous renderers and still applies specific CSS rules to them. It is also spending too much time trying to adapt the parent flexbox renderer class which has at the same time too much features for what is needed for MathML (essentially automatic box alignment) and not enough to get exact placement and measuring needed for high-quality rendering (the TeXBook rules are more complex, taking into account various parameters for box shifts, drops, gaps etc).

During the hackfest, we started to rewrite a clean implementation of some MathML renderer classes similar to Gecko's one and based on the MathML in HTML5 implementation note. The code now becomes very simple and understandable. It can be summarized into four main functions. For instance, to draw a fraction we need:

  • computePreferredLogicalWidths which sets the preferred width of the fraction during the first layout pass, by considering the widest between numerator and denominator.
  • layoutBlock and firstLineBaseline which calculate the final width/height/ascent of the fraction element and position the numerator and denominator.
  • paint which draws the fraction bar.

Perhaps, the best example to illustrate how the complexity has been reduced is the case of the renderer of mmultiscripts/msub/msup/msubsup elements (attaching an arbitrary number of subscripts and superscripts before or after a base). In the current WebKit implementation, we have to create three anonymous wrappers (a first one for the base, a second one for prescripts and a third one for postscripts) and an anonymous wrapper for each subscript/superscript pair, add alignment styles for these wrappers and spend a lot of time maintaining the integrity of the renderer tree when dynamic changes happen. With the new code, we just need to do arithmetic calculations to position the base and script boxes. This is somewhat more complex than the fraction example above but still, it remains arithmetic calculations and we can not reduce any further if we wish quality comparable to TeXBook / MATH rules. We actually take into account many parameters from the OpenType MATH table to get much better placement of scripts. We were able to fix bug 130325 in about twenty minutes instead of fighting with a CSS "negative margin" hack on anonymous renderers.

MathML dicussions

The first day of the hackfest we also had an interesting "breakout session" to define the tasks to work on during the hackfest. Alejandro briefly presented the status of his refactoring branch and his plan for the hackfest. As said in the previous section, we have been quite successful in following this plan: Although it is not fully complete yet, we expect to merge the current changes soon. Dominik Röttsches who works on Blink's font and layout modules was present at the MathML session and it was a good opportunity to discuss the future of MathML in Chrome. I gave him some references regarding the OpenType MATH table, math fonts and the MathML in HTML5 implementation note. Dominik said he will forward the references to his colleagues working on layout so that we can have deeper technical dicussion about MathML in Blink in the long term. Also, I mentioned noto-fonts issue 330, which will be important for math rendering in Android and actually does not depend on the MathML issue, so that's certainly something we could focus on in the short term.

Álex and Fred during the MathML breakout session
Photo from @webengineshackfest licensed under Creative Commons Attribution-ShareAlike

Alejandro also proposed to me to prepare a talk about MathML in Web Engines and exciting stuff happening with the MathML Association. I thus gave a brief overview of MathML and presented some demos of native support in Gecko. I also explained how we are trying to start a constructive approach to solve miscommunication between users, spec authors and implementers ; and gather technical and financial resources to obtain a proper solution. In a slightly more technical part, I presented Microsoft's OpenType MATH table and how it can be used for math rendering (and MathML in particular). Finally, I proposed my personal roadmap for MathML in Web engines. Although I do not believe I am a really great speaker, I received positive feedback from attendees. One of the thing I realized is that I do not know anything about the status and plan in EdgeHTML and so overlooked to mention it in my presentation. Its proprietary nature makes hard for external people to contribute to a MathML implementation and the fact that Microsoft is moving away from ActiveX de facto excludes third-party plugin for good and fast math rendering in the future. After I went back to Paris, I thus wrote to Microsoft employee Christian Heilmann (previously working for Mozilla), mentioning at least the MathML in HTML5 Implementation Note and its test suite as a starting point. MathML is currently on the first page of the most voted feature requested for Microsoft Edge and given the new direction taken with Microsoft Edge, I hope we could start a discussion on MathML in EdgeHTML...

Conclusion

This was a great hackfest and I'd like to thank again all the participants and sponsors for making it possible! As Alejandro wrote to me, "I think we have started a very interesting work regarding MathML and web engines in the future.". The short term plan is now to land the WebKit MathML refactoring started during the hackfest and to finish the work. I hope people now understand the importance of fonts with an OpenType MATH table for good mathematical rendering and we will continue to encourage browser vendors and font designers to make such fonts wide spread.

The new approach for WebKit MathML support gives good reason to be optmimistic in the long term and we hope we will be able to get high-quality rendering. The fact that the new approach addresses all the issues formulated by Google and that we started a dialogue on math rendering, gives several options for MathML in Blink. It remains to get Microsoft involved in implementing its own OpenType table in EdgeHTML. Overall, I believe that we can now have a very constructive discussion with the individuals/companies who really want native math support, with the Math WG members willing to have their specification implemented in browsers and with the browser vendors who want a math implementation which is good, clean and compatible with the rest of their code base. Hopefully, the MathML Association will be instrumental in achieving that. If everybody get involved, 2016 will definitely be an exciting year for native MathML in Web engines!

Sunday, January 5 2014

Funding MathML Developments in Gecko and WebKit (part 2)

As I mentioned three months ago, I wanted to start a crowdfunding campaign so that I can have more time to devote to MathML developments in browsers and (at least for Mozilla) continue to mentor volunteer contributors. Rather than doing several crowdfunding campaigns for small features, I finally decided to do a single crowdfunding campaign with Ulule so that I only have to worry only once about the funding. This also sounded more convenient for me to rely on some French/EU website regarding legal issues, taxes etc. Also, just like Kickstarter it's possible with Ulule to offer some "rewards" to backers according to the level of contributions, so that gives a better way to motivate them.

As everybody following MathML activities noticed, big companies/organizations do not want to significantly invest in funding MathML developments at the moment. So the rationale for a crowdfunding campaign is to rely on the support of the current community and on the help of smaller companies/organizations that have business interest in it. Each one can give a small contribution and these contributions sum up in enough money to fund the project. Of course this model is probably not viable for a long term perspective, but at least this allows to start something instead of complaining without acting ; and to show bigger actors that there is a demand for these developments. As indicated on the Ulule Website, this is a way to start some relationship and to build a community around a common project. My hope is that it could lead to a long term funding of MathML developments and better partnership between the various actors.

Because one of the main demand for MathML (besides accessibility) is in EPUB, I've included in the project goals a collection of documents that demonstrate advanced Web features with native MathML. That way I can offer more concrete rewards to people and federate them around the project. Indeed, many of the work needed to improve the MathML rendering requires some preliminary "code refactoring" which is not really exciting or immediately visible to users...

Hence I launched the crowdfunding campaign the 19th of November and we reached 1/3 of the minimal funding goal in only three days! This was mainly thanks to the support of individuals from the MathML community. In mid december we reached the minimal funding goal after a significant contribution from the KWARC Group (Jacobs University Bremen, Germany) with which I have been in communication since the launch of the campaign. Currently, we are at 125% and this means that, minus the Ulule commision and my social/fiscal obligations, I will be able to work on the project during about 3 months.

I'd like to thank again all the companies, organizations and people who have supported the project so far! The crowdfunding campaign continues until the end of January so I hope more people will get involved. If you want better MathML in Web rendering engines and ebooks then please support this project, even a symbolic contribution. If you want to do a more significant contribution as a company/organization then note that Ulule is only providing a service to organize the crowdfunding campaign but otherwise the funding is legally treated the same as required by my self-employed status; feel free to contact me for any questions on the project or funding and discuss the long term perspective.

Finally, note that I've used my savings and I plan to continue like that until the official project launch in February. Below is a summary of what have been done during the five weeks before the holiday season. This is based on my weekly updates for supporters where you can also find references to the Bugzilla entries. Thanks to the Apple & Mozilla developers who spent time to review my patches!

Collection of documents

The goal is to show how to use existing tools (LaTeXML, itex2MML, tex4ht etc) to build EPUB books for science and education using Web standards. The idea is to cover various domains (maths, physics, chemistry, education, engineering...) as well as Web features. Given that many scientific circles are too much biased by "math on paper / PDF" and closed research practices, it may look innovative to use the Open Web but to be honest the MathML language and its integration with other Web formats is well established for a long time. Hence in theory it should "just work" once you have native MathML support, without any circonvolutions or hacks. Here are a couple of features that are tested in the sample EPUB books that I wrote:

  • Rendering of MathML equations (of course!). Since the screen size and resolution vary for e-readers, automatic line breaking / reflowing of the page is "naturally" tested and is an important distinction with respect to paper / PDF documents.
  • CSS styling of the page and equations. This includes using (Web) fonts, which are very important for mathematical publishing.
  • Using SVG schemas and how they can be mixed with MathML equations.
  • Using non-ASCII (Arabic) characters and RTL/LTR rendering of both the text and equations.
  • Interactive document using Javascript and <maction>, <input>, <button> etc. For those who are curious, I've created some videos for an algebra course and a lab practical.
  • Using the <video> element to include short sequences of an experiment in a physics course.
  • Using the <canvas> element to draw graphs of functions or of physical measurements.
  • Using WebGL to draw interactive 3D schemas. At the moment, I've only adapted a chemistry course and used ChemDoodle to load Crystallographic Information Files (CIF) and provide 3D-representation of crystal structures. But of course, there is not any problem to put MathML equations in WebGL to create other kinds of scientific 3D schemas.

WebKit

I've finished some work started as a MathJax developer, including the maction support requested by the KWARC Group. I then tried to focus on the main goals: rendering of token elements and more specifically operators (spacing and stretching).

  • I improved LTR/RTL handling of equations (full RTL support is not implemented yet and not part of the project goal).
  • I improved the maction elements and implemented the toggle actiontype.
  • I refactored the code of some "mrow-like" elements to make them all behave like an <mrow> element. For example while WebKit stretched (some) operators in <mrow> elements it could not stretch them in <mstyle>, <merror> etc Similarly, this will be needed to implement correct spacing around operators in <mrow> and other "mrow-like" elements.
  • I analyzed more carefully the vertical stretching of operators. I see at least two serious bugs to fix: baseline alignment and stretch size. I've uploaded an experimental patch to improve that.
  • Preliminary work on the MathML Operator Dictionary. This dictionary contains various properties of operators like spacing and stretchiness and is fundamental for later work on operators.
  • I have started to refactor the code for mi, mo and mfenced elements. This is also necessary for many serious bugs like the operator dictionary and the style of mi elements.
  • I have written a patch to restore support for foreign objects in annotation-xml elements and to implement the same selection algorithm as Gecko.

Gecko

I've continued to clean up the MathML code and to mentor volunteer contributors. The main goal is the support for the Open Type MATH table, at least for operator stretching.

  • Xuan Hu's work on the <mpadded> element landed in trunk. This element is used to modify the spacing of equations, for example by some TeX-to-MathML generators.
  • On Linux, I fixed a bug with preferred widths of MathML token elements. Concretely, when equations are used inside table cells or similar containers there is a bug that makes equations overflow the containers. Unfortunately, this bug is still present on Mac and Windows...
  • James Kitchener implemented the mathvariant attribute (e.g used by some tools to write symbols like double-struck, fraktur etc). This also fixed remaining issues with preferred widths of MathML token elements. Khaled Hosny started to update his Amiri and XITS fonts to add the glyphs for Arabic mathvariants.
  • I finished Quentin Headen's code refactoring of mtable. This allowed to fix some bugs like bad alignment with columnalign. This is also a preparation for future support for rowspacing and columnspacing.
  • After the two previous points, it was finally possible to remove the private "_moz-" attributes. These were visible in the DOM or when manipulating MathML via Javascript (e.g. in editors, tree inspector, the html5lib etc)
  • Khaled Hosny fixed a regression with script alignments. He started to work on improvements regarding italic correction when positioning scripts. Also, James Kitchener made some progress on script size correction via the Open Type "ssty" feature.
  • I've refactored the stretchy operator code and prepared some patches to read the OpenType MATH table. You can try experimental support for new math fonts with e.g. Bill Gianopoulos' builds and the MathML Torture Tests.

Blink/Trident

MathML developments in Chrome or Internet Explorer is not part of the project goal, even if obviously MathML improvements to WebKit could hopefully be imported to Blink in the future. Users keep asking for MathML in IE and I hope that a solution will be found to save MathPlayer's work. In the meantime, I've sent a proposal to Google and Microsoft to implement fallback content (alttext and semantics annotation) so that authors can use it. This is just a couple of CSS rules that could be integrated in the user agent style sheet. Let's see which of the two companies is the most reactive...

Saturday, October 12 2013

Funding MathML Developments in Gecko and WebKit

update 2013-10-15: since I got feedback, I have to say that my funding plan is independent of my work at MathJax ; I'm not a MathJax employee but I have an independent contractor status. Actually, I already used my business to fund an intern for Gecko MathML developments during Summer 2011 :-)

Retrospect

Since last April, I have been allowed by the MathJax Consortium to dedicate a small amount of my time to do MathML development in browsers, until possibly more serious involvement later. At the same time, we mentioned this plan to Google developers but unfortunately they just decided to drop the WebKit MathML code from Blink, making external contributions hard and unwelcome...

Hence I have focused mainly on Gecko and WebKit: You can find the MathML bugs that have been closed during that period on bugzilla.mozilla.org and bugs.webkit.org. For Gecko, this has allowed me to finish some of the work I started as a volunteer before I was involved full-time in MathJax as well as to continue to mentor MathML contributors. Regarding WebKit, I added a few new basic features like MathML lengths, <mspace> or <mmultiscripts> while I was getting familiar with the MathML code and WebKit organization/community. I also started to work on <semantics> and <maction>. More importantly, I worked with Martin Robinson to address the design concerns of Google developers and a patch to fix these issues finally landed early this week.

However, my progress has been slow so as I mentioned in my previous blog post, I am planning to find a way to fund MathML developments...

Why funding MathML?

Note: I am assuming that the readers of this blog know why MathML is important and are aware of the benefits it can bring to the Web community. If not, please check Peter Krautzberger's Interview by Fidus Writer or the MozSummit MathML slides for a quick introduction. Here my point is to explain why we need more than volunteer-driven development for MathML.

First the obvious thing: Volunteer time is limited so if we really want to see serious progress in MathML support we need to give a boost to MathML developments. e-book publishers/readers, researchers/educators who are stuck outside the Web in a LaTeX-to-PDF world, developers/users of accessibility tools or the MathML community in general want good math support in browsers now and not to wait again for 15 more years until all layout engines catch up with Gecko or that the old Gecko bugs are fixed.

MathML

There are classical misunderstandings from people thinking that non-native MathML solutions and other polyfills are the future or that math on the Web could be implemented via PNG/SVG images or Web Components. Just open a math book and you will see that e.g. inline equations must be correctly aligned with the text or participate in line wrapping. Moreover we are considering math on the Web not math on paper, so we want it to be compatible with HTML, SVG, CSS, Javascript, Unicode, Bidi etc and also something that is fast and responsive. Technically, this means that a clean solution must be in the core rendering engine, spread over several parts of the code and must have strong interaction with the various components like the HTML5 parser, the layout tree, the graphic and font libraries, the DOM module, the style tree and so forth. I do not see any volunteer-driven Blink/Gecko/WebKit feature off the top of my head that has this characteristic and actually even SVG or any other kind of language for graphics have less interaction with HTML than MathML has.

The consequence of this is that it is extremely difficult for volunteers to get involved in native MathML and to do quick progress because they have to understand how the various components of the Blink/Gecko/WebKit code work and be sure to do things correctly. Good mathematical rendering is already something hard by itself, so that is even more complicated when you are not writing an isolated rendering engine for math on which you can have full control. Also, working at the Blink/Gecko/WebKit level requires technical skills above the average so finding volunteers who can work with the high-minded engineers of the big browser companies is not something easy. For instance, among the enthusiastic people coming to me and willing to help MathML in Gecko, many got stuck when e.g. they tried to build the Firefox source or do something more advanced and I never heard back from them. In the other direction, Blink/Gecko/WebKit paid developers are generally not familiar with MathML and do not have time to learn more about it and thus can not always provide a relevant review of the code, or they may break something while trying to modify code they do not entirely understand. Moreover, both the volunteers and paid staff have only a small amount of time to do MathML stuff while the other parts of the engine evolve very quickly, so it's sometimes hard to keep everything in sync. Finally, the core layout engines have strong security requirements that are difficult to satisfy in a volunteer-driven situation...

Beyond volunteer-driven MathML developments

At that point, there are several options. First the lazy one: Give up with native math rendering, only focus on features that have impact on the widest Web audience (i.e. those that would allow browser vendors to get more market share and thus increase their profit), thank the math people for creating the Web and kindly ask them to use whatever hacks they can imagine to display equations on the Web. Of course as a Mozillian, I think people must decide the Web they want and thus exclude this option.

Next there is the ingenuous option: Expect that browser companies will understand the importance of math-on-the-Web and start investing seriously in MathML support. However, Netscape and Microsoft rejected the <MATH> tag from the 1995 HTML 3.0 draft and the browser companies have kept repeating they would only rely on volunteer contributions to move MathML forward, despite the repeated requests from MathML folks and other scientific communities. So that option is excluded too, at least in the short to medium term.

So it remains the ambitious option: Math people and other interested parties must get together and try to fund native MathML developments. Despite the effort of my manager at MathJax to convince partners and raise funds, my situation has not changed much since April and it is not clear when/if the MathJax Consortium can take the lead in native MathML developments. Given my expertise in Gecko, WebKit and MathML, I feel the duty to do something. Hence I wish to reorganize my work time: Decrease my involvement in MathJax core in order to increase my involvement in Gecko/WebKit developments. But I need the help of the community for that purpose. If you run a business with interest for math-on-the-Web and are willing to fund my work, then feel free to contact me directly by mail for further discussion. In the short term, I want to experiment with Crowd Funding as discussed in the next section. If this is successful we can think of a better organization for MathML developments in the long term.

Crowd Funding

Wikipedia defines Crowd funding as "the collective effort of individuals who network and pool their money, usually via the Internet, to support efforts initiated by other people or organizations". There are several Crowd Funding platforms with similar rule/interface. I am considering Catincan which is specialized in Open Source Crowd Funding, can be used by any backer/developer around the world, can rely on Bugzilla to track the bug status and seems to have good process to collect the fund from backers and to pay developers. You can easily login to the Catincan Website if you have a GitHub, Facebook or Google account (apparently Persona is not supported yet...). Finally, it seems to have a communication interface between backers and developers, so that everybody can follow the development on the funded features.

One distinctive feature of catincan is that only well-established Open Source projects can be funded and only developers from these projects can propose and work on the new features ; so that backers can trust that the features will be implemented. Of course, I have been working on Gecko, WebKit and MathML projects so I hope people believe I sincerely want to improve MathML support in browsers and that I have the skills to do so ;-)

As said in my previous blog post, it is not clear at all (at least to me) whether Crowd Funding can be a reliable method, but it is worth trying. There are many individuals and small businesses showing interest in MathML, without the technical knowledge or appropriate staff to improve MathML in browsers. So if each one fund a small amount of money, perhaps we can get something.

One constraint is that each feature has 60 days to reach the funding goal. I do not have any idea about how many people are willing to contribute to MathML and how much money they can give. The statistical sample of projects currently funded is too small to extract relevant information. However, I essentially see two options: Either propose small features and split the big ones in small steps, so that each catincat submission will need less work/money and improvements will be progressive with regular feedback to backers ; or propose larger features so they look more attractive and exciting to people and will require less frequent submissions to catincat. At the beginning, I plan to start with the former and if the crowd funding is successful perhaps try the latter.

Status in Open Source Layout Engines

Note: Obviously, Open Source Crowd Funding does not apply to Internet Explorer, which is the one main rendering engine not mentioned below. Although Microsoft has done a great job on MathML for Microsoft Word, they did not give any public statement about MathML in Internet Explorer and all the bug reports for MathML have been resolved "by design" so far. If you are interested in MathML rendering and accessibility in Internet Explorer, please check Design Science blog for the latest updates and tools.

Blink

Note: I am actually focusing on the history of Chromium here but of course there are other Blink-based browsers. Note that programs like QtWebEngine (formerly WebKit-based) or Opera (formerly Presto-based) lost the opportunity to get MathML support when they switched to Blink.

Alex Milowski and François Sausset's first MathML implementation did not pass Google's security review. Dave Barton fixed many issues in that implementation and as far as I know, there were not any known security vulnerabilities when Dave submitted his last version. MathML was enabled in Chrome 24 but Chrome developers had some concerns about the design of the MathML implementation in WebKit, which indeed violated some assumptions of WebKit layout code. So MathML was disabled in Chrome 25 and as said in the introduction, the source code was entirely removed when they forked.

Currently, the Chromium Dashboard indicates that MathML is shipped in Firefox/Safari, has positive feedback from developers and is an established standard ; but the Chromium status remains "No active development". If I understand correctly, Google's official position is that they do not plan to invest in MathML development but will accept external contributions and may re-enable MathML when it is ready (for some sense of "ready" to be defined). Given the MathML story in Chrome, it seems really unlikely that any volunteer will magically show up and be willing to submit a MathML patch. Incidentally, note the interesting work of the ChromeVox team regarding MathML accessibility: Their recent video provides a good overview of what they achieve (where Volker Sorge politely regrets that "MathML is not implemented in all browsers").

Although Google's design concerns have now been addressed in WebKit, one most serious remark from one Google engineer is that the WebKit MathML implementation is of too low quality to be shipped so they just prefer to have no MathML at all. As a consequence, the best short term strategy seems to be improving WebKit MathML support and, once it is good enough, to submit a patch to Google. The immediate corollary is that if you wish to see MathML in Chrome or other Blink-based browsers you should help WebKit MathML development. See the next section for more details.

Chromatic chromatic

Actually, I tried to import MathML into Blink one day this summer. However, there were divergences between the WebKit and Blink code bases that made that a bit difficult. I do not plan to try again anytime soon, but if someone is interested, I have published my script and patch on GitHub. Note there may be even more divergences now and the patch is certainly bit-rotten. I also thought about creating/maintaining a "Chromatic" browser (Chrome + mathematics) that would be a temporary fork to let Blink users benefit from native MathML until it is integrated back in Blink. But at the moment, that would probably be too much effort for one person and I would prefer to focus on WebKit/Gecko developments for now.

WebKit

The situation in WebKit is much better. As said before, Google's concerns are now addressed and MathML will be enabled again in all WebKit releases soon. Martin Robinson is interested in helping the MathML developments in WebKit and his knowledge of fonts will be important to improve operator stretching, which is one of the biggest issue right now. One new volunteer contributor, Gurpreet Kaur, also started to do some work on WebKit like support for the *scriptshifts attributes or for the <menclose> element. Last but not least, a couple of Apple/WebKit developers reviewed and accepted patches and even helped to fix a few bugs, which made possible to move development forward.

WebKineTic

When he was still working on WebKit, Dave Barton opened bug 99623 to track the top priorities. When the bugs below and their related dependencies are fixed, I think the rendering in WebKit will be good enough to be usable for advanced math notations and WebKit will pass the MathML Acid 1 test.

  • Bug 44208: For example, in expression like sin(x), the "x" should be in italic but not the "sin". This is actually slightly more complicated: It says when the default mathvariant value must be normal/italic. mathvariant is more like the text-transform CSS property in the sense that it remaps some characters to the corresponding mathematical characters (italic, bold, fraktur, double-struck...) for example A (mathvariant=fraktur A) should render exactly the same as 𝔄 (U+1D504). By the way, there is the related bug 24230 on Windows, that prevents to use plane 1 characters. The best solution will probably be to implement mathvariant correctly. See also Gecko's ongoing work by James Kitchener below.
  • Bug 99618: Implement <mmultiscripts>, that allows expressions like C614 or Rij;j=12S;i. As said in the introduction, this is fixed in WebKit Nightly.
  • Bug 99614: Support for stretchy operators like in (z1+z2¯3)4. Currently, WebKit can only stretch operators vertically using a few Unicode constructions like ⎛ (U+239B) + ⎜ (U+239C) + ⎝ (U+239D) for the left parenthesis. Essentially only similar delimiters like brackets, braces etc are supported. For small sizes like ( or for large operators like n2 it is necessary to use non-unicode glyphs in various math fonts, but this is not possible in WebKit MathML yet. All of this will require a fair amount of work: implementing horizontal stretching, font-specific stuff, largeop/symmetric properties etc
  • Bug 99620: Implement the operator dictionary. Currently, WebKit treats all the operators the same way, so for example it will use the same 0.2em spacing before and after parenthesis, equal sign or invisible operators in e.g. f(x)=x2. Instead it should use the information provided by the MathML operator dictionary. This dictionary also specifies whether operators are stretchy, symmetric or largeop and thus is related to the previous point.
  • Bug 119038: Use the code for vertical stretchy operators to draw the radical symbols in roots like 23. Currently, WebKit uses graphic primitives which do not give a really good rendering.
  • Bug 115610: Implement <mspace> which is used by many MathML generators to do some spacing in mathematical formulas. As said in the introduction, this is fixed in WebKit Nightly.

In order to pass the Mozilla MathML torture tests, at least displaystyle and scriptlevel must be implemented too, probably as internal CSS properties. This should also allow to pass Joe Java's MathML test, although that one relies on the infamous <mfenced> that duplicates the stretchy operator features and is implemented inconsistently in rendering engines. I think passing the MathML Acid 2 test will require slightly more effort, but I expect this goal to be achievable if I have more time to work on WebKit:

  • Bug 115610: Implement <mspace>. Fixed!
  • Bug 120164: Implement negative spacing for <mspace> (I have an experimental patch).
  • Bug 85730: Implement <mpadded>, which is also used by MathML generators to do some tweaking of formulas. I have only done some experiments, that would be a generalization of <mspace>
  • Bug 85733: Implement the href attribute ; well I guess the name is explicit enough to understand what it is used for! I only have some experimental patch here too. That would be mimicing what is done in SVG or HTML.
  • Bug 120059 and bug 100626: Implement <maction> (at least partially) and <semantics>, which have been asked by long-time MathML users Jacques Distler and Michael Kohlhase. I have patches ready for that and this could be fixed relatively soon, I just need to find time to finish the work.

In general passing the MathML Acid 2 test is not too hard, you merely only need to implement those few MathML elements whose exact rendering is clearly defined by the MathML specification. Passing the MathML Acid 3 test is not expected in the medium term. However, the score will naturally increase while we improve WebKit MathML implementation. The priority is to implement what is currently known to be important to users. To give examples of bugs not previously mentioned: Implementing menclose or fixing various DOM issues like bugs 57695, 57696 or 107392.

More advanced features like those mentioned in the next section for Gecko are probably worth considering later (Open type MATH, linebreaking, mlabeledtr...). It is worth noting that Apple has already done some work on accessibility (with MathML being readable by VoiceOver in iOS7), authoring and EPUB (MathML is enabled in WebKit-based ebook readers and ibooks-author has an integrated LaTeX-to-MathML converter).

Gecko

Mathzilla

In general I think I have a good relationship with the Mozilla community and most people have respect for the work that has been done by volunteers for almost 15 years now. The situation has greatly improved since I joined the project, at that time some people claimed the Mozilla MathML project was dead after Roger Sidge's departure. One important point is that Karl Tomlinson has worked on repairing the MathML support when Roger Sidge left the project. Hence there is at least one Mozilla employee with good knowledge of MathML who can review the volunteer patches. Another key ingredient is the work that has recently been made by Mozilla to increase engagement of the volunteer community like good documentation on MDN, the #Introduction channel, Josh Matthews' mentored bugs and of course programs like GSOC. However, as said above, it is one thing to attract enthusiastic contributors and another thing to get long-term contributors who can work independently on more advanced features. So let's go back to my latest Roadmap for the Mozilla MathML Project and see what has been accomplished for one year:

  • Font support: Dmitry Shachnev created a Debian package for the MathJax fonts and Mike Hommey added MathJax and Asana fonts in the list of suggested packages for Iceweasel. The STIX fonts have also been updated in Fedora and are installed by default on Mac OS X Lion (10.7). For Linux distributions, it would be helpful to implement Auto Installation Support. The bug to add mathematical fonts to Android has been assigned in June but no more progress has happened so far. Henri Sinoven opened a bug for FirefoxOS but there has not been any progress there either. I had some patches to restore the "missing MathML fonts" warning (using an information bar) but it was refused by Firefox reviewers. However, the code to detect missing MathML font could still be used for the similar bug 648548, which also seems inactive since January. There are still some issues on the MathJax side that prevent to integrate Web fonts for the native MathML output mode. So at the moment the solution is still to inform visitors about MathML fonts or to add MathML Web fonts to your Web site. Khaled Hosny (font and LaTeX expert) recently updated my patches to prepare the support for Open Type fonts and he offered to help on that feature. After James Kitchener's work on mathvariant, we realized that we will probably need to provide Arabic mathematical fonts too.
  • Spacing: Xuan Hu continued to work on <mpadded> improvements and I think his patch is close to be accepted. Quentin Headen has done some progress on <mtable> before focusing on his InstantBird GSOC project. He is still far from being able to work on mtable@rowspacing/columnspacing but a work around for that has been added to MathJax. I fixed the negative space regression which was missing to pass the MathML Acid 2 test and is used in MathJax. Again, Khaled Hosny is willing to help to use the spacing of the Open Type MATH, but that will still be a lot of work.
  • <mlabeledtr>: A work around for native MathML has been added in MathJax.
  • Linebreaking: No progress except that I have worked on fixing a bug with intrinsic width computation. The unrelated printing issues mentioned in the blog post have been fixed, though.
  • Operator Stretching: No progress. I tried to analyze the regression more carefully, but nothing is ready yet.
  • Tabular elements: As said above, Quentin Headen has worked a bit on cleaning up <mtable> but not much improvements on that feature so far.
  • Token elements: My patch for <ms> landed and I have done significant progress on the bad measurement of intrinsic width for token elements (however, the fix only seems to work on Linux right now). James Kitchener has taken over my work on improving our mathvariant support and doing related refactoring of the code. I am confident that he will be able to have something ready soon. The primes in exponents should render correctly with MathJax fonts but for other math fonts we will have to do some glyph substitutions.
  • Dynamic MathML: No progress here but there are not so many bugs regarding Javascript+MathML, so that should not be too serious.
  • Documentation: It is now possible to use MathML in code sample or directly in the source code. The MathML project pages have been entirely migrated to MDN. Also, Florian Scholz has recently been hired by Mozilla as a documentation writer (congrats!) and will in particular continue the work he started as a volunteer to document MathML on MDN.

I apologize to volunteers who worked on bugs that are not mentioned above or who are doing documentation or testing that do not appear here. For a complete list of activity since September 2012, Bugzilla is your friend. There are two ways to consider the progress above. If you see the glass half full, then you see that several people have continued the work on various MathML issues, they have made some progress and we now pass the MathML Acid 2 test. If you see the glass half empty, then you see that most issues have not been addressed yet and in particular those that are blocking the native MathML to be enabled in MathJax: bug 687807, bug 415413, the math font issues discussed in the first point, and perhaps linebreaking too. That is why I believe we should go beyond volunteer-driven MathML developments.

Most of the bugs mentioned above are tested by the MathML Acid 3 tests and we will win a few points when they are fixed. Again, passing MathML Acid 3 test is not a goal by itself so let's consider what are the big remaining areas it contains:

  • Improving Tabular Elements and Operator Stretching, which are obviously important and used a lot in e.g. MathJax.
  • Linebreaking, which as I said is likely to become fundamental with small screens and ebooks.
  • Elementary Mathematics (you know addition, subtraction, multiplication, and division that kids learn), which I suspect will be important for educational tools and ebooks.
  • Alignment: This is the one part of MathML that I am not entirely sure is relevant to work on in the short term. I understand it is useful for advanced layout but most MathML tools currently just rely on tables to do that job and as far as I know the only important engine that implements that is MathPlayer.

Finally there are other features outside the MathML rendering engines that I also find important but for which I have less expertise:

  • Transferring MathML that is implementing copy/cut/drag and paste. Currently, we can do that by treating MathML as normal HTML5 code or by using the "show MathML source" feature and copying the source code. However, it would be best to implement a standard way to communicate with other MathML applications like Microsoft Word, Mathematica, Mapple, Windows' Handwriting panel etc I wrote some work-in-progress patches last year.
  • Authoring MathML: Essentially implementing things like deletion, insertion etc maybe simple MathML token creation ; in Gecko's core editor, which is used by BlueGriffon, KompoZer, SeaMonkey, Thunderbird or even MDN. Other things like integrating Javascript parsers (e.g. ASCIIMath) or equation panels with buttons like are probably better done at the higher JS/HTML/XUL level. Daniel Glazman already wrote math input panels for BlueGriffon and Thunderbird.
  • MathML Accessibility: This is one important application of MathML for which there is strong demand and where Mozilla is behind the competitors. James Teh started some experimental work on his NVDA tool before the summit.
  • EPUB reader for FirefoxOS (and other mobile platforms): During the "Co-creating Action Plans" session, the Mozilla Taipei people were thinking about missing features for FirefoxOS and this idea about EPUB reader was my modest contribution. There are a few EPUB readers relying on Gecko and it would be good to check if they work in FirefoxOS and if they could be integrated by default, just like Apple has iBooks. BTW, there is a version of BlueGriffon that can edit EPUB books.

Conclusion

I hope I have convinced some of the readers about the need to fund MathMLin browsers. There is a lot of MathML work to do on Gecko and WebKit but both projects have volunteers and core engineers who are willing to help. There are also several individuals / companies relying on MathML support in rendering engines for their projects and could support the MathML developments in some way. I am willing to put more of my time on Gecko and WebKit developments, but I need financial help for that purpose. I'm proposing catincan Crowd Funding in the short term so that anyone can contribute at the appropriate level, but other alternatives to fund the MathML development can be found like asking Peter Krautzberger about native MathML funding in MathJax, discussing with Igalia about funding Martin Robinson to work more on WebKit MathML or contacting me directly to establish some kind of part-time consulting agreement.

Please leave a comment on this blog or send me a private mail, if you agree that funding MathML in browsers is important, if you like the crowd funding idea and plan to contribute ; or if you have any opinions about alternative funding options. Also, please tell me what seem to be the priority for you and your projects among what I have mentioned above (layout engines, features etc) or among others that I may have forgotten. Of course, any other constructive comment to help MathML support in browsers is welcome. I plan to submit features on catincan soon, once I have more feedback on what people are interested in. Thank you!

Friday, January 11 2013

MathML in Chrome, a couple of demos and some perspectives...

For those who missed the news, Google Chrome 24 has recently been released with native MathML support. I'd like to thank Dave Barton again for his efforts during the past year, that have allowed to make this happen. Obviously, some people may ironize on how long it took for Google to make this happen (Mozilla MathML project started in 1999) or criticize the bad rendering quality. However the MathML folks, aware of the history of the language in browsers, will tend to be more tolerant and appreciate this important step towards MathML adoption. After all, this now means that among the most popular browsers, Firefox, Safari and Chrome have MathML support and Opera a basic CSS-based implementation. This also means that about three people out of four will be able to read pages with MathML without the need of any third-party rendering engine.

After some testing, I think the Webkit MathML support is now good enough to be used on my Website. There are a few annoyances with stretchy characters or positioning, but in general the formulas are readable. Hence in order to encourage the use of MathML and let people report bugs upstream and hopefully help to fix them, I decided to rely on the native MathML support for Webkit-based browsers. I'll still keep MathJax for Internet Explorer (when MathPlayer is not installed) and Opera.

I had the chance to meet Dave Barton when I was at the Silicon Valley last October for the GSoC mentor summit. We could exchange our views on the MathML implementations in browsers and discuss the perspectives for the future of MathML. The history of MathML in Webkit is actually quite similar to Gecko's one: one volunteer Alex Milowski decided to write the initial implementation. This idea attracted more volunteers who joined the effort and helped to add new features and to conduct the project. Dave told me that the initial Webkit implementation did not pass the Google's security review and that's why MathML was not enabled in Chrome. It was actually quite surprising that Apple decided to enable it in Safari and in particular all Apple's mobile products. Dave's main achievement has been to fix all these security bugs so that MathML could finally appear in Chrome.

One of the idea I share with Dave is how important it is to have native MathML support in browsers, rather than to delegate the rendering to Javascript libraries like MathJax or browser plug-in like MathPlayer. That's always a bit sad to see that third-party tools are necessary to improve the native browser support of a language that is sometimes considered a core XML language for the Web together with XHTML and SVG. Not only native support is faster but also it integrates better in the browser environment: zooming text, using links, applying CSS style, mixing with SVG diagrams, doing dynamic updates with e.g. Javascript etc all of the features Web users are familiar with are immediately available. In order to illustrate this concretely, here is a couple of demos. Some of them are inspired from the Mozilla's MathML demo pages, recently moved to MDN. By the way, the famous MathML torture page is now here. Also, try this test page to quickly determine whether you need to install additional fonts.

MathML with CSS text-shadow & transform properties, href & dir attributes as well as Javascript events

det ( 1 2 3 4 5 6 7 8 9 ) = 45 + 84 + 96 ( 105 + 48 + 72 ) = 0 محدد ( ١‎ ٢ ٣ ٤ ٥ ٦ ٧‎ ٨ ٩ ) = ٤٥ + ٨٤ + ٩٦ ( ١‎٠٥ + ٤٨ + ٧‎٢ ) = ٠

HTML and animated SVG inside MathML tokens

tr ( ) n = = 0 π 2 θ θ

MathML inside animated SVG (via the <foreignObject> element):

n = 0 + α n n ! exp(α)

Note that although Dave was focused on improving MathML, the language naturally integrates with the rest of Webkit's technologies and almost all the demos above work as expected, without any additional efforts. Actually, Gecko's MathML support relies less on the CSS layout engine than Webkit does and this has been a recurrent source of bugs. For example in the first demo, the text-shadow property is not applied to some operators (bug 827039) while it is in Webkit.

In my opinion, one of the problem with MathML is that the browser vendors never really shown a lot of interest in this language and the standardization and implementation efforts were mainly lead and funded by organizations from the publishing industry or by volunteer contributors. As the MathML WG members keep repeating, they would love to get more feedback from the browser developers. This is quite a problem for a language that has among the main goal the publication of mathematics on the Web. This leads for example to MathML features (some of them are now deprecated) duplicating CSS properties or to the <mstyle> element which has most of its attributes unused and do similar things as CSS inheritance in an incompatible way. As a consequence, it was difficult to implement all MathML features properly in Gecko and this is the source of many bugs like the one I mention in the previous paragraph.

Hopefully, the new MathML support in Chrome will bring more interest to MathML from contributors or Web companies. Dave told me that Google could hire a full-time engineer to work on MathML. Apparently, this is mostly because of demands from companies working on Webkit-based mobile devices or involved in EPUB. Although I don't have the same impression from Mozilla Corporation at the moment, I'm confident that with the upcoming FirefoxOS release, things might change a bit.

Finally I also expect that we, at MathJax, will continue to accompany the MathML implementations in browsers. One of the ideas I proposed to the team was to let MathJax select the output mode according to the MathML features supported by the browser. Hence the native MathML support could be used if the page contains only basic mathematics while MathJax's rendering engine will be used when more advanced mathematical constructions are involved. Another goal to achieve will be to make MathJax the default rendering in Wikipedia, which will be much better than the current raster image approach and will allow the users to switch to their browser's MathML support if they wish...