What can lxml do? Verify whether the xhtml is valid? But it doesn't mean the post is safe. Yes you can filter out certain dangerous tags on the server side but it's still a "explicitly forbidden or it's allowed" approach, and it's not as secure as "explicitly allow or it's forbidden". And you have to constantly patch your filter whenever a new feature, and the corresponding HTML extension, is developed on any major browser.

Also as mentioned in the other thread, markup is easier to write by hand, not everyone likes WYSIWYG. For a forum may be not, but for a wiki many people will definitely prefer a plain textfield.

Oh, and also, directly using the HTML generated by tinymce is also the opposite of "clean code". A bold line which uses <strong> now will probably become <span style="font-weight:bold"> tomorrow. You can view the effort to convert HTML to a custom markup as equivalent to trying to get rid of these differences.


--- Last Edited by lootgoal at 2009-07-05 11:04:00 ---

Well, I don't have much experience with different wiki markups yet.. But as to the wysiwyg editor, I do have several new thoughts.

I'm planning to completely rewrite it, and dump the ugly "regex conversion" approach, in favor of a *simple* DOM parser. Looks like it's not as difficult as I originally thought to do it this way.

The basic goals include:
1. It should be as extensible as it can get. The core is only responsible for DOM parsing, event dispatching, plugin management, etc, and every user-level functionality is implemented through a plugin. Plugins may correspond to a toolbar widget, a right-click menu entry, both, or neither. The only requirement is to provide some event handlers.

2. For each plugin, the appearance (how its widgets are displayed), the frontend (how it modifies the html content) and the backend (what markup syntax is used and how to do the conversions) should be completely decoupled, meaning you can change each of them independently of others to suit your needs. Several versions of each can be provided out of the box for the user to choose from. For example, you may have a backend for BBcode and another backend for some wiki markup and they are interchangable.

3. Some kind of theming support.

Worth mentioning is the backend. Plugins register the HTML tags, attributes or styles they are responsible for, and provide corresponding replace strings or functions to do the conversion. When a registered element is encountered in the html during a conversion, either the replace string is injected, or the function is called with necessary parameters to allow custom processing.

The replace string supports a simple syntax for variables and necessary flow control. I'm currently thinking of the following special characters: $, @, (, and ). $ indicates variables, and @ means "for each".

For example, to implement the "bold" button, the plugin would register the tags <b> and <strong>, and the style font-weight with value "bold". The corresponding replace string would be something like "[ b]$tag.html[ /b]" for bbcode and "'''$tag.html'''" for wiki markup, which indicates that the markups are placed around the html content of the tag.

A more complex example is to deal with tables. The plugin would register the tag <table>, and the replace string might look like:
"{|( width=$table.width)( border=$table.border)( cellpadding=$table.cellpadding)( cellspacing=$table.cellspacing)
@tr(|-
@td(|$td.html)
)
|}"
The braces in the first line indicate that if the variable doesn't exist (no such attribute), don't include the entire content in the braces (it's like an exception handler, if none of these is encountered and a "variable not existed" exception propagates to the top level, an error occurs). And those in the second and third lines are used to mark the boundaries of the loops.

Each tag can only be registered by one plugin.

If what you need to do can't be achieved with replace strings you can use a custom function, which is passed an object containing the entire DOM tree as well as your position on it. So you have the complete info you might need at hand. The drawback is that you have to also provide another function to convert the markup back to html. While for the replace string case, it is not needed.

Plugins are ordered and called in this order. This would lead to more consistent code. For example, "[ b][ i]blah blah[ /i][ /b]" won't become "[ i][ b]blah blah[ /b][ /i]" next time it's submitted.

I'm also thinking of supporting various wiki macros. The problem here is that some macros need global info that is not available on the client side. So the plugin might want to contact the server using Ajax and inject the result later.


I'm kind of using this post to record my thoughts on the implementation, so sorry if it contains too many details. On the other hand, I'm sure there are many complications and the details would probably change a lot later, but I think the big picture is here. Sounds interesting to you?

I more or less had these thoughts more than a month ago but haven't been able to write any code, I guess I'll try another strategy. So instead of trying to squeeze out a whole day I'll try to spend an hour everyday to work on it. Let's see whether it works out...

OK, the main complexity I see with this approach is that the bbcode<->html conversions may not work in such an "incremental" way. They may conflict with each other and/or with the base one. But I think it should still be doable, we just need to make sure these transformers all obey certain rules. I guess the first step is trying to completely "decouple" the current implementation and make every button an independent "plugin", I'll see if it's possible at all.

For your second question, I don't know whether it's possible, and I don't have time to investigate right now. I'll be busy in the next 2-3 weeks, after that maybe I'll have some time to play with this new idea.

Let's see whether I have got it. You want to see this editor be like a javascript library, you can extend it, say, add a button, by calling a function, providing maybe 2 callbacks with the related frontend and backend code, respectively, and maybe an image, and a positioning param. Am I right?

Javascript is indeed particularly powerful in doing things like this, but I can imagine there are complexities too, especially in the backend. Are you talking about just adding new interface functionalities which will be mapped to the same bbcode as before or you want to modify the underlying bbcode implementation to include more tags?

Ummm... but why not just use tinymce for content pages? Make it more configurable, yes, that would be nice, and not hard to do; but what do you mean by "additional functionality"? The functionality of this editor is restricted by the power of BBcode anyway, or is there a widely accepted extension to BBcode for content page editing?

Maybe we can separate the backend from the frontend though, so that we can have a "wiki backend", which converts HTML to the Wikipedia-style markup, etc. But this can also be done as tinymce plugins, and this way we don't have to rewrite the frontend code.

TinyMCE is a HTML WYSIWYG editor, meaning you store HTML directly into your database, this is OK for authors of a website but not suitable in a forum because it allows too much flexibility, and a user will be able to write some bad HTML (intentionally or not) to ruin your whole page.

That's why forums typically use a much more restricted BBcode system, but TinyMCE's support for BBcode is crappy. And you can see no major forum software uses TinyMCE, they either have their own solution or don't support WYSIWYG at all. The main strength of my editor is the ability to convert BBcode to HTML for display and editing and then convert HTML to BBcode for storage, thus eliminating the risk. Actually my code for this functionality is derived from and much improved over TinyMCE's BBcode plugin so I know it's more sophisticated, feature rich, and has better cross-browser support.

Thanks for pointing out the problems.

The first one is indeed a problem on all browsers, while the second one seems to be opera-specific, anyway, they should be straightforward to fix, I'll do it when I have time.

Thanks
My new website will launch in around a month, I'll contact you then.

Cool




Just some minor notes:

1. I added two buttons, 'undo' and 'redo', in the 0.2 version, and 2 corresponding icons, it seems you didn't add these icons to the site, leaving two blanks in the interface.




2. I used a fixed width layout in my own site, and coded this editor accordingly, which looks ugly on this site when the width of the page is large (the subject field is much wider than the editor), you might want to fix it.

Oh, what I actually meant was to go mod_wsgi with multiple threads, it'd be good still to use the worker mpm to reduce memory use