Sphene Community Tools

Hello,

I'd like to know what is the rationale behind the adoption of Markdown as wiki language.

The main issue with MD is that it is designed to express just a tiny fraction of what can be expressed in html (as is cleanly stated), and to let users to use html as fallback. The problem is that in the wiki you are not supposed to use html tags: the wiki is in safe mode, as rightly so, and html/js injection is not supposed to be allowed.

What remains of MD is the core language (which is frankly poor designed, with feature such as <br> relying on invisible suffix whitespaces, no tables etc.) and whose implementation is hard to extend (e.g. you had to patch the source to extend the language with your tags, but now that new MD versions have been released, porting those patches has to be done yet).

There were other choices as wiki markup language: a couple that come to mind are:

- reStructuredText: it is a more complete markup language and one particularly famous in the Python world (since Python 2.6 it replaced LaTeX as documentation)

- BBCode: I don't know about any wiki using such language, but being already used in the Board it may be a consistent choice.

reST is more easily extendible: your extension macros could have easily been plugged in as "extension directives" without hacking the source. It has the disadvantage to not allow WikiWords to be automatically converted into links (but actually with a preprocessor it can be done). The more natural way to create a link could be the "default interpreted text role", e.g. `page` (which would work both with CamelCase words and not), but for us (the Italian Python community) there is the extra disadvantage that the backtick character is not available on the Italian keyboard (at least on Windows, where an ALT+0096 is required... and laptop users are out of luck).

I don't know neither BBCode design (if any exists) nor the implementation used by Sphene, so I don't know if it could be successfully used in the wiki and if it is easily extendible, but at least it would be something not to be learned: everybody already using the Board would be already able to create and edit wiki page.

So I ask: is Markdown really a choice you've done after pondering about other possibilities, and about which you are still convinced? Or at this moment you would consider other choices (reST, BB or other I am not aware of)?

Some of my users already expressed a preference in reST, but none of them has actually started a sphene-docutils integration. I have currently planned to implement other features before doing it (which could actually be a bad decision: if then I'll find a lot of wiki pages to convert, that could discourage me). What is your opinion about:

- having MD replaced by reST

- having a choice of different wiki languages (e.g. with a global setting, with a dropdown under the page etc.)

I'd like to know other Sphene users opinion as well.

Thank you very much.

sorry for the late answer..

i have used markdown because it felt like the most natural to be written.. but i have to admit i am far from satisfied by any of those solutions.. no option is perfect.. and everything has it's limitation.. i also thought for a while about using BBCode as wiki markup.. but i don't think this is the best solution either..

probably the best way would be to use a subset of XHTML - it is well defined and most people on the web are familiar with it. on top of it it should be easy to hook up a WYSIWYG editor with custom features for the wiki. i guess this would be the perfect solution. what do you think?

Hello,

Herbert Poul said @ 2009-07-02 09:32:59:

probably the best way would be to use a subset of XHTML - it is well defined and most people on the web are familiar with it. on top of it it should be easy to hook up a WYSIWYG editor with custom features for the wiki. i guess this would be the perfect solution. what do you think?

well, i think the path of designing a markup language on our own is tricky: it has already been taken by a lot of people and groups and there are already dozen of them (I should have mentioned WikiMedia, probably the most used, and Creole, which is a tentative standard).

I think the choice of a wiki syntax is really a matter of personal preference, so i started implementing the possibility to choose the wiki language: I added a "markup" to the WikiSnip and WikiSnipChange objects with the relevant code to manage them, created a "register" of renderers and added a few configuration settings: default markup, list of available markups, whether the markup choice is available to the web interface... But then I noticed that a lot of the same things are already available in the sphboard. So now i'd take a different path:

- move the renderers from sphboard to community
- instead of having a render() method, add render_post(), render_wikisnip() methods to the Renderer class.

If that's ok for you, I can provide a patch for that (but I won't be able to work on it the next week). I already implemented a reST wiki renderer, but it doesn't implement the proposed interface: i'd be glad to refactor it and have a single register of renderers, for both wiki and board.

Cheers.

that's exactly why i have suggested XHTML - it is available.. no need to design anything new.. what i meant with adding custom features would be something like <sct:wikiurl snip="AnotherSnip" /> or <sct:attachmenturl id="123" /> etc. - everything else would use the default XHTML namespace and tags. but obviously it has to be a subset of XHTML since you don't want to allow <script ../> tags or anything like it..

Daniele Varrazzo said @ 2009-07-04 03:23:50:

choose the wiki language: I added a "markup" to the WikiSnip and WikiSnipChange objects with the relevant code to manage them, created a "register" of renderers and added a few configuration settings: default markup, list of available markups,

i think choosing a markup on a per-wiki snip basis makes even less sense than for posts. i came to the conclusion after experimenting with the forum that it hardly ever makes sense. what if you have written your post in ReST - and now i want to quote you.. but i write BBCode.. how is that going to work out? -- this is solvable obviously, and i have thought about strategies.. but it still won't be perfect no matter how it is implemented.

for the wiki, it will be even worse.. so if you want people to contribute to a wiki, they won't have to know one markup language, but dozens, because everyone who created a new wiki snip will probably have chosen another markup.

so probably the only thing that would make sense.. is to make it configurable for the whole website (or at least on a per-Group basis).. not for every snip.


but what you have to always think about is that once you made a choice of markup language, and are starting to use it on your site - there is no way on going back. i see it on this site right here.. i have enabled the autolinking of CamelCase words into the wiki.. this sounds great and everything, but in most cases it is just plain annoying. but i can't just turn it off, because it might be used in one or two locations. (the same goes for wiki links in brackets .. or even bbcode.. on http://thedjangoforum.com we were talking about some python code and in that code an array was accessed with the variable 'i' .. so you've got blah[ i ] .. without the spaces it would turn into 'italic'.. this is why i'm saying.. this whole markup stuff is extremely painful. it always gets into my way. so even for me i slowly start to doubt that anything beside XHTML - where i KNOW the rules, and they make sense to me is the right choice. and every user who is not familiar with XHTML will probably be annoyed if he has to learn another markup language - so give him a WYSIWYG editor and he will be more than happy.)


So maybe the right choice would be to use TinyMCE, FCKeditor or any other wysiwyg editor and add custom extensions to create wiki links, attachments, insertions of macros, etc.


what is your incentive on using another markup language?

hehe, sorry for that long reply :) i am somehow not satisfied with any markup solution in SCT yet (and probably not for any wiki out there) and i fear implementing my vision of a simple wysiwyg editor which generates beautiful code which result in useful diffs, is not really easy to do. :(

I got this idea: this is not xhtml, which is a well defined xml language, but another xml language of your own. You are right that this would save you from the hassle of deciding whether [ i] is a link or the start of an italic section: the "well formedness" part is ok. You can't directly render your language because you have some tags more <sct:wikiurl> and some less <script> so you will have to undergo a translation phase: in this respect the language would be like the other ones (it may easily be accomplished by tools such xslt... if you are into them).

I see good parts in your idea (the language is soundly well formed) but xml is not meant to be edited by people: you should heavily rely on a wysiwyg editor or the user experience would be pretty bad. I am not a great fan of them, and I would like the idea of being able to use a regular textarea and edit something easily hand-readable/writable.

Daniele Varrazzo said @ 2009-07-04 03:23:50:

choose the wiki language: I added a "markup" to the WikiSnip and WikiSnipChange objects with the relevant code to manage them, created a "register" of renderers and added a few configuration settings: default markup, list of available markups,

i think choosing a markup on a per-wiki snip basis makes even less sense than for posts. i came to the conclusion after experimenting with the forum that it hardly ever makes sense. what if you have written your post in ReST - and now i want to quote you.. but i write BBCode.. how is that going to work out? -- this is solvable obviously, and i have thought about strategies.. but it still won't be perfect no matter how it is implemented.

for the wiki, it will be even worse.. so if you want people to contribute to a wiki, they won't have to know one markup language, but dozens, because everyone who created a new wiki snip will probably have chosen another markup.

so probably the only thing that would make sense.. is to make it configurable for the whole website (or at least on a per-Group basis).. not for every snip.

You are perfectly right: i like the idea to have the language selectable per site or per group basis, but having it per snippet could quickly become a mess. I would probably leave the option to have the markup selectable by record (e.g. to be able to import pages from other wikis at database level) but personally, in my website, I would choose a markup language of my taste and stick with that.

but what you have to always think about is that once you made a choice of markup language, and are starting to use it on your site - there is no way on going back. i see it on this site right here.. i have enabled the autolinking of CamelCase words into the wiki.. this sounds great and everything, but in most cases it is just plain annoying. but i can't just turn it off, because it might be used in one or two locations. (the same goes for wiki links in brackets .. or even bbcode.. on http://thedjangoforum.com we were talking about some python code and in that code an array was accessed with the variable 'i' .. so you've got blah[ i ] .. without the spaces it would turn into 'italic'.. this is why i'm saying.. this whole markup stuff is extremely painful. it always gets into my way. so even for me i slowly start to doubt that anything beside XHTML - where i KNOW the rules, and they make sense to me is the right choice. and every user who is not familiar with XHTML will probably be annoyed if he has to learn another markup language - so give him a WYSIWYG editor and he will be more than happy.)

Yes, this is the sort of problems for which I think designing a markup language is not an easy task. Our conclusion are instead different instead :D I'm for using an already established markup language, you seem more willing to rely on a wysiwyg editor.

So maybe the right choice would be to use TinyMCE, FCKeditor or any other wysiwyg editor and add custom extensions to create wiki links, attachments, insertions of macros, etc.


what is your incentive on using another markup language?

My main point is that using xml by hand is a pain, while markup languages are designed to be used by hand. But I know there are camps in which that would be a lovely choice, and I don't want my ideas to stop the adoption of your tool elsewhere. I've also had in the past bad experiences in using wysiwyg editor: let's say that for me as an user I prefer to be able to use a plain textarea because it gives me more precision (if the markup supports me), "useful diffs" as you mention and such. But for me as a site admin I would like to be able to choose the markupof the wiki and eventually let people use an editor if they wish to.

I think your xml-based language would not be more easy to handle for a JS editor: because you have to basically whitelist the tags you want to accept (and tranlate some others), I think the roundtrips SpheneML <-> xhtml would requre the same "amount of works" of the others. My experience with JS editors as developer is limited anyway (the only one I looked into is the experimental sphene one) and i see it's everything a matter of bunches of regexp.

hehe, sorry for that long reply :) i am somehow not satisfied with any markup solution in SCT yet (and probably not for any wiki out there) and i fear implementing my vision of a simple wysiwyg editor which generates beautiful code which result in useful diffs, is not really easy to do. :(

It's a pleasure to be able to discuss about these points :) I think sct is a good package of products, the board rocks, and the wiki is already well structured at models level.

I did some work on the experimental wysiwyg in sphene, and it was relatively easy: because I am converting an existing board, I used my set of smileys and hacked the editor to inject smileys such ": )" instead of [ img]: It was, as I mentioned, a matter of regexp. You can check the result to this board (sorry, you should register to be able to see the editor: I've not been able to use the captchas yet).

After this experiece, I may say that I can refactor the "Renderer" into an object exposing these interfaces:

- render a board message
- render a wiki page
- provide JS for a wysiwyg editor to convert html -> markup
- provide JS for a wysiwyg editor to convert markup -> html

and make the markup language "pluggable" onto the current models.

Notice that the JS conversion are only relevant to the current editor: it may be that using FCK or others they end up not being request: I will investigate into that.

btw, in our forum, you may see other features in the board: message icons, Italian translation and a method to choose the smileys set: the patches are not polished yet for submission to sct: I will ask you something else in other post and then provide you good patches (but if you want "bad patches", they are already available ;) )

No, not really.. HTML and XHTML is very flexible - what i would do is use a default wysiwyg editor and add custom tags.. or.. since i now came to think about it, probably just attributes - for example an <img src="blah.png" sctattachmentid="123" /> .. this is still valid HTML and every WYSIWYG editor probably supports (ignores) this.. this would save me completely from modifying the HTML code with regexes. i would only need to parse it on the server side to make sure an <img ...> tag with a 'sctattachmentid' is always correctly linked to that attachment.

does this make sense? :)

i think you have to extend any existing markup language - this is why i added macros to markdown (e.g. adding news, including an URL to an attachment, displaying a ToC, etc.). so the question is what is the easiest way :)

and when you think about it - what markup do you need in a wiki? the only annoying thing in (X)HTML i can think of are lists, because they are quite verbose.. but if you think about <strong>blah</strong> or <p>my cool paragraph</p> or <pre>here comes my code</pre> .. XHTML is not much more verbose than BBCode, but way more powerful and there are existing wysiwyg editors you can use without doing regex conversions.

Daniele Varrazzo said @ 2009-07-04 11:34:25:

My main point is that using xml by hand is a pain, while markup languages are designed to be used by hand. But I know there are camps in which that would be a lovely choice, and I don't want my ideas to stop the adoption of your tool elsewhere. I've also had in the past bad experiences in using wysiwyg editor: let's say that for me as an user I prefer to be able to use a plain textarea because it gives me more precision (if the markup supports me), "useful diffs" as you mention and such. But for me as a site admin I would like to be able to choose the markupof the wiki and eventually let people use an editor if they wish to.

i always thought that way too - until i started to write a blog and wanted to include images.. how do you do that in BBCode? [ img ]myimage.png[/ img ] easy enough? ok.. how do you define a width/height? i want to define a width/height and the blog should auto-resize the image on the server side. how about adding an align? BBCode does not have attributes for tags.. this is all very limiting. if you have valid xhtml it would be easy to write <img src="myimage.png" sctattachmentid="123" height="500" width="300" align="right" /> - in the wysiwyg editor the image will be resized on the client side, and loaded from 'myimage.png'.. when i render it on the server side i look for sctattachmentid, resize the image, place it somewhere in the media directory and modify src=".."

such an img tag would be easy to write by hand (simply leave out the src=".." you don't need it anyway, it will be inserted on the server side) and you can use a wysiwyg editor out of the box .. (with the obvious extension of adding an attachment as image).. do you know how that looks on my blog right now?

[ img ]/path/to/my/image.png" align="right[ /img ]

i use bugs (not to call them XSS vulnerability) of my own code to work around limitations ;)

i see absolute no way to do this with markdown or ReST? but i might be wrong?

(btw. i am actually still looking for a solution, i'm not saying xhtml + customization is the way to go, i am just not satisfied at all with the current markdown/bbcode combo :) )

Daniele Varrazzo said @ 2009-07-04 11:34:25:

I did some work on the experimental wysiwyg in sphene, and it was relatively easy: because I am converting an existing board, I used my set of smileys and hacked the editor to inject smileys such ": )" instead of [ img]: It was, as I mentioned, a matter of regexp. You can check the result to this board (sorry, you should register to be able to see the editor: I've not been able to use the captchas yet).

WOW, really nice what you have done there! i'm always amazed how polished SCT can look if done right :)

Daniele Varrazzo said @ 2009-07-04 11:34:25:

After this experiece, I may say that I can refactor the "Renderer" into an object exposing these interfaces:

- render a board message
- render a wiki page
- provide JS for a wysiwyg editor to convert html -> markup
- provide JS for a wysiwyg editor to convert markup -> html

hmm.. i'm not sure if there needs to be a difference between board and wiki.. although well.. i guess both require custom features.. but it somehow looks weird to have e.g. one markdown renderer which doesn't just renders markdown, but board messages or wiki pages, if you know what i mean

Daniele Varrazzo said @ 2009-07-04 11:34:25:

Notice that the JS conversion are only relevant to the current editor: it may be that using FCK or others they end up not being request: I will investigate into that.

i'm not sure if it is a matter of which editor is used, but more which browser is used. because all editors use the designMode of the browsers, which potentially generate different html output.. :/

Daniele Varrazzo said @ 2009-07-04 11:34:25:

btw, in our forum, you may see other features in the board: message icons, Italian translation and a method to choose the smileys set: the patches are not polished yet for submission to sct: I will ask you something else in other post and then provide you good patches (but if you want "bad patches", they are already available ;) )

it would be great if you find the time to contribute patches. although i prefer smaller patches than one big one so i understand what it does ;)