Updating the Tidal website and documentation

[ndr_brt]

@yaxu

Put reference documentation for each function into the source. Compile that into a folder with a flat structure, one file per function, e.g. jux goes in ref/jux.md no matter where it is in the source. The only exception is internal functions not intended for use outside the source, which could be somehow tagged and compiled into ref/internal/nameoffunction.md. An index to functions can then be generated, with a separate list of documented internal functions for the curious.

Could be a nice approach, I took a look at the haddock docs, markdown is not a supported format.
Maybe a first version could be obtained scraping the hackage haddock htmls (Directory listing for tidal-1.9.4 documentation | Hackage) taking every function and converting its content to a markdown file (html to markdown conversion should already exists in the oss world)

[munshkr]

Looks like Pandoc can be used to convert Haddock markup (and MediaWiki too) to Markdown.

[yaxu]

Yes Pandoc is great - we could handroll a custom website builder using that rather than mkdocs/docsify.

I use it for generating this PDF from markdown: https://github.com/yaxu/pihkal/blob/master/pihkal.pdf, source here: https://github.com/yaxu/pihkal - that book project is a bit stallled but could be repurposed towards a collective documentation effort.

It would be really good to have something that doesn't require installing anything to build the website though.. So:

• reference documentation comes from source code

  • con: the documentation already there is incomplete and out of date so stuff needs bringing across from the wiki
  • con: harder to edit, and not natively in markdown
  • con: needs periodically building into markdown using a pandoc script or similar
  • pro: potentially easier to keep up to date
  • pro: hackage will have docs in it
  • pro: potentially easier to have in-editor context-sensitive documentation

• reference documentation written natively in markdown

  • pro: documented in the same way as everything else
  • pro: no extra compile step
  • con: miss out on above pros

• other documentation in markdown, via static html generator

  • pro: easier on the browser?
  • con: needs rebuilding every edit (although github pages could do this for us, it takes a little while)

• other documentation in markdown, using a javascript thingie to render markdown

  • pro: no tools need to be installed/run for edits to tutorial etc
  • con: ?

[munshkr]

@yaxu

other documentation in markdown, using a javascript thingie to render markdown

• pro: no tools need to be installed/run for edits to tutorial etc
• con: ?

I guess a con would be that it might rank lower on searches, unless we use SSR (server side rendering). Apparently Docsify supports this, but you would need a proper server and won't work with Github pages/Gitlab pages/Netlify anymore.

[yaxu]

Aha, that seems quite a large 'con' if it's less searchable/discoverable. So maybe pre-rendered flat files are best, but using standard enough markdown that github is a good enough preview.

There's also the issue of how to generate visual/audio examples. For that book project I'm generating them with a pandoc plugin.

[munshkr]

@yaxu

There's also the issue of how to generate visual/audio examples. For that book project I'm generating them with a pandoc plugin.

For this maybe we could adapt your plugin (render.hs) to an mkdocs plugin for example (in case we end up using mkdocs). It should be very similar, but rendering Markdown or HTML instead of Latex code.

Now I wonder if rendering audio on CI is possible at all...

I can try doing some tests and see how far it goes.

[cleary]

@Raph

I've spent quite some time on the french translation a year ago. It's far from perfect but at least it's something to start from. I think that nobody uses it though

You might be surprised! ping @th4

https://club.tidalcycles.org/t/translating-the-tidal-userbase/2329

[jwaldmann]

@munshkr

rendering audio on CI

related: nonrealtime synthesis (output to audio file) · Issue #410 · tidalcycles/Tidal · GitHub

[yaxu]

Ok maybe we take this in two stages:

First we convert all wiki documentation (including all translations) to markdown, including tutorial and reference material, with the latter having one file per function. We render that to html organised into a new website with something like mkdocs, integrating with discourse for discussion as per the patchbox os website.

Second we look into moving the reference documentation into the code, and then writing a converter to turn it back into markdown.. Or some other way of getting the docs into text editors and hackage.

[yaxu]

This script looks useful for converting from mediawiki to markdown, including putting edit history into git: https://github.com/peterjc/mediawiki_to_git_md

I'm just doing a trial run now.

Shall we try mkdocs first? It's good to have a few people in this thread up for helping - how shall we break up the tasks?

[yaxu]

Sorry for the multiple posts.

Here's the output:
https://github.com/tidalcycles/tidal-doc/

I ran the script twice, once with the deprecated markdown_github flavour, and again with the newer gfm (github flavoured markdown) flavour. They came out pretty similar.

I didn't keep the change history so far, I'm not sure if it's useful and was hitting strange git errors..

[plamere]

hmmm... getting a 404 on that github link

[munshkr]

@yaxu

It's good to have a few people in this thread up for helping - how shall we break up the tasks?

I can start preparing the main mkdocs project if you want. Should I use this new tidal-doc repo or a new one?

I guess we will need to make some modifications to this mediawiki_to_git_md script too, to make sure translations and sections/subsections are stored in subdirectories, generate the YML config files with the ToC etc. Besides that, output looks good.

[munshkr]

The repo was private, link should work now.

[yaxu]

That would be great if you could look at mkdocs. Feel free to use the tidal-doc repo. Thanks for making it public (not sure how I managed to set it private)

If you want to re-run mediawiki_to_git_md, the mediawiki dump is in wiki-export/tidal_dump.xml.

True there probably are a couple of things we want to change in the script, e.g. also paths for links. I made a fork here for us https://github.com/tidalcycles/mediawiki_to_git_md

The only change I made so far was to switch to markdown_github to gfm which we probably want, and to explicitly make it a python2 script (I don't think it works in python3).

[munshkr]

Cool. I've set up mkdocs there, with the Material theme (the same the PatchBox OS docs uses), it has a lot of features out of the box and colors, fonts, logo, etc. can be customized easily, so it looks like a good starting point.

I tried publishing to GitHub pages and it says that it was published on https://pages.tidalcycles.org/tidal-doc/ but I don't think it's working correctly because it goes to the wiki. Is there something to configure on nginx or the DNS?

[yaxu]

Great I pointed http://doc.tidalcycles.org/ there for now

[yaxu]

I fiddled about with it a bit more, adding a community page.. What other tasks are there?

• Get translation working
• Work out how to get the mediawiki exports generating nicely and integrated with translations
• Check over all the translations..
• Integrate this discourse forum, with a hidden thread per page as per https://blokas.io/patchbox-os/docs/first-run-options/
• Work on visual web design
• ?

[yaxu]

Do you think it's best to fork the Material theme like patchbox has done? Or can we override some things while keeping updates? Not sure how best to integrate the discourse comments

edit the language selection stuff in Material, among other useful features is only for sponsors.. I sponsored for a month to get access to it

[yaxu]

To answer my own question it seems customising themes is easy enough https://squidfunk.github.io/mkdocs-material/customization/

However I can't work out how to make use of the 'insiders' stuff, in particular language selection. I subscribed for $10 but don't want to stay subscribed forever.. I tried just copying it into https://github.com/tidalcycles/tidal-doc/tidal-doc-theme, and pointing to that in https://github.com/tidalcycles/tidal-doc/blob/main/mkdocs.yml, but the language selector isn't appearing.. Hm!