Gcode documentation change

zapta

Thanks @t3p3tony. The file is less than 1M and downloaded quickly and loaded easily into my browser.

As an example, I tried a page search search for 'acceleration' and the new model didn't hit M593 and M669.

When you search the single page, you know that it will hit every mentioning of it. With the new summaries page, it will hit only if that page was manually curated properly, so it reduces the confident in the search.

(If there is a hard technical constraint that prohibits the single page than I understand, that's psychics and we just need to live with it, but if not, please reconsider).

oozeBot

We just talked about this in our standup and agreed that we'd rather it all be on a single page for many of the same reasons already listed. It's an invaluable resource..!

bot

For those worried about losing the searchability of a single page: have you tried using the dozuki search feature?

Searching for, as an example from this thread, "acceleration," finds all the appropriate pages, and more.

What is missing from the individual G-Code page listings, in the dozuki search results, is a description of the contents of the page. Not sure why that doesn't appear, but the search function works very well to find all the g-codes that reference acceleration.

https://duet3d.dozuki.com/Search?query=acceleration

royal32

Is the old page still exist somewhere so we can find examples that worked well there but not in the new model?

You can access the last point in the page's history before it was broken up by adding ?revisionid=5349 to the url, like so:

https://duet3d.dozuki.com/Wiki/Gcode?revisionid=5349

I'll be using the locally downloaded version, but that bookmark will be useful if people need it on mobile devices etc while we figure out a good solution here.

Reviewing the initial reasoning for this change, I'm really not understanding any of them.
As far as slow loading times on either side... you've replaced a single page with all of the information on every gcode that measured 1.57mb (transferred/compressed) with a page of no information and just links that weighs 1.37mb, and every individual gcode page the user has to load weighs 1.34mb.
How does this change improve load times for either the server or the client, am I missing something? It seems to be significantly worse by my measurements. Am i fundamentally misunderstanding this?

As far as difficulty editing, If i were editing this I would prefer a monolithic document that i could Ctrl-F around in, rather than having to manage literally hundreds of individual pages. I managed a wordpress site for a few years, I would have lost my mind trying to do that. I can see that the editing window is very small, but you can Ctrl-A, paste it into something like vscode. It can fill up your whole screen, its color coded because vscode recognizes markdown, you get a minimap to help you navigate... thats how i would do it.

As far as getting lost while you scroll... i'm more concerned about getting lost in all of the different tabs i have open now with different gcodes. Yes it is a long document, but it is also very easy to search around and find what youre looking for between Ctrl-F and the index on the side.

Just my 2c.

Seth

zapta

@bot said in Gcode documentation change:

https://duet3d.dozuki.com/Search?query=acceleration

I rarely use the dozuki search because the results are kind of opaque and you need to enter each item to get an idea what is inside. Does it have a next-next mode where each click takes you to the page/location of next result?

This one provides more context but still not as functional as the one page search.

https://duckduckgo.com/?q=site%3Ahttps%3A%2F%2Fduet3d.dozuki.com+acceleration&atb=v88-4_g&ia=web

T3P3Tony

Thanks for everyone's feedback. One of the downsides of having a solution like Dozuki is that there are some aspects that are out of our direct control. One of those was having such a large and frequently accessed page was an issue for Dozuki. I am not sure of the architecture reasons behind why it was an issue but it was, not specifically for the file size transferred (so maybe for the server resource in rendering? not sure).

There have been some good suggestions on here on how to improve this (client side aggregation, related gcodes on individual pages, having a single global ToC, for the individual gcode pages, adding keywords to help search on the summary page).

droftarts

@bot said in Gcode documentation change:

What is missing from the individual G-Code page listings, in the dozuki search results, is a description of the contents of the page. Not sure why that doesn't appear, but the search function works very well to find all the g-codes that reference acceleration.

Ah, useful! It's because no summary has been put in for the individual pages. That's something I can fix. I've added them to G0 and G1, and now they show up. I'll add more when we've decided what to include.

@royal32 said in Gcode documentation change:

I can see that the editing window is very small, but you can Ctrl-A, paste it into something like vscode. It can fill up your whole screen, its color coded because vscode recognizes markdown, you get a minimap to help you navigate... thats how i would do it.

Thanks, I had a quick look at Visual Studio Code; I have been using BBEdit (on Mac) for editing large documents. Unfortunately, Dozuki's markdown version is not standard, so it doesn't preview nicely on VSCode (or BBEdit), so there's no major improvement. VSCode looks good for other reasons, though!

@zapta said in Gcode documentation change:

As an example, I tried a page search search for 'acceleration' and the new model didn't hit M593 and M669.

Thanks, I'll look at this. Summaries could include keywords, eg for M593 it could include "DAA, acceleration, frequency, damping" etc. 'Acceleration' in M669 (Polar kinematics Maximum turntable acceleration parameter) might just be a bit too esoteric to count as a keyword. Got to draw the line somewhere... Interestingly, M669 does show up in a Dozuki search for 'acceleration'.

Ian

Dr_Ju_Ju

What Browser was this update tested on ?? because personally I normally use a VERY restricted version of Microsoft Edge, with ALL the social claptrap etc removed...

Whereas in the 'old' version I could quickly get answers for any queries e.g. M586 network, but this 'new' version is a steaming pile of shite, where all I see a brief description of what M586 supposedly controls, but there are no links to any additional info etc. and this goes for ALL the G/M codes

A Former User

This post is deleted!

droftarts

@dr_ju_ju the G/M code at the beginning of the summary line is a link to the full entry.

Ian

Dr_Ju_Ju

@droftarts not for me it isn't.....

droftarts

@dr_ju_ju see https://forum.duet3d.com/post/251387

Ian

Dr_Ju_Ju

I don't know what has changed this morning, if anything, but I now see links to individual explanation pages from the GCode dictionary page (https://duet3d.dozuki.com/Wiki/Gcode#Section_G_commands), which at least gets me started in finding the info I need, but WHY are there no return links, to the dictionary page, on each of explanation pages ????

zapta

@droftarts, here is another example, searching the main gcode page for 'StealthChop' or 'Stealth' or 'Chop' brings nothing.

https://duet3d.dozuki.com/Wiki/Gcode

Searching that page for M569 doesn't help either since it has 8 entries which should be opened and searched individually, not to mention finding other commands that may mention 'stealthchop'.

I miss the old times.

alankilian

@t3p3tony

I have noticed a fairly substantial change in my ability to use the GCODE page after this change.

As I try to help people in this forum, I search the GCODE page for things like:

current (Which Mxxx sets motor current.
idle (Which Mxxx sets the idle current)
endstop (how are endstops configured)
pullup (How are pullups configured for I/O pins)
leveling (which Mxxx are involved in bed/mesh leveling)

None of those get me where I have to be to help Duet users with their problems and I have to use duckduckgo.

Which potentially reduces my ability to help your customers solve their problems.

I wasn't opposed to the change in the beginning, but as I use the page more and more, I'm now feeling the issues of just having one-line summaries.

I like to help people, but when the burden gets too high, I ignore their problems and move on to something easier for me to do.

oliof

@alankilian I've half a mind of creating a "classic" GCodes library page which directly includes the individual entries, but I don't want to cause issues with the dozuki.

@droftarts if the gcode entries would be properly linked under "Gcodes/" rather than on the wiki root I could create a local copy every now and then for easy access. Since they do reside at the top level, this is more tedious since I'd need to parse the main GCodes page for links to get local copies.

T3P3Tony

@alankilian thanks for the feedback. We are looking at alternatives as I agree this change makes the gcode dictionary more difficult to use.

@oliof there is now an "individual gcodes" category. the Gn codes have been put in there. the M codes will follow soon.

sinned6915

@droftarts said in Gcode documentation change:

To maintain existing links (yes, they should all still work), the headings have to be shown in the new page, as they appeared in the old page, without a link in them. This lead to the layout with the link to the individual page and description on another line.

Nope, not universally true. There are MANY that are formatted like this one-

https://duet3d.dozuki.com/Wiki/Gcode#Section_M558_in_RepRapFirmware_Num_3

this one works, but not the above one

https://duet3d.dozuki.com/Wiki/Gcode#Section_M558_Set_Z_probe_type

theKM

This post is deleted!

theKM

For what it's worth...

A script that puts a link at the top of the left hand menu, that if clicked, will re-assemble the single mega-page from all the separate pages.

It is written in a way that if included as a script tag in the wiki's root html template, it will only wake up on the correct page, and simply add a link to the menu. User can click it to assemble the mega page, or ignore it all together to leave it as-is. I'd recommend putting it in a file somewhere and referring to it so the browser can cache it so there's no download impact each page load.

For anyone that wants to run it themselves, just go to the gcode page, open JS console ( ctrl + shift + J for chrome, ctrl + shift + i for firefox ) , paste it in, press enter...

// ==UserScript==
// @name     GCode Wiki script
// @version  1
// @include     https://duet3d.dozuki.com/Wiki/Gcode*
// ==/UserScript==

/** for processing injected menu clicks, gets changed
 * to proper implementation when needed
 */
let processMegaPage = () => {};
let crossLinkDoc = () => {};
let setupGcodeBrowser = () => {};

let codeMap = {};
let crossLinked = false;

// for identifying gcode references
let codeRx = /[GM]+[0-9]+(.[0-9])?/g;

/** Helper function with finding things, so I don't need a framework
 */
let docE = (f, inp) => {
    if (Array.isArray(inp)) {
        return (inp.map(i => docE(f, i))).flat();
    }
    let seek = document[f](inp);
    if (!seek || seek instanceof HTMLElement) return seek;
    let tmp = [];
    for (let t of seek) tmp.push(t);
    return tmp;
};

let content = document.getElementById('content');
let isInContent = function (e) {
    if (!e) return false;
    if (!e.parentElement || e.parentElement === document.body) return false;
    if (e.parentElement === content) return true;
    return isInContent(e.parentElement);
}

/** Handler to take input and process links in gcode browser
 */
let ref = null;
let paster = () => {
    if (ref) clearTimeout(ref);
    ref = setTimeout(() => {
        let linkStyle = 'all:unset;cursor:pointer;text-decoration:underline;color:blue;';
        let v = docE('getElementById', 'gcode-paste-input').value;
        v = v.replace(codeRx, (s) => {
            if (codeMap[s]) return '<a href="#' + codeMap[s] + '" style="' + linkStyle + '">' + s + '</a>';
            return s;
        });

        docE('getElementById', 'gcode-paste').innerHTML = '<pre style="text-align:left">' + v + '</pre>';
    }, 500);
};


// check to see if this page is the one we want, otherwise do nothing
let loc = null;
let pos = null;
let hash = null;
let qstring = null;

const evalLocation = () => {
    loc = document.location + '';
    pos = loc.indexOf('#');
    hash = null;
    if (pos > -1) {
        hash = loc.substring(pos);
        loc = loc.substring(0, pos);
    }

    qstring = '';
    pos = loc.indexOf('?');
    if (pos > -1) {
        qstring = loc.substring(pos + 1);
        loc = loc.substring(0, pos);
    }
};

const moveToHash = () => {
    if (hash) {
        document.location = hash;
    }
};

// current location details...
evalLocation();

if (loc.endsWith('/Wiki/Gcode')) {
    // current page is the gcode wiki...

    // find the menu
    let topTitle = docE('getElementsByClassName', 'toc-title')[0];

    // add the link to run the mega-page import...
    let squirt = document.createElement('div');
    squirt.innerHTML = '<a id="create-mega-page-link" href="javascript:;">Create Mega-Page</a>';

    topTitle.parentElement.insertBefore(squirt, topTitle.nextSibling);

    setTimeout(() => {
        // take a beat, wire the event clicker (greasemonkey complication)...
        let tmp = null;
        (tmp = document.getElementById('create-mega-page-link')) ? tmp.addEventListener('click', () => processMegaPage()) : null;
    }, 50);

    // function handler for the click...
    processMegaPage = () => {
        squirt.innerHTML = '';

        // find all the specific links
        let glinks = docE('getElementsByTagName', 'a').filter(a => {
            let t = a.innerText;
            let h = '' + a.href;
            h = h.substring(h.indexOf('/', 10));
            return (t.match(codeRx) && !h.startsWith('/Wiki/Gcode') && (h.startsWith('/Wiki/M') || h.startsWith('/Wiki/G')));
        });

        // a function that represents the work for a page download
        let processor = a => {

            return new Promise((resolve, reject) => {

                // download...
                fetch(a.href).then(async response => {

                    // the document dext...
                    let t = await response.text();

                    // pull out the good stuff...
                    let from = t.indexOf('<div id="Wiki_Details">');
                    let to = t.lastIndexOf('<div class="clearer">', t.lastIndexOf('<div itemprop="author"'));
                    t = t.substring(from, to);

                    // dont need this link in there now...
                    t = t.replace('<p>Back to the <a href="/Wiki/Gcode">Gcode Dictionary</a></p>', '');

                    // make a div for it
                    const d = document.createElement('div');
                    d.innerHTML = t;

                    // replace the link tag's parent <p>
                    let p = a.parentElement;
                    p.parentNode.replaceChild(d, p);

                    // end of task
                    resolve();

                }).catch(reject);
            });
        };

        let count = 0;

        // a function that represents a processing thread so we can start N of them
        let runner = async () => {
            while (glinks.length > 0) {
                count++;
                if (count % 25 === 0) console.log(count);
                let x = glinks.shift();
                try {
                    await processor(x);
                } catch (e) {
                    console.log(e);
                }
            }
        };

        // start the runners...
        let runnerCount = 1;
        (async () => {
            let time = new Date().getTime();

            console.log('links to process: ' + glinks.length);

            let runners = [];
            for (let i = 0; i < runnerCount; i++) runners.push(runner());

            await Promise.all(runners);

            console.log('DONE!!! (' + ((new Date().getTime() - time) / 1000) + ' seconds)');

            moveToHash();

            squirt.innerHTML = '<a id="cross-linker-link" href="javascript:;">Cross-link gcodes</a><br>'
                + '<a id="setup-gcode-browser-link" href="javascript:;">Setup GCode Browser</a><br><br>';

            setTimeout(() => {
                // take a beat, wire the event clicker (greasemonkey complication)...
                let tmp = null;
                (tmp = document.getElementById('cross-linker-link')) ? tmp.addEventListener('click', () => crossLinkDoc()) : null;
                (tmp = document.getElementById('setup-gcode-browser-link')) ? tmp.addEventListener('click', () => setupGcodeBrowser()) : null;
            }, 50);
        })();

        crossLinkDoc = () => {
            if (crossLinked) return;
            crossLinked = true;

            let link = document.getElementById('cross-linker-link');
            link.parentElement.removeChild(link);

            /** Parse the references out of the document
             */
            docE('getElementsByClassName', 'header').forEach(sect => {
                let s = sect.id + '';
                s.replace(codeRx, sx => {
                    sx = sx.replace('_', '.');
                    if (!codeMap[sx]) {
                        codeMap[sx] = s;
                    }
                });
            });

            /** Cross-link gcode references through the document
             */
            docE('getElementsByTagName', ['p', 'li', 'pre', 'strong']).forEach(tag => {
                if (!tag) return;
                if (!isInContent(tag)) return;
                let linkStyle = 'all:unset;cursor:pointer;text-decoration:underline;color:blue;';
                let h = ('' + tag.innerHTML).trim();
              	if (h.startsWith('<div/')) return;
                tag.innerHTML = h.replace(/(?<!_)([GM]+[0-9]+(.[0-9])?)/g, (s) => {
                    if (codeMap[s]) return '<a href="#' + codeMap[s] + '" style="' + linkStyle + '" class="crosslink">' + s + '</a>';
                    return s;
                });
            });

            evalLocation();
            moveToHash();
        };

        setupGcodeBrowser = () => {
            crossLinkDoc();

            if (squirt) {
                squirt.parentElement.removeChild(squirt);
                squirt = null;
            }

            /** Strip the main wrapper element of styling
             */
            let wrap = docE('getElementById', 'page');
            wrap.className = '';
            wrap.id = 'gnavPage';

            /** Remove styling of sidebar nested element
             */
            docE('getElementById', 'sidebar-wiki-toc').className = '';
            docE('getElementById', 'sidebar-wiki-toc').id = 'gnavSidebar';

            /** Strip the sidebar
             */
            let sb = null;
            sb = docE('getElementById', 'page-sidebar');
            sb.innerHTML = '';
            sb.id = 'gnavSidebar';

            /** Strip the main area
             */
            let main = docE('getElementById', 'main');
            main.id = 'gnavMain';
            docE('getElementsByClassName', 'articleContainer')[0].className = '';

            /** Clear our other elements not needed
             */
            let mb = docE('getElementById', 'mainBody');
            for (let kid of mb.children) {
                if (kid.id !== 'contentFloat') mb.removeChild(kid);
            }

            let bg = docE('getElementById', 'background');
            for (let kid of bg.children) {
                if (kid.id === 'gnavPage') break;
                else bg.removeChild(kid);
            }

            for (let kid of document.body.children) {
                if (kid.id !== 'background') document.body.removeChild(kid);
            }

            docE('getElementById', 'content').id = 'offContent';

            /** Apply new styles to the wrapper, sidebar and main areas
             */
            wrap.setAttribute('style', 'display:flex;gap:1em;');
            sb.setAttribute('style', 'flex: 1 1 45%; height: 100vh;');
            main.setAttribute('style', 'flex: 1 1 50%; height: 100vh');

            /** Input field to paste the gcode
             */
            sb.innerHTML = `
<div style="width: 45vw; height: 100vh; position: fixed;overflow: scroll;">
    <textarea id="gcode-paste-input"></textArea>
  <hr />
  <pre id="gcode-paste" style="text-align: left"><center>( paste gcode above )</center></pre>
</div>`;

            setTimeout(() => {
                // take a beat, wire the event clicker (greasemonkey complication)...
                let tmp = null;
                (tmp = document.getElementById('gcode-paste-input')) ? tmp.addEventListener('keyup', () => paster()) : null;
            }, 50);
        };
    };
}

// auto-trigger...
if (qstring === 'inflateContent') {
    processMegaPage();
}