Contributing to projects hosted on github: a step by step howto, and an illustration with Marlin (3D printer firmware).

Git is powerful... and very painful the first time! / Github contribution howto / Tweaking Marlin for better 3D printer menus.

Git mess, only partially resumed by Oliver Steele.
Indeed, the upstream higher repository is not shown here,
as seen on this other bigger, clean and useful cheat sheet.

I eventually wrote a feature to fix something that annoyed me for years: allow multi-line commands in the LCD menus of Marlin, a very well known firmware for 3D printer.

I needed it for my Delta printer: these printer do require precise calibration, often with a sequence of gcode commands (check the end of this post for more and why I wanted these to be in my menus and not as initialization files on all my SD cards nor on a PC over a USB cable).

Now, Marlin is hosted on github, a community front end to many other open source projects. Actually, the linux kernel itself is developed with git so it works, for sure.

I often tried to use github, but I never went past a simple git clone of a repository. It was still better than to download a zip archive, because you can easily get the new stuff with a git pull. But here and partially out of curiosity, I wanted to try and contribute to a project at the source.

Now... what a huge and painful procedure just to give a hand! Seriously, it is mind boggling how much crap and megabytes need to be handled just to help and submit a few dozen f*king lines of code to an opensource project hosted on github. This is too bad since I am sure many programmers would be glad to give a hand to project they stumble upon (like me, often), but without this need to become administrators of complex and shared projects themselves!

This article is all about posting and contributing to a project hosted on github. You'll get the calibration g-code line I used at the end of the post if you really ask yourself.

This short introduction may be a good start, i.e. using git as a solo developer without a "mother" project that you have no right on it. But we want here to contribute to someone else's project.

Almost all github existing projects rely on the fork and pull model. Else, you probably know better than me when you are allowed to work directly in the repository of others ;)

So the deal is that git requires you to make yourself a full copy of the project, both remotely and locally. Then you work locally, you commit locally, then again remotely, and then tell the project maintainer that your copy has a change that may be interesting to them. Powerful, but convoluted for sure as it also means you need to get the changes from their own project that also moves on in between.

Anyway git and github are painful enough for me to record the full procedure here, and hopefully once for all. It may help some old school developers as me, and may be it would let them contribute more easily to open source projects hosted on github.

If this post is TL;DR, you may like this small and raw graphical resume instead.

Initial setup: create your own version of the original repository

OK. For now you want to use your browser at https://github.com/

1) You first need (to create) a github account (not that much painful so far, but it already gets annoying haha)

2) Then go to the github repository you want to work with.

For Marlin, the main one seems to be https://github.com/MarlinFirmware/Marlin.git

You will see that there are tons of secondary variants made by tons of people so make sure you are working on the "right" one. Now, this is really annoying to me when your goal as here is just to try to submit a piece of code to the "original" authors. And no, you have to create your own repository first!

This just due to the way github works: as a matter of fact we have to create yet another version of Marlin!

3) Now, still in the web page, click and fork the official Marlin repository (it will create copy in your account... and add to the mess for newcomers!).

4) Make sure to rename this copy to a meaningful name. Be it for you or for others, it will prove very valuable! Just click on the tiny almost hidden "project settings" icon in the bottom right margin (see below), then rename your project.

Now get back to your main project page by clicking on the name of the repository, then copy the content of the small box in the bottom right margin, labeled "HTTPS clone URL". This is your repository URL, its ID card in a way.

5) Now back to the console!

WARNING! You may want to use your public SSH keys to log in (so that you are safe and do not have to provide your login all the time). In this case, do NOT use the above link, but tweak it to this syntax instead:
  git clone git@github.com:MoonCactus/Marlin-JFR-features.git

Else, to use other means to identify yourself on github, you may use the copied URL as is:
  git clone https://github.com/MoonCactus/Marlin-JFR-features.git

Of course you want to specify your own repository, not the one you forked from.
Also, make sure you are using https and not http or will will make your life harder later on.

Now you can code... (but the pain is still to come!)

Now the funniest part: start coding and hacking your stuff around... Oh you did already, didn't you?

And you though about committing your changes to git afterwards, right?

In fact, I use a comparison tool like meld to re-incorporate my changes to a forked repository after I usually start working "just to try" on a cloned repository. There should be one single command to switch from the latter to the former... Alas, do not forget: Git was first, and only then came the source code... ;)

Now, it is time to submit your changes, and here it becomes annoying again.

Commit your changes... multiple times and all everywhere ;)

When you are done and want to save/store/submit/commit your changes, you have different commands to issue (no way it will never be as dead simple as a dumb cvs commit!).

• git status -- a useful command to check the state of your repository

E.g. here it tells me I modified three files in the project:

  On branch Development
  Your branch is up-to-date with 'origin/Development'.
  Changes not staged for commit:
   (use "git add ..." to update what will be committed)
   (use "git checkout -- ..." to discard changes in working directory)
        modified:   Marlin/Marlin.h
        modified:   Marlin/Marlin_main.cpp
        modified:   Marlin/ultralcd.cpp
  no changes added to commit (use "git add" and/or "git commit -a")

• git add -- Tell it you changed something (!)

You first need to commit your changes locally with a git add followed by the name of files that changed (listed above):

  cd Marlin

  git add Marlin.h Marlin_main.cpp ultralcd.cpp

• git commit  -- Now tell it to record that you changed something, still locally (!)

In fact you may merge the add + commit commands by typing git commit -a instead.

But no, it will not be uploaded yet. It may be useful though, so that you can work with a local history and you can work offline. But that is nonetheless more steps to the whole procedure (what the heck is then the need for an index and a git add really?). Anyway here it goes:

  git commit

  [Development dd301be] Added suport for multiline G-code commands in the

  LCD menus

  3 files changed, 43 insertions(+), 4 deletions(-)

As you seen it asked for a message related to the changes. Better write something useful because it will show in the history, and it helps you reverting to an older revision of the project when you screw it all, and it helps anyway to understand what you did months ago!

• git push  -- At last, send it to your repository!

Hooray? oh no, not at all!

First, it will ask for some more obscure options. Better do the following once before you push your changes:

  git config --global push.default matching

Then again:

  git push

But now it asks for your github credentials. Here I typed them manually:

  Username for 'https://github.com': MoonCactus

  Password for 'https://MoonCactus@github.com': XXXXX

  Counting objects: 6, done.

  Delta compression using up to 8 threads.

  Compressing objects: 100% (6/6), done.

  Writing objects: 100% (6/6), 1.22 KiB | 0 bytes/s, done.

  Total 6 (delta 5), reused 0 (delta 0)

  To https://github.com/MoonCactus/Marlin-JFR-features.git

     865ca0e..dd301be  Development -> Development

At least it worked! My remote repository on github is now updated with my local changes. But how do I get the remote changes from the original repository, and further, how do I tell them I added a feature to Marlin?

Credentials and github

Don't we just want to submit one patch and not to administrate repositories? No way...

As we have seen, git push asks for your credentials (login and password). And it is more likely we will be making changes and submitting them more than once.

To avoid having to type it all the time, you may check the many options given in this post.

If you are using a desktop PC and use an encrypted user directory, then you may take the risk to store your password in your ~/.netrc special file (I have no clue on windows, sorry there!).

But you would better go the safe way and upload your public SSH keys to github. This way it stops asking and you stay "fully" secure.

You can do that here on the web:

  github / edit profile / SSH keys (aka https://github.com/settings/ssh)

There you can add your own "*.pub" SSH key. Check this help if you need more about SSH keys, or if you want github to generate some for you because you do not have some already (this is not recommended imho).

Then, check you are OK when ssh -T git@github.com no more asks for your login.

But still... for SSH login to work, you must have cloned your project NOT with the given https repository link, but the following way instead. Else you are screwed and need to download the project all over again, by using:

    git clone git@github.com:MoonCactus/Marlin-JFR-features.git

Did I say git was just plain painful to set for newcomers? Well, at least you can first commit your existing changes in the "htpps style" by providing your login to the git push, then delete the local project and re-clone the project as just above.

So once done you can check your local and remote projects are synced:

  git status

  On branch Development

  Your branch is up-to-date with 'origin/Development'.

  nothing to commit, working directory clean

And git push should no more ask your ccredentials

  git push

  Everything up-to-date

Getting remote changes down to your local version and "syncing a fork" (sounds weird, eh?)

It is easy to get the changes from your repository to another PC. First you must clone the repository the same way you did above to your other PC. Then when you want to download the changes that were pushed from elsewhere, just do this:

  git pull

But... this only gets the changes from your own repository, and not from the initial upstream repository. As usual git does not make it easy. Unless it is a one-time help you are giving to a project and you are OK to redo all the mess above months later, syncing with upstream is quite important. Otherwise and with time, your project will drift away from the "official" build to the point it will be much harder to get and merge the changes while dealing with increasing discrepancies and even conflicts. Submitting patches to the original authors will be even more difficult.

You first need to tell that your version should also look for changes from within the original project you forked from (the upstream).

OK let us try to finalize our stuff so that it will also get the changes from the upstream.

Get its original URL from github, and add it to the local copy this way:

  git remote add upstream https://github.com/MarlinFirmware/Marlin.git

Now we check all the repositories which changes will be taken from:

  git remote -v

  origin  git@github.com:MoonCactus/Marlin-JFR-features.git (fetch)

  origin  git@github.com:MoonCactus/Marlin-JFR-features.git (push)

  upstream        https://github.com/MarlinFirmware/Marlin.git (fetch)

  upstream        https://github.com/MarlinFirmware/Marlin.git (push)

Now, to download specifically the changes that are made in the original repository into your repository, we need more obscure and brain-damaging commands:

  git fetch upstream

  From https://github.com/MarlinFirmware/Marlin

   * [new branch] Development -> upstream/Development

   * [new branch] revert-1350-Development -> upstream/revert-1350-Development

Branches? Only monkeys need branches!

Or... a typical hellish source code sharing system!
From this answer at stackoverflow.

Branches allow a project to move on while a stable version of the sources is kept. This is mostly for safety and to "freeze" the state of a project in a "stable" state, where only bug fixes are made. New features are made only in a development or experimental branch. Then you "simply" switch between the development and production branches, according to the current work that need to be done (do not get confused!). Indeed, when you want to make it funnier, you can create many branches, give them obscure names and get changes from one withing another and so... But branches are another hellish topic, i.e. advanced uses of git that I do not need here (e.g. this post).

Our own sub-project may not need branches... But the original project does have a few, so we need to deal with them a little bit at least.

In our case, the branch named revert-1350-Development looks enough dubious to me that I will not even go and try to get more information about it... I will assume the first one is the only useful one. Surprisingly, there is no "stable" or "production" branch for this project!

  git merge upstream/Development

If you need to deal with local branches, and not-yet committed changes, then you need to read about more gory details (drowning a spoon, oh no, syncing a fork).

Or check this useful, short but comprehensive dynamic cheat cheat.

Note: git comes with a very useful interactive tool based on a local web server (default is to use the light lighttpd that you may have to apt-get install lighttpd first - but it runs also also with apache, see the git reference here). Just move to your repository and type git instaweb. It will launch a web navigator, that lets you parse and compare branches and history in depth.

git instaweb is a very nice GUI to parse an existing repository!

Pull requests, aka submitting a change to the original authors... at last!

Everyone benefits from a project that is maintained in a good shape. This is also why there are often separate "stable" (even sometimes obsolete), and development branches. The stable branch ought to be fully functional, with only bug fixes pushed into it.

So, make sure your own stuff works well! Do not submit broken stuff to programmers or they will hate you :)

Also, make sure your new feature or bug fix is useful to others, and that it neither breaks something far far away in the original code source... or they will hate you alike.

You need also to comment it properly and follow the guidelines (code style and so). Anyway, your patch will probably be refused if it does not seem good enough for the maintainers! And they may also hate you ;)

So enough joking... the deal is to tell the maintainer that your branch has something new, that they can grab and integrate it in their own (upstream) repository. This is mostly why your changes should be minimal: be as close as possible to their latest versions before you submit your so-called pull request.

Strangely, the pull request must be made from a web browser, not from the console.

Actually here the official documentation is linear and easy to follow. May be this is because it moves a bit away from the overly complex guts of git into something much easier now (and I really am a console guy though!).

So the next steps is to ask the maintainers to integrate your change. And this is made straightforward in the official documentation this time.

> Still, feel free to ask when you want me to explain it further in this post !

I just hope this post could help a few developers to contribute to open source projects.

A sierpinsky triangle... made with git by BooK !

A word on merge and conflicts

While I did minor modifications to my pull request to match expectations, the master evolved. Which produced conflicts in a few files. So I had to tackle merging the two sets of changes.

Tell git about your preferred comparison tool, which is the excellent meld for me on Linux, that also does 3-way merge nicely:

  git config merge.tool meld

Then to get up to date with the remote source:

  git fetch upstream
  git checkout Development # may generate conflicts
  git stash # in which case this saves your changes aside, see here

  git mergetool # open your interactive merging tool, to solve the conflict

Note that the file shown in the middle is the common ancestor to "theirs" and "ours". There is no need to change it as it is only included for reference and to better understand/solve the conflicts. So the deal here is to take bits from "theirs" (right file) and insert them in "ours", while not overriding our own new features.

Once the changes are merged/solved, you can commit the result and proceed:

  git commit

  git merge upstream/Development

Now check the status of our version with git push. In my case, I suspect that my editor removed spurious white spaces, and it resulted in 174 commits, but in fact it was the files and modifications that got added by the upstream into my own repository -- no harm!

  git status

  On branch Development

  Your branch is ahead of 'origin/Development' by 174 commits

  (...)

But still, there were two configuration files I inadvertently pushed into my pull request, that I wanted to revert to the remote pristine "upstream" versions (i.e. drop all my own changes on them). I followed information from this post: (the easiest is to check the pull request file list in a browser, e.g. here):

  git checkout upstream/Development -- Configuration_adv.h Configuration.h

Had to do this for a few files (huh?):
  git reset HEAD Configuration_adv.h Configuration.h
  git commit -a

And finally (had to git add a few files btw):

  git push # publish your local commits

Your "saved" differences are still there:
  git stash list
  stash@{0}: WIP on Development: 85e5aa4 Generalized enqueue_commands_P, and moved them to Marlin_main as they should

OK, OK, let's use branches, then...

Actually I learnt the hard way that a pull request is a good time to create a new branch... This way, the existing work is "saved" in a known state (i.e. the pull request), if I ever it needs to be re-worked. And at the same time it makes it possible to move further in other developments:

  git checkout -b for_RMUD # weird name eh? but I know exactly what it means!
  Switched to a new branch 'for_RMUD'

Here I created a new branch with a nice name (!) that will hold variations that are specific to my own needs, but that I do not want to commit to upstream. As shown, creating a branch sets you in by the way.

So we can now apply the stashed changes we recorded in the previous paragraph:
  git stash apply stash@{0}
  Auto-merging Marlin/Marlin_main.cpp
  Auto-merging Marlin/Marlin.h
  Auto-merging Marlin/Configuration.h
  CONFLICT (content): Merge conflict in Marlin/Configuration.h

When we get another conflict, we solve it again with git mergetool, as previously.

Addendum: add your own LCD controller menus in Marlin

Just clone the source code from https://github.com/MarlinFirmware/Marlin.git , then open ultralcd.cpp and to the function named lcd_prepare_menu() for example.

There you will see commands like this:

  MENU_ITEM(gcode, MSG_DISABLE_STEPPERS, PSTR("M84"));

This just adds a command named thanks to MSG_DISABLE_STEPPERS (check the translation strings in the project files!), and that inserts the M84 command in the current buffer that is being processed.

But the existing state of Marlin will not allow more than one command.

My Delta printer used M7XX commands to calibrate the level of its bed. These functions have since disappeared from Marlin (replaced by something else). But the full story was:

  M702 ; reset the leveling bed values

  G28 ; home

  G1 X-77.94 Y-45 Z36 F8000 ; move above first corner

  G4 S3 ; wait for 3 seconds

  M701 P0 ; record first calibration point with the Z probe

  G1 X77.94 Y-45 Z36 ; go to second calibration point

  G4 S3 ; and so on...

  M701 P1

  G1 X0 Y90 Z36

  G4 S3

  M701 P2

  M700 ; compute and store the leveling values

  G1 X0 Y0 Z100 F8000

This was definitely not one unique g-code command... With the patch I have just submitted I now write it this way:

    MENU_ITEM(gcode, "Calibrate bed", PSTR("M702\nG28\nG1 X-77.94 Y-45 Z36 F8000\nG4 S3\nM701 P0\nG1 X77.94 Y-45 Z36\nG4 S3\nM701 P1\nG1 X0 Y90 Z36\nG4 S3\nM701 P2\nM700\nG1 X0 Y0 Z100 F8000"));

I also added a "hovering/spiral" triangular movement above all three corners to check that my bed is level. First pass is 1mm above bed, second pass is 0.3mm above bed, and slower:

    MENU_ITEM(gcode, "Check level", PSTR("G28\nG1 X0 Y0 Z1 F4000\nG1 X-77.94 Y-45 Z1\nG1 X77.94 Y-45\nG1 X0 Y90\nG1 X-77.94 Y-45\nG4 S2\nG1 X-77.94 Y-45 Z0.3 F2000\nG1 X-77.94 Y-45\nG1 X77.94 Y-45\nG1 X0 Y90\nG1 X-77.94 Y-45\nG1 X0 Y0 Z0"));

Finally three more commands, to insert or retract the filament:

    MENU_ITEM(gcode, "Retract filament", PSTR("M302\nM82\nG92 E0\nG1 F4000 E-800"));

    MENU_ITEM(gcode, "Insert filament", PSTR("M302\nM82\nG92 E0\nG1 F4000 E60"));

    MENU_ITEM(gcode, "Finalize filament", PSTR("G1 F4000 E790"));

Once done, you must run the Arduino IDE, open your modified Marlin project, rebuild and upload the firmware to your printer. And here it is, a nice and tailored firmware menu entry:

Improved menu entries in Marlin with multiple g-code commands.
This is very useful to calibrate the bed of my Delta printer.

Without the patch to Marlin, I had to save these to separate files on each of the SD cards I could use with the printer, or to send them over USB to the printer from a PC, which I do not like at all. Another solution would have been to write a dedicated command in ultralcd.c and attach it to a menu entry.

What about Printrun custom buttons? Multi line commands support alike!

By the way, I used Pronterface a lot during the initial calibration of my printer. It has very convenient "custom buttons" but they suffer from the same limitation: you can only give them one g-code command!

In fact, copy/pasting a sequence of multiline commands does work, until you restart restart Pronterface since it then keeps only the first line. This was quite annoying because of the lng sequence I used to calibrate my printer.

It can be tweaked very easily, by adding the line in bold to the following function in printrun/pronsole.py:

    def precmd(self, line):

        line= line.replace("|", "\n") # JFR allow multiline GCODE button

        if line.upper().startswith("M114"):

            self.userm114 += 1

        (...)

This way, custom buttons may hold "multiline" commands, by separating them with a "|" (pipe) in place of a newline "\n". This tricks lets Pronterface reload the buttons with no additional fix to its source code.

update: oops. The author just told me I could instead have used the "macro" option, and then have added a custom button that plays the macro. This also does keep the newlines unchanged. Still, it is a bit counter-intuitive that the custom button shall be restricted to a single command imho.

And no, I will certainly NOT set up an entire github fork just for one single line of source code!  :D

This is a modified Pronterface that allows multiline gcode commands in the custom buttons!