Experience with having a non Dist::Zilla user contribute to a Dist::Zilla project (it's not hard for them or me)

I've heard many times that Dist::Zilla makes it harder for people to contribute to the project. This is not true, it is either unfortunately either ignorance or FUD (much like saying Linux is harder to use than Windows). Truly, there are things that some dzil users do that can make it harder, but it doesn't have to be that way. Michael Schwern recently contributed to one of my dzil projects without ever realizing I was using dzil, until I told him. He more recently stated on twitter, "While your solution works, it seems like it makes more work for you to shield contributors from dzil". This was true in this case because I wasn't sure how to effectively move a series of multiple patches, I now know it's easy to do with git. Here's how you can allow contributors to contribute to your dzil project without causing you or them undo pain.

1. Don't use anything that changes the line numbers of your source

   Examples: don't use DZP::Prepender. Use modules like OurPkgVersion to insert VERSION, and make sure your # ABSTRACT and any pod is below the 1; at the end of your module. This will keep the line numbers of errors in your code from being different from the final build. It's still ok to use PodWeaver as it will save you way more time than it'll hurt, so long as you follow the rules about pod being at the bottom





2. Use a plugin that commits builds to your source control software

   I personally recommend DZP::Git::CommitBuild









3. Make your build branch your default branch

   I'm not sure how to do this with just git, but on github you can go to the admin section of your repository and change the default branch there. This makes it so that when someone clone's your repository the initial checkout is of your build branch. Your build branch shouldn't require dzil, it's the final build.

Now that you have your final build branch as the default branch anyone who wants to contribute can simply clone your repo and start hacking. Their are a couple of mistakes they could make, They could either change or add files that are maintained, or pruned by dzil. This did happen when Schwern sent me patches, one of them was the addition of a .gitignore, which I already had, but is being pruned out of the build branch. I can also see it happening to meta's and makefiles. These patches can simply be rejected as unnecessary, dzil-ified, or if they are truly a bug, then they can be reported and fixed upstream.

So what if someone sends you a pull request from build/master? well if it's just one or two patches, you add their remote, and do a git remote update and then you can git log remote build/master to find the sha1 of this patch. Now that you have the sha1 all you have to do is git cherry-pick [sha1] and it should apply. If there are any conflicts you may have to resolve them with git mergetool. However, the only conflict with Schwern's patches for me was the .gitignore, all other patches applied without assistance and applied correcly, surprisingly even the pod patch applied without issue.

If you have more patches than is comfortable with git cherry-pick then you need git rebase. The command you want is git rebase -i --onto master [sha1 before first sha1 in series] [tip of remote branch checkout]. So in my case git rebase -i --onto master 3f1e3748 schwern. What this appears to do is ends up rewriting my local schwern/build/master checkout and removes all the build commits, and then applies the patches on top. This means I can now do git merge schwern from the master branch, and all of his patches that I want will be successfully merged. For more on this strategy you may want to read this stackoverflow question.

given this is not quite as easy as a git pull that's a fast forward, but reality is it's not that hard once you know how to do the rebase and how it works. Of course this isn't ideal for constant contributors, those should simply learn to use dzil, but for the random contributor it should be ok.

My solution to not using PluginBundle:AUTHOR for dzil is git

I am now tired of updating my dist.ini's for my Dist::Zilla projects. For many people this is when they start using a PluginBundle with their authorname. I discussed why you shouldn't do this a year ago. Now that I'm tired of managing my dist.ini's on an individual basis I'm going to show you how I'm going to solve the same problem everyone else is, which is getting tired of updating your dzil configuration for all of your projects. I'm using git to do it. You probably haven't considered this, or am thinking I'm wrong because you believe that git can't merge branches without a common history, which is not true. I did it with Regen2, Funtoo, Portage, and Sunrise, which have way more files than any perl repo, including Perl.


First let's talk about the advantages and disadvantages of doing this with git.

I'll start with the disadvantages. You'll have another repository to manage. Git isn't completely automatic, you'll have to remember to merge your changes. You'll also have to add the repository to your existing repositories. You will have to resolve merge conflicts at least once, and probably occasionally more, though most should be fast-forwards.

The advantages are... now the changes to your dist.ini are getting recorded in your history. You can now have a master dist.ini, but remove items without filters in your individual modules. You can share even more configuration as merged differences allow you to maintain differences in downstream commits. It doesn't automatically update all of your modules. Wait didn't I list that last one in disadvantages? Yep, it's an advantage because what if you update your PluginBundle and that update breaks one of your modules, but you don't know it because you haven't worked on that module in a while. You can use this for more than just dist.ini.

Ok so the first thing you you want to do is create your master dzil repo. This is not a git or dzil tutorial so go do that. I did it by creating a dzil new project and removing the files I couldn't use and making a few tweaks and amending all of those changes to the initial commit. Check out my repo for inspiration. Remember the directory structure has to be the same as a dzil repo structure for any common files.

Now add a remote from your new repo to an existing dzil project. Next you need to merge the branch from the remote into your project. The first time you do this you'll have to resolve conflicts. If they're like mine then they'll be easy and obvious. Once that's done future changes will be fast-forwards unless you change something in your perl module repo, then you might end up with a simple merge conflict. One thing to remember NEVER EVER try to merge from your module repo to your dzil repo, it will cause you extreme pain in the future. cherry-picking that direction is possible but not merging.

After that, all that is left to do is run your dzil tests and fix any breakage in your module. Happy merging.

Writing a simple Dist::Zilla::Tester test

Hopefully, someone will use the blog post to write an actual doc patch, seeing as how this is undocumented.

I finally wrote A test for DZP::OurPkgVersion with the help of CJM. So I figure it's best to share the knowledge imparted upon me to all those who are writing plugins without tests.

Before we get started I'm going to advise that this test will only check the output that dzil built, if you need it to test anything more sophisticated, you'll have to learn more.

First you'll want to create a corpus repo like /corpus/MyDZTRepo with a basic minimal repo. This repo is simply a repo that you are using to test your plugin against, to make sure it works right. You put it in corpus so that if you have tests that you have to check in your corpus, those tests themselves aren't run when the test suite is run. The dist.ini doesn't need to contain anymore than the basic stuff needed to build. You .pm files need not have anymore data than what you're going to need to make your dzil plugin do its job. In the case of DZP::OurPkgVersion I only needed to test that the output found the # VERSION string correctly in a couple of scenario's. So that meant having # VERSION in the .pm's and [OurPkgVersion] in the dist.ini.






First DZT (Dist::Zilla::Tester) doesn't provide any tests of its own so you still need to use Test::More or some other testing framework. Next you need to initialize the tester object by telling it where the root of your corpus repo is. After that, unless you need to do other work, you can run $tzil->build so that the build is run.

So now lets slurp a file into memory so we can check to see if it was built right. You'll want to look in 'build/*' as the basic root of the build directory. So 'build/t/test1.t' if you need to slurp a test.



Now that we've pulled our build files into memory lets code up what the result should be. We can just do this with a simple heredoc, obviously you can do it another way.



Now that we've gotten that, all we have to do is compare the file that we expect dzil to output and the file that dzil actually built. This is just standard Test::More



Now let's take a look at it all together.


Pretty simple huh? Hope this means more dzil modules getting tested now. Including more of mine.

Creating new projects with dzil new and templates

Here I talked about creating a new catalyst project using a minting profile for Dist::Zilla. If you don't know how to create a minting profile read that first. I'm sure once you've tried that you'll agree that having a little bit more than the basics in a newly minted dist would be a good thing.

first we need to create our profile.ini correctly (note: if you've got [DistINI] plugin loaded you'll probably want to remove it) Now you can put any file in the subdirectory repo of your profile (if you leave out 'include_dotfiles = 1' then anything beginning with a . won't be included), and it can be a template using Text::Template. Dist::Zilla uses {{ }} for Text::Template Delimiters.

Let's start with adding a .gitignore file (if you're using git) we can create {profile}/repo/.gitignore the {{$dist->name}}* will exclude the directories and archives dzil creates on release and .build will of course ignore the .build dirctory.

Now for a more complex issue, creating a Changes file that has the the {{$NEXT}} variable in it to insert the date and such on build. Obviously you can format your Changes file however you want.

Now we want to create a much more complicated dist.ini All of the stuff between the first set of {{ }} is boilerplate mostly taken from the DistINI plugin so that we can use our settings from our config.ini, I really wish there were some convenience accessors for this. I also wish we had an arbitrary stash we could use in config.ini so I wouldn't have had to hardcode my username in this. I think it's all fairly self explanatory beyond that. Of course you can set up your dist.ini anyway you want. Also if you use this format you have to have your module's repo name on GitHub in camel case like it is on CPAN.

Announcing Dist::Zilla::Plugin::Catalyst

So I just recently finished reading Restful Web Services and decided I wanted to go back and play with Catalyst and REST some.

The original way to create a Catalyst skeleton is to run catalyst.pl MyApp. This creates a lot of nice files to get you started. dzil new basically does the same thing for a generic cpan module. Honestly, without any plugins dzil new isn't that useful. However, once you add Git::Init , you remove several steps from the creation of a new module and repository. Git::Init also makes your first commit of everything it added. I got to thinking why on earth would I want to do the following to get a cat module going and convert it to dzil.

catalyst.pl MyApp && cd MyApp
vi dist.ini # and add numerous lines
rm Makefile.PL README t/* # and maybe more since dzil is better at managing these
git init
git add .
git commit

when I could just be doing this

dzil new -p catalyst MyApp

A lot simpler huh? to get you started you need a few things but then creating cat apps will be easy.

first you can run dzil setup or if your version of dzil doesn't support that yet, you need to create the following ~/.dzil/config.ini file by hand obviously fill in your own credentials and license preferences. This file is pretty much needed for any dzil new operations.

Next you need to create a minting profile (not necessary for barebones doesn't do much for you). run mkdir -p ~/.dzil/profiles/catalyst/ (note: catalyst is arbitrary, it can be anything). now create a profile.ini in that directory. The only mandatory line in this file for this to work is the [Catalyst ...] one, I think you'll want the other two however. You also want to set AUTHOR="your name youremail@example.com since that's how Catalyst::Helper inserts it into its files (I'll probably work on fixing helper later).

Now you can just run dzil new -p catalyst MyApp. Hopefully, this simplifies your catalyst app creation a little bit.

Good patches are welcome, so are feature suggestions and bug reports. Also I registered it as Dist::Zilla::Plugin::Catalyst So if there are any other plugins that you think could be useful that are specific to Catalyst I'd be willing to add them.

Special thanks to Tomas Doran who helped me (ok... he wrote most of it) create this module.

Solving code generation problems in dzil

Firstly I want to clarify a bit on my opinions of PluginBundle::USERNAME modules, as some comments there have inspired this post. I don't think you should use them because it makes it harder to disable plugins, and I think Robin Smidsrød put it best:

Mostly it is because the Dist::Zilla::PluginBundle::USERNAME doesn't actually say anything about its intention. It only says use Dist::Zilla as this person does, but what does that actually mean? If you don't know the person it doesn't really tell you anything.

I'd much more prefer PluginBundles that actually advocate certain types of standards or behaviors.

...

Essentially Bundle's like @Git and @Basic don't cause problems because they're generic well defined and contained within their distributions. A bundle that wasn't contained within its dist might not cause a problem if it's for a generic well defined purpose. I'd be in favor of a BundleQATests or similar so long as the author was considerate that some tests (PodSpellingTests) don't work well on some *nix distributions and isn't included in the bundle. But Something like @Xeno (my cpan username) would be my own special settings... who wants to use something that's only for me? and more importantly why would they ever think they have a right to bug something that the name itself implies it's only for me.

But another problem was brought up... code generation. Here's what Nilson said:

dzil is very nice, but I don't if I like the idea of different line numbers and files in the repository vs. the CPAN release. I'm still trying to fully digest this idea.

I remember one of the main mentioned drawbacks of source filters were the possibilities of error messages in the wrong lines. And now, everyone seems to embrace this without hesitation.

So to start dzil's generation isn't quite as bad as a source filter which will give you the wrong line number period, using this generation will still give you the right line number for the module being run at the time. It just means the cpan line number may not match the repo line number. I'm going to show you how to fix this, but you don't always have to. The most common place I've found problems with in code generation has been tests created by extending InlineFiles. In these cases you just want to go and bug the author of the Plugin. I think using these plugins is much better than writing your own EOL Tests, NoTabs, Critic, and Kwalitee etc. It helps keep your dist quality up without extra work, ultimately leaving you with just the responsibility of writing good tests that are specific to just your distribution.

Now on to making sure that your repository matches your cpan dist. The first thing you want to do is use [Git::CommitBuild] (if you're not using git look for something like it or write something like it) This will take all the generated output and commit it to another branch each time you build. Then make sure this branch is pushed to your public repository. I think the biggest problem this solves is having things like your README and your LICENSE actually be in your public repository (if your repo doesn't have a LICENSE... what's the legal situation?). If you're using github you can also change this to be the default branch to be displayed in the repo admin settings.

Next thing is don't use ::Plugin::Prepender This will evil-y insert lines at the beginning of your code which will definitely throw off the line numbers being output. It even seems to suggest using it to insert use strict; use warnings;. If you need output prepended to all files, I suggest writing a plugin that takes advantage of dzil new and maybe just write a script that you can fire off whenever you need to create a new file. This is the only module I know for sure that does this, but avoid ANY that insert actual code or prepend lines to your files (in a way that isn't added to your actual 'master'/'trunk' that should be patched).

POD can also screw up your line numbers, if you're using PodWeaver (or module that has similar side effects) with dzil, which you likely are. Perl Best Practices page 140 will save you here (actual quote pg 475 a summary chapter).

• Keep all user documentation in a single place within your source file. [Contiguity]
• Place POD as close as possible to the end of the file. [Position]

This includes the # ABSTRACT: my abstract here line. If you put the # ABSTRACT and any actual pod after the code then all the pod generated will be after the code in your build, and thus any line number errors will be correct. Of course this doesn't save you if your error is in the output pod, but I suspect that's not the original complaint anyhow, and there are lots of dzil plugins to help you keep your pod sane.

Essentially don't do anything that will change line numbers for the code in the resulting build output. Following this will ease contribution, and debugging; I do not believe it significantly increases maintainer load. Happy Hacking!

UPDATE:
just remembered... dzil inserts a BEGIN block... sigh... can't win for nothing.

UPDATE:
I recommend using Dist::Zilla::Plugin::OurPkgVersion to avoid dzil's BEGIN block / VERSION insertion.

please don't use Dist::Zilla::PluginBundle::USERNAME

or create them. Here's the problem.... (short version is Don't put PodSpellingTests in them)
normally you'd have

[pluginA]
[pluginB]
[pluginC]
[pluginXTests]
[pluginYTests]
[pluginZTests]

and one of them doesn't work on your system (for whatever reason), well you can just do this.

[pluginA]
[pluginB]
[pluginC]
;[pluginXTests]
[pluginYTests]
[pluginZTests]

the ; is a comment in ini, now dzil won't use that plugin. But people will say well you don't want to do that of course I want that plugin enabled. Here's why you may not temporarily. Casual user X has a bug in /your/ module that's using dzil, they code up a patch, and they want to run your test suite. The can't, because [pluginXTests] won't even install properly on their system due to a non perl dependency. They could have just commented it out, but now because you've used this PluginBundle it become difficult. They can't just comment it out.

They might be able to
[@Filter]
-bundle = @USERNAME
-remove = pluginXTests

IF they can get your bundle installed in the first place. This requires them find a way to install your Bundle without installing said broken module.

as I stated at the top my problem is with [PodSpellingTests] it itself isn't broken, but it's dependency Test::Spelling is on some systems, due to the lack of a 'spell' command.

Yes I know there's a workaround... I could just copy a shell script into my path that makes aspell or something work, but really that's not the right solution. No I'm not packaging GNU Spell for my system either, the other spell programs work much better (from what I've read).

Also I find modules with your own username in them to be fairly obnoxious.

If you'd like less code, can't we start making a few more generic Bundle's? I wouldn't minde seeing a Bundle for Tests, and maybe another one for other stuff. I think 1/3 of the lines in my dist.ini's are tests.

Dist::Zilla vs xenoterracide

I solved my problem with Dist::Zilla. If you agree with me that you should be able to install from your git repository, without requiring your users to have Dist::Zilla installed here's one way of doing it.

First install Dist::Zilla::Plugin::CopyTo. Now Edit your dist.ini. Most people probably use one of the Dist::Zilla bundles. Assuming you use Basic here's what you do. Now as you can see we use GatherDir to get where the source should come from and CopyTo to tell additional places where the output of the script should be sent to. It will still send to the default directory, that's ok, you can just ignore those files. add . to your repo (making sure to add the default generated location to your ignore list) and commit. now if dzil changes any of these you can see it.

If you wanted to with git you could use a git-new-workdir and another branch and copy to it and commit this stuff to a different branch. I haven't found another way to do it in a seperate branch yet, although someone said it's possible.

P.S. 1
If dzil ever gives you some vague error about some util file... run dzil clean I don't know why it was doing that to me occasionally through all this but I spent like 8 hours thinking my config was screwed up when the directory just needed to be cleaned.

P.S. 2 Thanks to rjbs for putting up with my asininity while I asked a lot of questions and ranted.

P.S 3
My stuff is 99% working now... I just have a problem where my licensing is wrong :( I've set it to Perl_5 but it really should be GPL3 and Artistic2 (or whatever those are in Software::License).

My new Love/Hate Relationship with Dist::Zilla

Dist::Zilla is a great release tool, code generator, and it just plain takes the boring part of doing a release away from you.That having been said, it does, imo, horrible things to your source repository. "But, Caleb, it cleans up your source repository you can remove a lot of excess stuff you don't really need to be storing there". Well... I disagree, to the point that I've used every explicative in the book to express my displeasure. Let me explain why I disagree.

First let me explain my use cases. I have only one package on CPAN, Template::ShowStartStop and it's relatively simple to the point that Dist::Zilla is almost overkill (but I still find it nice). I also maintain a sizeable portion of CPAN on Arch Linux's AUR. In addition to standard CPAN Packaging for my own package I created a -git package which allows me to install directly from my source repository on Github. That worked wonderfully well until I started using Dist::Zilla (note: I haven't gotten a poorly working (read hack that's buggy) version now).

So Dist::Zilla is great for CPAN, no problem for normal downstream packages, but makes things more painful for Version Control Users. "But Caleb, it does nice things for me for version control". Yes it does, but I think it also has some consequences that people don't realize. If you know how Dist::Zilla works pretend that you don't.

Look at my repo What License is my code licensed under? usually this is in the COPYING file. There's no COPYING file there. How do I install it? No README either, oh and the module isn't complete as is things like version are stripped so Dist::Zilla can generate them for me. This list goes on and on all basically leading back to my issue. Why does Dist::Zilla have to take these things out of my source repository?

The answer was... you wouldn't store .o files in your repository would you? why would you store these generated files... oh I dunno... because someone shouldn't have to install Dist::Zilla just to use my latest git? and they aren't like .o files because those are binarily specific to a lot of things and won't be useful on most other machines, unlike these files.

Let's say Dist::Zilla has a bug in what they generate for code. They fix it, I update, all is good for me. Someone else who hasn't updated Dist::Zilla downloads the code and builds and hits the bug... they complain to me. I explain it to them, they update Dist::Zilla, they're happy, I'm happy. But we could have avoided this, if I stored all the output in my repo they'd never have to worry about Dist::Zilla.

So I've been told this will never be fixed because they don't want to clobber the existing files, plus the whole... these are like .o files, thing. Ok, clobber == bad. So how could we fix it so I'm happy? and everyones happy. We should a directory src shall be the source of all non generated code. Much like when working with compiled code you have a src directory for source files and a directory for output. the lib that's still here will be the final generated lib. All the top level files will also be generated. Only the stuff in src is what should be worked on. And the generated README will give all these wonderful instructions that will now be presented to users visiting the github repo. And when I'm writing a -git package I won't have to include any dzil instructions because I don't need to generate this code all the time.

and you know what? if people don't want to store generated files in there vcs they can always use something like .gitignore. So everybody should be happy.

For backcompat just make it a configuration in dist.ini. We can have a [src] dir which could be . for those who want it the way it works now. and an [output] dir.

Will Dist::Zilla be worth this pain? I'm not sure.