31 August 2016, by Simon Wilson
- Local version control: local commits and local branches. Proper branches (not TFS folder-branches) which are lightning-fast to switch between, and you can cherry-pick between.
- Rebase temporary dev branches back onto master instead of clumsily re-applying old shelvesets.
- If more than one person is using it, push and pull git branches between yourselves, and keep developing even when the central TFS server (inevitably) goes down.
- Separate source control from Visual Studio: generally a good idea in my opinion, and particularly useful if VS is already running slowly.
- Reliable version control: TFS get-latest is a bit buggy, but git-TFS seemd to work reliably. Perhaps it abstracts away for you the tedium of always telling TFS explicitly to get the latest version of all files, not just some.
- You can only checkout one directory at a time so you need separate checkouts for different root-level projects in the same repo. This is pretty minor though.
- Visual Studio gets a bit confused and integration isn’t as good. Keep it happy by using msys git (git for windows) not cygwin git if you can.
- Changing between TFS and git-tfs, or just using visual studio with a git-tfs checkout, leads to corrupt TFS workspaces.
- Doing things differently from the rest of the team means you lose out on their TFS knowledge and support, for what it’s worth.
- You won’t be able to add projects to the solution properly.
- Visual Studio will constantly try to remove the tfs element from the .sln file, which will break TFS integration for everyone else if you commit it.
- If you need to keep local changes, e.g. for local environment config, then it’s a pain to stash the changes all the time (you need a clean checkout to rebase).
- It’s just really slow – mainly because my team’s TFS server was slow; but git-tfs was even slower.
For me, unfortunately the advantages don’t outweigh the disadvantages, and I probably wouldn’t use it again. it’s a bit of a pain to set it up and to use different tools to the rest of the team. On my next TFS project (and I’ll do everything in my power to make sure there won’t be one) I’ll just embrace the shelveset workflow instead.
What I might try is just to
git init a local repo in my TFS checkout and get local branching that way, adding the .git folder etc to the local ignore list, although it might lead to some manual effort keeping branches in sync with the remote.
- Install git, and configure it:
Where XXX is a recent changeset – you’ll only have history available from this changeset, but the further back you go the longer the clone will take.
- Open in Visual Studio!
Don’t bother to un-map your workspace: the Visual Studio TFS integration will go away when you open your new “git” project. And it will be easier to go back to pure TFS if you want to.
Be careful though, this makes a number of commits in a row, and because TFS is so sloooow the can be nearly a minute apart and trigger more than one consecutive CI build. If the build time is slow this might hold other people up from committing, so keep an eye on it: you can always cancel the first build(s) if this happens. You might just want to squash up the commits to save time committing!
If your project requires certain custom notes fields to be supplied, use the
--notes) argument to supply it from the command line:
24 August 2016, by Tim Perry
Git Confirm is a git hook, which asks you to confirm when you commit a change that includes additions from a (configurable) list of risky matches. Think ‘TODO’, ‘FIXME’, ‘@Ignore’, ‘describe.skip/it.skip’ and ‘describe.only/it.only’. You can drop Git Confirm in, and effortlessly stop yourself ever committing anything like this by accident. TODO is the easiest example. It’s really useful to sprinkle TODO comments in your code as you work, to mark things that will need fixing in future, and jot down notes as you work. It’s really terrible to end up with a codebase riddled with them though. You need to actively deal with them in the short term, either fixing them immediately as you wrap up your larger piece of work, or moving them into a proper task tracking system somewhere. Unfortunately, it’s easy to fail to do that; to add todos as you go, skim your code later and assume you’ve to-done them, and end up slowly rotting your codebase over time. With Git Confirm, you put ‘TODO’ on the watchlist, and any time you commit code that adds TODO, you’ll see it (in context) so you can confirm you definitely want to commit it. Demo time! Simple, effective and ready to go. There’s other options for this, but personally they’re either too simplistic (rejecting commits to any matching files outright) or more complex and overpowered than I want by default (automatically opening TODO issues on Github, or fully synchronized tracking of TODOs with external issue trackers). All of these have their place, but there’s a big gap in the middle for a simple tool that stays out of your way and catches you if you slip up, without being too opinionated. There’s also a lot of edge cases that these often don’t cover: changing other parts of files that do already contain TODOs, removing lines with TODOs in, or including context with failures so you can see find the issue. Deep interactions with Git like this create surprisingly fiddly edge cases, but Git Confirm has your back all the way. In addition, as far as I can tell there’s no tools for the general case at all, where I want to match arbitrary patterns. I’d really like to never accidentally commit an ignored test, and never ever ever ever accidentally ignore all my tests but one. With Git-Confirm, that never happens. Want to try it? In the root of your repo, run: Done! By default it’ll just match ‘TODO’, but you can override that and add any regular expressions you like with: Happy committing!
Get easy confidence on exactly what you’re committing.
curl https://cdn.rawgit.com/pimterry/git-confirm/v0.2.1/hook.sh > .git/hooks/pre-commit
chmod +x .git/hooks/pre-commit
git config --add hooks.confirm.match "pattern-to-match"
Git Confirm is a git hook, which asks you to confirm when you commit a change that includes additions from a (configurable) list of risky matches. Think ‘TODO’, ‘FIXME’, ‘@Ignore’, ‘describe.skip/it.skip’ and ‘describe.only/it.only’. You can drop Git Confirm in, and effortlessly stop yourself ever committing anything like this by accident.
TODO is the easiest example. It’s really useful to sprinkle TODO comments in your code as you work, to mark things that will need fixing in future, and jot down notes as you work. It’s really terrible to end up with a codebase riddled with them though. You need to actively deal with them in the short term, either fixing them immediately as you wrap up your larger piece of work, or moving them into a proper task tracking system somewhere.
Unfortunately, it’s easy to fail to do that; to add todos as you go, skim your code later and assume you’ve to-done them, and end up slowly rotting your codebase over time.
With Git Confirm, you put ‘TODO’ on the watchlist, and any time you commit code that adds TODO, you’ll see it (in context) so you can confirm you definitely want to commit it. Demo time!
Simple, effective and ready to go. There’s other options for this, but personally they’re either too simplistic (rejecting commits to any matching files outright) or more complex and overpowered than I want by default (automatically opening TODO issues on Github, or fully synchronized tracking of TODOs with external issue trackers). All of these have their place, but there’s a big gap in the middle for a simple tool that stays out of your way and catches you if you slip up, without being too opinionated.
There’s also a lot of edge cases that these often don’t cover: changing other parts of files that do already contain TODOs, removing lines with TODOs in, or including context with failures so you can see find the issue. Deep interactions with Git like this create surprisingly fiddly edge cases, but Git Confirm has your back all the way.
In addition, as far as I can tell there’s no tools for the general case at all, where I want to match arbitrary patterns. I’d really like to never accidentally commit an ignored test, and never ever ever ever accidentally ignore all my tests but one. With Git-Confirm, that never happens.
Want to try it? In the root of your repo, run:
Done! By default it’ll just match ‘TODO’, but you can override that and add any regular expressions you like with:
10 February 2016, by Chris Arnott
It is very common to write ToDos in code. This makes sense. You’re working on a piece of functionality and suddenly remember something relevant, but slightly off topic, which needs to be done. So you add a ToDo. Brilliant, you’ve successfully not shaved any yaks.
But when will you actually get around to finishing off the ToDo? How urgent is it? Will you remember that the ToDo is there when you do get around to implementing that extra functionality?
23 December 2015, by Tim Perry
This post is the first post in a series looking at Build Smells – signs of major issues in team build processes – and the things we can do to try and solve them. Watch this space, or follow us on twitter to see more of these as we publish them.
If you’re not careful, it’s easy to end up in a situation on a project where you end up consistently wasting time fighting merge conflicts, rather than getting useful things done. This isn’t unavoidable, and it’s a build smell: a sign that all is not quite right here.
We see cases where this is not just occasional, but actually becomes completely standard, and there’s simply an accepted expectation that teams working together will burn time every couple of weeks, as somebody spends a day merging everybody’s changes together (or a week a month, or worse). It’s not uncommon to see cases where this is even encouraged: where everybody in the team automatically starts a branch every sprint, and everybody merges them all together all at once at the end (nearly two weeks later). This is a recipe for pain.
This wastes time and saps morale. It’s even worse than that in reality though, because if you’re continually expecting to hit merge conflicts then you can’t really refactor; any wide-ranging changes to the codebase either require everybody to down tools, or to risk huge time-wasting conflicts. You end up in a situation where your build processes and tools are pressuring you not to refactor, where your own build is encouraging you to write worse code. Renaming a method becomes a painful and potentially costly process, so you don’t.
This is a build smell. Merges should be frequent, small, and typically easy. If you see people burning large amounts of time merging, actively avoiding merging their code, and racing each other to push changes to avoid having to deal with the repercussions, you’re seeing this in action.
How do these happen?
We can examine this more closely at this by looking at some of the most common forms of merge conflict:
Conflicts where two changes have been made that are totally independent, but which don’t have a clear resolution when merging textually.
For example: two people adding two new fields to a class in Java, on the same line. The fact they chose the same line typically makes no difference, and the fields typically independent, but all major merge tools will ask you to resolve this manually, as they have no way to decide which line should appear above the other, or whether that can be done safely.
Interestingly, this conflict transparently disappears if we make these changes in sequence. If one person adds a field, and pushes the code, and another pulls the code, and adds a field, then there are no merge conflicts, and the situation is invisibility resolved with zero extra effort from either party.
Simple semantic conflicts
Conflicts where two changes have some interaction that produces incorrect behaviour, but aren’t necessarily in direct opposition. Changing a
getDistance()method to return a distance in meters instead of centimeters for example, while somebody else adds some code that calls that method.
These changes may well not textually conflict, but do have a relationship: code that’s expecting centimeters will (probably) not act correctly when given a different unit.
(As an aside: issues like this are a good reason to include units of measure in your types, to use things like F#’s built-in unit of measure support where possible in languages that support that, or to at least use clearer method names).
Again though these conflicts are trivial to resolve if the changes are made in sequence. If you are already aware that the method returns meters, because that change has already been shared, then you’ll ‘resolve’ the conflict without even noticing, by writing your code with consideration of that built-in from the start.
Fundamental semantic conflicts
These are the conflicts where you and somebody else are trying to make changes that are fundamentally in opposition.
Perhaps I need a button to be green, and you need it to be red, or I need to make a major refactoring of a class that makes some functionality impossible, and you’re writing some new code that depends fundamentally on that functionality existing.
These are conflicts without easy resolutions. To solve this, we’re going to have to have a discussion, and come to some compromise in our goals and overall approach.
These issues are also far rarer than people imagine though. Try to think of a conflict recently that would’ve been an issue if you’d made your changes entirely in sequence, rather than in parallel. It’s rare that two developers on the team are chasing mutually exclusive goals at the same time.
Notably when this does happen it’s still much easier if we can resolve these immediately. If I find out that we can’t make buttons red when I make the first button red, I’ve got a problem to solve. If I find out that we can’t make it red after I’ve spent three days redesigning the whole application in shades of burgundy then I’ve wasted a huge amount of time, I have to undo all that, and I still have the same problem to solve too.
When we hit painful merges, it’s rarely because we’ve actually hit fundamental conflicts. Most of the time, we’ve just built up a large set of textual and relatively simple semantic conflicts all together, and they’ve snowballed into a huge conglomerated mess that we now have to unravel.
The underlying conflicts here come from isolation. Code that’s kept on a branch or kept locally unpushed is code that’s isolated from everybody else on the team. They’re secret changes you’ve made, that you’re not sharing with everybody else. Because you haven’t shared them, nobody can take them into account in their own changes, and conflicts quickly proliferate.
This gets dramatically worse as isolation increases. If you keep code on a branch for a day, you have one day of your changes to potentially combine with one day of everybody else’s on the team. If you all stay on your branches for a week, you’ve got a whole week’s worth of your own changes to combine, with every other person’s week-sized chunk of work too. This quickly gets out of hand.
What can we do about it?
The solution here comes neatly out of the above: large conflicts are typically a collection of many relatively simple conflicts, and each of those would almost disappear if the changes were made in sequence. Therefore, if we want to get rid of merge conflicts, we need to move as much possible away from isolating our codebases, and get as close to being a team of people making small changes in sequence as we can.
To fix this smell, share your code early. As early as possible; I’d suggest you integrate your code with the rest of your team every time you have code that doesn’t break anything. As soon as you have code that compiles, passes your tests, and functionally works (to a level where you’d be happy showing it to a customer): share it with your team. Sharing this won’t break anything (by definition), and completely eliminates any risk of conflicts within that code.
That’s all well and good, but much easier said than done. Let’s look at some examples:
If you’ve written a couple of new passing tests in a clean codebase (for example, just before you’re about to refactor something), you can commit them and share them immediately. They’re passing tests, so you can be pretty confident you’re not going to break anything, and once pushed you remove any risk that they’ll conflict with others in the team.
If you don’t do this for a while, you build up isolated changes. When somebody attempts to refactor test code in the meantime, touch any of the dependencies you’re using, or actually touch the code you’re testing, they have no way to easily take your changes into consideration.
Meanwhile if you have pushed your tests then it’s easy for others to check that their changes don’t break it, or to apply their wider refactorings to your test too (often automatically, with automatically applied refactorings).
Doing TDD, and you’ve written a failing test instead? There’s reasonably mileage to be gained in many cases in ignoring the test (e.g. with
@ignorein JUnit), and sharing that (although this does require discipline to ensure you don’t leave tests ignored long term).
You don’t gain any guarantees in this case that the behaviour under test won’t change, but you do ensure your code will continue to compile, and others making wider changes that affect your code will fix it in the process of those changes like any other code in the codebase. By sharing your code early here, you allow people making other changes to do so with respect to yours immediately, rather than leaving you to tidy up the mess later.
Sharing model and component changes
If you’ve just added a field, or a new method to the models or internal components of your system, you’re almost certainly very quickly back in a functionally working state where your changes could be shared. A remarkable amount of the code you add as you progress towards building new functionality doesn’t break your application for your users or other developers, and can be easily shared immediately.
Added a new
calculateCost()method to your
Invoiceclass as the first step in the feature you’re building? Share that now, and the other developer who needs to refactor
Invoicetomorrow for their changes won’t step on your toes, and you’ll both live happier lives.
This is especially relevant in wider-ranging cross-cutting changes. If you’ve refactored big chunks of code, changing how logging works, or renamed some fields, it’s should be very unlikely you’ve made any user-facing changes to your application at all, and your tests should be catching any such issues if you have. In addition, the stakes are higher; refactorings particularly quickly collect conflicts, while the majority of them won’t have any fundamental conflicts at all (rename field/variable, extract method/class/interface, etc), and fundamental refactoring conflicts that do appear will explode far more painfully if left to ferment for a few weeks.
Sharing changes like early drastically reduces the potential chance and damage of conflicts, and is well worth pushing for where possible.
Sharing trickier changes
These are two examples where it’s particularly easy, but they don’t cover every case. Often you want to make a change that is fairly large in itself, and individual changes within that will result in broken or incomplete functionality. Rewriting the interface for a page of your application is a good example, performing a huge refactoring or rewriting a major hefty algorithm within your application; in many cases this might be a large and challenging change, and nobody wants to share a broken or incomplete page or algorithm with the rest of their team (let alone their users).
There are things we can do in this case however. Using feature flags/toggles and branching by abstraction are two particular approaches. Both revolve around sharing code changes immediately, but isolating the user-facing functionality within the codebase (rather than isolating the functionality and the code changes, using version control).
Branch by abstraction works by introducing an abstraction layer above the code you’re going to change, which delegates to either the old or new implementations. Initially this calls the old implementation (changing nothing), but once standalone parts of the new implementation are completed and ready the abstraction can be changed to call through to new code instead, and eventually the old implementation (and potentially the abstraction) can be removed.
Feature flags work somewhat similarly, but dynamically with configuration, either application-wide, or in per-user settings. A conditional statement is added at a relevant branching point for your change, switching between the two implementations based on the current configuration (
return flags.newUserPageEnabled ? renderNewUserPage() : renderOldUserPage()).
This has a few advantages, notably making local development of visible features like new pages easier (just enable the config property only on your machine), and allowing you to take this even further to become a feature in itself (as done by Github, Flickr and Travis CI) providing you fine-grained control over your feature release process, for progressive rollouts or A/B testing.
All of these techniques help you share code earlier, and sharing code early enough can almost totally remove conflicts from your team’s process, and make them easier to deal with when they do happen.
I should be clear that I’m not saying branching is always a bad idea. Branching has many sensible practical and effective uses, and it’s an incredibly powerful tool. Unfortunately many teams fall back to it as their goto tool for every change they make, and they start to build the way they work around actively keeping things on branches.
Branching comes with a cost, that rapidly increases over the lifetime of the branch. That’s fine if you’ve got a specific benefit you want to get from branches, but not a good idea if you don’t. Some good reasons to use branches for example: very experimental code that’s unlikely to make it into production, situations where you can’t allow everybody push access to master directly (as in open-source and some regulatory environments), and situations where your review processes don’t work effectively with it (on of the main reasons I see for not following this, as Crucible is one of few good tools that will easily let you review commits on master).
Nonetheless, in each of these cases it’s still worth aiming to merge more frequently. Encourage many small pull requests to add a feature, don’t allow experiments to go on too long with deciding whether they’re worth committing to, and push for more smaller reviews with minimal turnaround time before merging.
In general, smaller and more frequently shared changes disproportionately reduce the risk of each change, make refactoring vastly easier, and can often make conflicts disappear with no extra effort. A large proportion of merge conflicts simply are simply invisible if the changes are made in sequence instead, and sharing code earlier moves you closer and closer to this situation.
Finally, it’s notable that this way of working isn’t just a benefit for conflict resolution. It is a good idea all the time to aim to break your work into many small chunks, with many steps at which the code is fully working. Doing so helps avoid yak shaving, makes each of your changes easier to understand, and has lots of benefits for time management and estimation too. There’s a lot of upsides.
Hopefully that’s an interesting insight into the kind of issue we see in projects, and some of our approaches to improving them!
We do encourage all our teams to find their own solutions independently for their problems however, and I’d be lying if I claimed this is our standard go-to approach in every case. It is an approach that many of us have found effective in places though, and it’s well worth considering as an option if you’re facing these same challenges in your team.
I should be clear also that I’m not suggesting that branches are never useful, that this is the One True Way of managing version control, or that you should dive into this 100% immediately. If your team is facing concrete issues with conflicts and merges, experiment with reducing isolation and sharing code earlier and more frequently, and measure and discuss the changes you see in practice. We’ve found benefits in many cases, but that is by no means definitive, and there will be cases where this is not the right choice.
Let us know your thoughts all this below, or on twitter! In the next post, coming soon, we’ll take a look at the next smell in the series: Nervous Pushes.
12 December 2014, by Chris Arnott
Usman has been busy writing a webapp for use within Softwire called “Hook”. Hook provides webhooks on our git repositories. Watch his lunch and learn if you’d like to know more: