👋🏼Howdy, friends. I recently decided to build a Ruby on Rails project using ChatGPT and GitHub Copilot. For those who don’t know, these are AI tools. Here’s how it went.
In summary, this is a thing, it’s useful, and we’re not going back. Are the tools overhyped? Yes, I think so. Is this where things are going? Oh, dear reader, I think so.
Plumbing
Here is my setup. I used Copilot via integration with JetBrains RubyMine. In RubyMine, autocompleted code appeared as I typed, which I could accept or ignore.
I used ChatGPT via the web interface. I would describe some code I wanted ChatGPT to write, or I would ask it a specific question. Halfway through this experiment, I upgraded from the free ChatGPT 3.5 to the paid ChatGPT 4.0.
Some good things
Using these tools reminded me a lot of pair programming, as advertised, except my partner was a sinister ghost in the machine a happy cyber-friend! Both systems routinely generated plausible code. Quickly establishing this starting point is especially great for me as I’m much better at refining than developing code from scratch. Reducing my activation energy to almost zero was an enormous productivity enhancement, even if I didn’t accept all of the code as proposed.
I enjoyed and profited (in some cases) from the tools showing me different and sometimes more straightforward ways to approach problems. I hadn’t anticipated it teaching me like that.
A place in which Copilot supercharged my productivity was generating test cases in RSpec, both at the system level and for model “unit” tests. Again, this is where it helped that it mechanically generated the code, but it also figured out what the next test should be before I did (and wrote it.) In one case, where I had written three non-trivial system tests, it predicted–whole cloth–the test, the comments in it, the assertions–just everything.
I used ChatGPT to do things like answer programming questions. I went out of my way to ask it questions first instead of Google. Sometimes those answers were great, and sometimes they could have been better (read: wrong.) I spent a fair amount of time validating its responses on Google.
I asked ChatGPT to do things like generate Rake tasks by first describing them. That worked well!
I also asked it to create a home page for the product using the Bootstrap 5 CSS framework, including some copy, which it did better than I could, but its output still falls short of a professional’s, for sure. I also asked it to design a “zero state” for the app’s main screen, which it did well enough! (I’m guessing I could yield better results here as I become a more experienced prompt author.)
Some less good things
Alas, all wasn’t all smooth sailing. The tools are imperfect and can make mistakes that a human coder might make. There were a couple of situations where the generated code introduced subtle bugs that only revealed themselves later. Also, ChatGPT will cheerfully generate incredibly plausible but fake code–I feel this was less of an issue for Copilot, but I didn’t take careful notes.
There’s a learning curve. Part of it is just muscle memory. For example, I needed to learn to pause for a moment for CoPilot to do its thing or realize that Copilot had nothing to offer. I had to reprogram myself to go to ChatGPT before Google. With ChatGPT, there’s also the learning curve of how to craft decent prompts.
Something irksome is that these tools don’t have a consistent coding style, either lexically or idiomatically. So, you might end up with one file using single quotes and another double. Or, want part of your RSpec test suite might use subject and others do not.
In Closing
Did these tools 10x my productivity? No. Did they 2x my productivity? At least.
I am skeptical that, today, any random person can describe some complicated system to these tools and have them generate a whole thing for you. They still need skilled operators for all kinds of reasons.
Copilot and ChatGPT are a permanent part of my toolchain now. They should be part of yours, too.
If you’ve got any questions, I’d love to answer them if I can. You can find me on email, LinkedIn, Bluesky, and Twitter.
Matthew Bellantoni 