Stupidly Smart
17 Nov 2023Introduction
Generative Artificial Intelligence
A hot topic in the last year, generative artificial intelligence, aka generative AI, has had a huge impact on many different aspects of society and the world. While many people fear its power, most of us in the computer science world know very well that it is a long way from its dystopian depiction in media. Although we may not have robots taking over the world any time soon (Age of Ultron is just going to have to wait…), generative AI still has quite a bit to offer. With capabilities ranging from workout planning to differential equation calculations, generative AI is a tool, but that’s about all it is. As with all tools, you must know how to use it correctly for it to be helpful. Unlike many tools however, you must be very careful not to be misled.
ChatGPT
Although there are many different generative AI products available today, my personal experience is almost exclusively with ChatGPT, and therefore the content within this essay is in the context of my work with ChatGPT.
Springboard
As a computer science major, I have personally found generative AI very helpful in a variety of different subjects. Although it almost never gets the answer exactly right, it can almost always provide a helpful starting point. If you put 100% confidence in generative A.I., you will almost certainly be led astray. However, in order to effectively use generative AI, there are a few important principals:
Always Assume it’s Incorrect
Generative AI has advanced enough to where you can use it as a knowledge base. However, in the context of computer science academia, you must always assume the details it gives you are incorrect. In my experience, I have found that about half of the time, if you ask ChatGPT “are you sure?”, it will change its answer! Sometimes the first answer is correct, sometimes the second is, and occasionally, neither is correct! Generative AI may certainly serve as an inspiration, but when asking questions with any nuance, you should always follow up your AI expedition with good old-fashioned research. You are still the better half of Generative AI, isn’t that good news?
K.I.S.S.
The K.I.S.S. (Keep it simple, stupid!) rule is a good principle to adopt in many different contexts. In the context of generative AI, it means you can pretty much bet that if your knowledge is getting down to the ‘nitty-gritty’ or very ‘in the weeds’, you’re not going to get helpful answers. For example, if you ask it to write some basic java code, it’ll give you a pretty good answer, but if you ask it to design data structures and algorithms for a specific prompt, I’d be willing to bet you you’re going to have to do a pretty good amount of rewriting!
Springboard (Only!)
Using Generative A.I. can be very helpful, however there are many limitations. If you stick to well known facts, you have pretty good odds of getting dependable answers. However, if you’re trying to get really into the ‘generative’ part, asking for synthesis of complex ideas or advanced analyses, you’re better off using it exclusively for inspiration. Generative AI is a very good springboard. It can help you start a discussion post, come up with the main ideas for a paper, or even give you a design to help get your brain wheels turning, but (as with all things), you should not take its word as the truth, and you should not stop your digging there.
ICS 314 Course Involvement
ICS 314
ICS 314 is a software development course. Over the semester, we have covered a wide range of topics to include JavaScript, HTML, CSS, Meteor, MongoDB, and more! In coordination with the weekly material, there is a particular pedagogy to the course. Each week, you have a “WOD”, or workout of the day, where you implement whatever you have been working on in a small, timed project.
Experience WODs
In experience WODs, generative AI was moderately helpful. As with many AI queries, you must know what question to ask. For example, when working with the Underscore JavaScript library, you can give ChatGPT a prompt of what you want to do, and it can reliably give you a solution function. For the experience WODs however, instructor videos were also provided. I found it much more helpful to watch the video and then solve it myself a separate time or in an alternate context, as this allows you to understand how experienced professionals go about solving it. The “I need X, so I will look for Y” is very helpful in learning a new process and how to filter a large range of information into exactly what you want.
In Class Practice WODs and In Class WODs
For In Class Practice WODs (and almost identically, In Class WODs), the helpfulness of generative AI depended greatly on the context of the WOD. For example, with basic JavaScript, HTML, and CSS, you can almost certainly get the right answer spit out at you if you try enough times. You can copy and paste your code in, if you have an error, and ChatGPT will tell you some things to try to fix it. While I do not have a definitive percentage, with topics such as the aforementioned ones, you have an about 95% chance of being able to have generative A.I. completely create, or at least solve errors, within a specific task. For example, on the beginning WODs where we were learning JavaScript, although I tried to synthesize code myself to learn the language, I would always copy and paste my code into ChatGPT to see the syntax mistakes I was making, and it almost always gave me a good fix!
However, when it came to more complex and interrelated concepts, such as using MongoDB with Meteor and using React with Meteor, ChatGPT is almost completely unhelpful. While it can give you some ‘skeletons’ to build onto, they’re almost always poor solutions. For example, just today I tried to have ChatGPT create a button which interacts with a custom Collection. The result ChatGPT gave was not only incorrect but convoluted and poorly organized.
Essays
For Essays or any writing synthesis, I have found ChatGPT to be a very poor substitute for actual writing. Even on my worst days, I’m able to explain a concept more eloquently and in detail than ChatGPT. In the context of ICS 314, I haven’t used generative AI for any essays. I have used ChatGPT to create discussion posts for other classes, however, have found that without a single exception, I have had to greatly modify the provided output to create an actual post. I would say, generative AI has an essay IQ slightly below an exceedingly average individual.
Final Project
Generative AI, namely ChatGPT, is not very good with Meteor + React + MongoDB. I have tried asking it to create code to work with a database, and it does not do well at all. Any time you want ChatGPT to work with multiple files, even if you copy and paste the involved files directly in the prompt, it gets very confused and is largely unhelpful. Despite this, you can get some helpful information if you break it down…like really far down. But the information is going to be outdated and you’re still going to spend a considerable amount of time combing through documentation.
Learning a Concept/Tutorial
If it’s simple, you’re solid. ChatGPT can handle explanation of simple concepts well. It can identify the relationships between two components or explain how two things interact. I have asked ChatGPT to give an answer for a certain topic, and then asked “why?”, and it almost always gives an understandable explanation. A good rule of thumb for learning concepts is that if there are (shorter) youtube videos explaining it, ChatGPT can probably do better. If you have to dig really deep into youtube, or Abdul Bari has a video for you, Abdul can do it better. For example, I had asked ChatGPT what meteor does, and while I certainly did not get an all inclusive answer, the response was conceptually helpful.
Questions
Simple questions are great for ChatGPT. However, I use ChatGPT as my first go-to for any question, so if it’s puzzling me enough to need to ask it in smart-questions or in class, ChatGPT has already failed quite a few times. ChatGPT does not do well with open ended or complex questions. You have to break it down! If you can break your question down into many smaller questions, ChatGPT can usually help. I personally rarely struggle with this, however many students often do not know what they need to know, so here ChatGPT would be largely unhelpful.
Coding Examples
For simple code, ChatGPT can definitely do it for you! Especially if you ask it for specific examples or give it a prompt to solve and say, “use X”, ChatGPT can absolutely solve your problem. In contexts such as entry level javascript or libraries such as underscore, you don’t even have to think twice, generative A.I. knows what it’s doing here!
Explaining and Writing Code
If it can write it, it can explain it. If you have a solution already and want it to be explained, you’ve got a 90% chance of getting a high-quality explanation, as long as you’re not working with many interrelated files. I have rarely encountered an explanation from ChatGPT that was not enough to gain a working knowledge of the code and be able to synthesize a similar solution from scratch. You do need to be careful though, as sometimes ChatGPT will break conventions, create redundant code, or not include somewhat necessary elements (proper error checking, for example). With JavaScript, Java, C, and C++, I have had very positive experiences asking ChatGPT to code things for me, however exclusively when there are few files involved.
Documenting Code
Generative AI does not do a great job with documentation. I have found that it generally explains things that are not necessary to explain or excludes important concepts. Although it does comment almost all code it generates, you will almost certainly need to be able to understand the code it creates and then go create your own commentary. Also, it does not follow conventions, and you will need to implement these yourself. I have not used ChatGPT to create code documentation, but rather will ask it what the code is doing and then explain it myself once I understand.
Quality Assurance
A thousand times yes! I cannot count the number of times I have pasted problematic code into ChatGPT and gotten a fix implemented within 10 minutes! This is by far one of my favorite uses for ChatGPT, as once I have checked over my code a few times and am not able to find a problem, I get “concentration blind”, where I sometimes miss silly little (important) things like commas, misspellings, or brackets.
Impact on Learning and Understanding
ChatGPT has had an overwhelmingly positive influence on my learning and understanding in a variety of contexts. If you can break it down, it can help you. It is certainly not a substitute for human instruction, nor is it a substitute for good old fashioned “grinding it out”, however it will almost always make the process quicker. In fact, it is so helpful that my first resource for a problem is almost always ChatGPT. It is not advanced enough to solve all your problems or create solutions the way humans can, but it can give you a strong nudge in the right direction. Showing you where to look and what you might want to look for is a significant amount of the battle, and ChatGPT can consistently shine a flashlight to the dark path (or if not, it can light a candle).
Practical Applications
Low-Level Coding
Most of the developers I know who work out in the ‘real world’ use ChatGPT or some other generative AI to do the work a low-level programmer would. It can create simple solutions very effectively and efficiently, and you don’t have to teach it. To my knowledge, the code generative AI creates is always audited and often changed a little, but if you break it down correctly for the AI, it usually provides a pretty good solution. Plus, if you have an entry level coder, odds are you’re auditing their code and breaking it down for them as well.
Data Analysis
AI models can be used to perform fast and in-depth data analysis. Not only can AI read and analyze data, but it can clean data, recognize patterns, predict future data based on past data, detect anomalies, make predictions, and generate automated reports. Vitally, AI models can be customized to fit your exact needs. A funny but true saying that applies here very well: “Why spend 10 minutes doing a task, when you can spend 10 hours automating it?”. Certainly, the saying is meant to be ironic, however although it is a greater (temporary) time commitment, all those sets of 10 minutes doing mind-numbing tasks will definitively add up to more than 10 hours in the long run - and think of how much happier you will be to not be doing such mundane tasks!
Challenges and Opportunities
While AI has been a great help to me, it certainly does not help as much as I’d like it to. Especially in the context of working with many files at once, ChatGPT is absolutely at a loss. In the context of my final project, ChatGPT has only offered poorly written, incorrectly organized, and outdated code. Teaching it to use modern software development tools like Meteor, MongoDB, and React, would absolutely be worthwhile.
Virtual Machine
As mentioned previously, AI does not do great with a large number of files or interconnected components across files. A possible solution is to implement AI that can utilize a virtual machine. This way, it can read many files, write command prompts, and run sites to see what is happening and provide more involved and accurate solutions.
Comparative Analysis
Traditional teaching does not offer customized information to a student. More often than not, they throw the book at you and walk away. With AI, students can query exactly what they want to know and usually get a very helpful result. Additionally, this helps students stay engaged, as they are actively interacting with an interface to gain insight into a topic. Not only this, but students can ask for applications beyond what is in a book or lecture, potentially connecting the material they are learning with something that truly speaks to their interests. The ability to make mundane textbook knowledge come to life is precious. Applying knowledge is the best way to learn it and actually keep it in your brain. These are valuable tenets not available before generative AI.
The main failure of generative AI is the depth and accuracy of the knowledge. Unfortunately, I have found far too many instances in which AI offers an incorrect answer, stating it as fact. In addition, it cannot teach you to think and problem solve. The most valuable part of my education, in my opinion, is that I am learning how to problem solve, regardless of the content the problem covers. Without being able to pick instructors’ brains, I wouldn’t be able to observe helpful methods for information synthesis and content creation.
Future Considerations
Generative AI plays a huge role in the future of software engineering education. It has the potential to completely knockout the mundane tasks associated with learning big concepts. More often than not, the mundane tasks and technicalities create a HUGE mess, especially in the context of bigger coding projects. Generative AI can allow students to focus on the bigger picture, allowing only more and more advancements in education as well as the profession itself.
Conclusion
Overall, AI is a helpful tool. It is not a end-all be-all solution. I would compare the introduction of AI to the introduction of Google. Certainly, there is misinformation just as there is helpful information. You cannot expect it to solve all your issues. But it is a great springboard into whatever else you may want to do! The involvement of AI in (all) courses is extremely important, as it is going to be a tool out in the real world as well. Being able to query AI effectively and use it to one’s advantage is a huge part of the learning process now. Just as using Google is an integral skill to a college student (as well as the industry professional), so is using AI. Despite its many limitations, AI will continue to develop and become a large part of our society. Giving students exposure to and skills to manipulate AI is a vital component of their education and will continue to be a vital part of their lives after graduation.