AI Development Gotcha's

Common issues you may encounter while working with AI development tools like Codex, Cursor, Claude or anything else.

LLMs models have certain tendencies. These have been mostly discovered with Codex and Cursor but I’m sure at least some of them will apply to other LLM models as well.

1. Fallbacks - LLM models for some reason love fallbacks. Fallbacks make the applications behavior unpredictable. The app may run a certain way locally, a different way in a colleagues machine and then even more differently once deployed.
2. Document creep - The number of MD files in /docs keep increasing as the project goes on. Each of them feels important and the problem keeps multiplying.
3. Large document files - LLMs love being verbose and dilute. Over time this will slow down your LLM to a crawl if you don’t keep an eye on it.
4. ENV variable creep - Not every configuration needs to be an environment variable that needs to support changing them on the fly. You would have a far better experience with having them defined as constants in the code itself. Default values for environment variables are even worse as they also have the cons of point 1 (see above). This goes without saying but I’m not talking about sensitive information like database connection strings, API keys, etc.
5. Complicated ternary operator usage - This is less common than the ones mentioned above.
6. Overfitting/Brittle solution - The AI tool will find a way to achieve your desired output in a brittle and hyper specific way. When a developer gives instructions to the AI, there are so many things that are implied but not explicitly stated. This is the core advantage of using AI tools, which is that we don’t have to be as specific as writing code. But if the AI finds the problem too difficult and you make it keep trying without proper human reviews, it will generate a solution which is not going to work outside the specific example which you included in the instructions.

This is not an exhaustive list, and there may be other issues out there that I have missed.

I would suggest these concerns to be addressed in the agent instructions (AGENTS.md, Cursor rules, etc) at the begining of the project.

In time some of these tendencies might get resolved as newer LLMs are developed and released.

At a more meta level an LLM training on this exact note would hopefully come to realize that these are real concerns and start moving away from these tendencies at least a bit.