An application generated by an AI can work on the first try and still carry five risks that you do not see while using it: a dependency invented by the model, a secret sent in an instruction, a doubt about the rights over the code, an authorisation flaw and an author who leaves. This page is written for decision makers, business owners and beginner developers, with no technical prerequisite. It follows a single textbook case, the "purchasing portal", to link each risk to a precise cause and to the control that stops it.
What you will learn
- Describe a concrete incident from five distinct causes.
- Recognise an invented dependency and explain why it is dangerous.
- Explain what becomes of a secret or data pasted into a prompt.
- Say why the legal status of generated code is verified rather than assumed.
- Define an object-level authorisation flaw and the test that reveals it.
- Match each risk with the control that stops it, without believing that a single tool is enough.
Why does a single case show five risks?
An incident rarely comes from a single fault: it comes from several small oversights that add up. To see this, the training uses a case invented for teaching, the purchasing portal. It is not a real story.
A buyer wants to look up the company's supplier orders faster. She describes her need to a coding assistant, which produces a web application in one afternoon. The application connects to the ERP, the management software that holds the orders, suppliers and invoices. It is part of the information system, the set of IT tools of the company. The colleagues in the purchasing team use it from the following week.
Nobody decided anything wrong. Yet the application accumulates the five defects that this page describes. Each has a simple cause, and each is checked with a precise gesture. Reading the five in order is enough to learn how to look for them in any other application.
What is a dependency invented by the AI?
An invented dependency is a package that the model suggests installing even though it does not exist. A dependency is a piece of code written by a third party and published in a public registry, for example npm for JavaScript or PyPI for Python. The application downloads it instead of rewriting it.
In the purchasing portal, the assistant adds to the dependency list a package that is supposed to read invoice files. The name seems plausible to it, but no package carries it. The trap appears when someone publishes exactly this name with malicious code: installing the application then runs it on the buyer's machine and on the server. This attack is called slopsquatting, formed on typosquatting, which exploits typing mistakes.
The phenomenon is measured: the study by Spracklen and his co-authors, presented at USENIX Security in 2025, found invented package names in all 16 models tested, commercial and open alike ("We Have a Package for You!"). The page Vibe coding gives the figures.
The control consists of checking that a package exists and deserves trust before installing it, then pinning what you install. The guide Securing npm, PyPI and Go dependencies explains the gesture, and the page SBOM shows how to draw up the list of everything the application contains.
What becomes of a secret or data pasted into a prompt?
What you paste into a prompt leaves the company. A prompt is the text sent to the model. A secret is a piece of information that gives access: password, API key, token. To get help with an ERP connection error, the buyer pastes into the assistant the error message, and with it the access key that it displays. She also pastes an extract of the supplier list, so that the AI "understands the format".
Two problems pile up. The key is now at a third party, and the assistant reuses the pasted key in the code it generates: it ends up hardcoded in the application. If the code repository is shared, everyone can read it, including in the history after deletion: removing the line is therefore not enough, the exposed key must be revoked and replaced. Moreover, an extract containing names and contact details of supplier contacts is personal data: sending it to an external service is a processing within the meaning of the GDPR, for which the company is accountable.
The controls are of two kinds. A technical control looks for secrets in the code before they leave, as explained in Secrets in code and Git history and Secrets scanning. An organisational control sets which data may be sent to which tool, according to the sensitivity of the application. The first without the second lets the use move elsewhere, the second without the first cannot be verified.
Who owns the generated code, and under which licence?
The status of code produced by an AI is not assumed, it is verified. A licence is the text by which the author of a piece of code sets the conditions for reusing it. Some licences are permissive, others require publishing, under the same conditions, everything that reuses them.
Two questions arise, and neither is settled with a tool. The first: can the generated code reproduce a passage published under a licence whose conditions the company does not accept? A model trained on public code can, rarely, serve back passages of its training set: the GitHub Copilot documentation acknowledges it. The second: who holds the rights over what was produced? The answer depends on the applicable law, which varies by country, and on the terms of the tool's provider.
On the technical side, you can draw up the list of the application's dependencies with their licences, and compare it with what the company accepts. On the legal side, the question belongs to a lawyer: this training does not give legal advice, and it does not present any control as a guarantee of compliance.
What is an object-level authorisation flaw (IDOR)?
An object-level authorisation flaw lets a user read or modify what belongs to someone else, simply by changing an identifier. The acronym IDOR comes from Insecure Direct Object Reference. It describes a particular case of the "broken access control" category of the OWASP Top 10 2025 ranking.
In the purchasing portal, a buyer opens her order at the address /commandes/4812. If
she replaces 4812 with 4813 and sees a colleague's order, the application does
check who she is, but not who owns the requested order. To authenticate is to
establish identity. To authorise is to check the right to access this precise
object. A model can generate the first half and forget the second, because the
author's natural test is to log in with her own account, and everything displays
correctly.
The control is a simple test: log in with a second account and ask for the first account's object. If the application answers, the flaw exists. An API gateway placed in front of the information system complements this control: it limits the operations allowed to each application. It does not replace the per-object control, because it does not know that order 4813 belongs to another buyer. The application must therefore check the right on each object itself, and the test with a second account remains mandatory. The page Hosting and isolating applications built with AI shows where this gateway sits.
What happens when the author leaves the company?
An application that nobody understands and that nobody answers for becomes a risk, even if it never had a flaw. The buyer changes jobs. The purchasing portal keeps running, its ERP access remains active, and the first security question left unanswered is: who can fix it?
Generated code makes the problem worse, because the author described a need without always knowing how the application meets it. When a dependency publishes a security fix, nobody receives it. When the ERP changes version, the application stops and nobody knows how to repair it.
The control is organisational before it is technical: a named business owner and a named technical owner, written in a document that goes with the application, an expiry date on the authorisation that forces a regular review, and a retirement procedure that also cuts the access rights. The post Your time saved must not become our technical debt details what the IT department expects before accepting the takeover of such an application.
Does the generated code itself contain flaws?
Yes: beyond the five causes, the code the assistant writes can contain security flaws, even when the application works. A study by the vendor Veracode, detailed in the page Vibe coding, found insecure code in about 45% of the programming tasks tested, with no security instruction given to the model. This figure is an order of magnitude from a vendor, not the flaw rate of your application.
The remedy is to check the code like a colleague's: a static analysis (SAST) before the merge, and a test of the running application (DAST) when it is critical. This is what the lesson Classify an application requires according to the class.
Which control stops which risk?
Each risk has a precise control, and no single tool stops all five. The table gathers what precedes. It reads line by line: the cause, the incident it produces, then the control that stops it.
| Risk | What happens | Control that stops it |
|---|---|---|
| Invented dependency | A malicious package is installed under a plausible name | Check that a package exists and is legitimate, pin the versions |
| Secret or data in a prompt | A key or personal data goes to a third party | Secret detection before the push, data rule by application class |
| Rights over the code | An incompatible licence enters the application | Licence list of the dependencies, a lawyer's opinion |
| Object-level authorisation flaw | A user reads another user's object | Test with a second account, check of the right on each object in the application, API gateway as a complement |
| Author leaving | Application without an owner, access rights still active | Named owners, expiry, retirement |
Two of these controls come up everywhere: deciding what an application is allowed to do according to what it handles, and having this decision signed by an authorised risk owner. The lesson Classify an application shows the first, the lesson Governance the second. The SOCLE reference on AI (in French) lists for its part the corresponding technical requirements.
Do these risks also apply to an application that embeds a model?
Yes, and other risks are added: this training only covers those of software produced with the help of AI. An application can be generated by an AI without containing a model, like the purchasing portal. It can also call a model, for example to summarise orders. In that case, specific flaws are added, such as prompt injection, which consists of slipping into a piece of data an instruction that the model will take for an order.
These flaws specific to embedded models are covered by the guide Securing an LLM application (in French). This training looks at what all software written with the help of AI has in common, whether it embeds a model or not.
How can you check that you understood?
Take an application you know and look for the five causes in it. The exercise needs no tool. For each risk, note what you can state today: does the list of its dependencies exist, has anyone ever pasted a key into an assistant, has a second account been tried, is the owner named anywhere. An "I don't know" is a valid result: it points to the control to put in place first.
If the application is used beyond its author, the page Shadow AI explains how to tell it apart from a simple experiment.
Key takeaways
- An application can work and carry risks that are invisible in use.
- An invented dependency becomes a malicious package if an attacker publishes the name.
- What is pasted into a prompt leaves the company, secrets and personal data included.
- The legal status of generated code is verified with a lawyer, no tool guarantees it.
- An object-level authorisation flaw is revealed with a second account.
- An application without a named owner stays in service without anyone fixing it.
- Generated code can also contain flaws: it is checked like a colleague's, by static analysis before the merge.
- Each risk has its own control, no single tool stops them all.
- The flaws specific to embedded models are added to these five risks, they do not replace them.
Knowledge check
These eight questions cover the five causes of incident and the control that stops each one.
Knowledge check
Check what you have learned with this interactive quiz
How it works
- The timer starts when you click Start
- Multiple choice, true/false and short answer questions
- You can move between questions
- Detailed results are shown at the end
Starts the quiz and the timer
Review
(0/0)Score by domain
A domain groups several guides and gets several questions, so its score is more reliable than a per-topic score, which often rests on a single question.
Summary
Strengths
No topic passed at 80% on at least two questions yet.
Weak areas
No topic below the pass mark.
Recommended next
Hints before you try again?
Continue the course: Hosting and isolating applications built with AI: the hosting architecture
A new full quiz with random questions
Work again only on the questions you missed