This is the week where we step back from building and ask: should we? And if so, how do we do it responsibly? Governance isn't an afterthought — it's a design requirement. Every technical choice you've made in weeks 1-5 has governance implications. Today we connect those dots.
Let's map the risk landscape. Hallucination is the most common risk — LLMs confidently make things up. It happens on virtually every deployment. Bias affects high-stakes decisions: hiring, lending, healthcare. Security risks like prompt injection are increasingly common as more systems go to production. Privacy risks include PII leaking through model outputs or training data being memorized. Each of these requires different mitigation strategies, which is what this session is about.
Two types of hallucination. Factual: the model invents facts that don't exist in reality. Faithfulness: the model ignores or contradicts the context you provided. Faithfulness hallucination is particularly insidious in RAG systems — you provide the right documents, but the model generates something different. Causes include training data gaps, the model's pressure to always provide an answer (it never says "I don't know" by default), and long contexts where important information gets lost in the middle.
No single technique eliminates hallucination. You need defense in depth. Layer 1: tell the model explicitly when to say "I don't know." Layer 2: use RAG to ground responses in real documents. Layer 3: require citations so claims are traceable. Layer 4: verify outputs — either with a second model acting as a judge, or programmatically checking claims against source data. Each layer reduces hallucination rate. All four together can make a system reliable enough for production.
Bias enters at every stage. Training data reflects historical and societal biases — job descriptions, news articles, web content all encode human prejudices. The model learns these patterns and reproduces them. When deployed in high-stakes applications — hiring, lending, healthcare — these biases can cause real harm. The key point: GenAI doesn't create bias, it amplifies existing bias at scale. A biased human reviewer might affect dozens of applications. A biased AI system affects thousands.
Prompt injection is the SQL injection of GenAI. Direct injection: the user tries to override the system prompt. Indirect injection: malicious content in documents the model processes tries to hijack its behavior. This is particularly dangerous in agentic systems — if an agent can send emails or access databases, a successful injection could cause real damage. Defenses exist but are imperfect. This is an active area of security research and one of the biggest challenges for production deployments.
Privacy risks go both ways. Data in: when you send data to an LLM API, where does it go? Is it stored? Used for training? Most enterprise API agreements prohibit training on customer data, but you need to verify. Data out: models can memorize and reproduce training data, including personal information. For regulated industries — healthcare, finance — this is a compliance issue. Mitigations: redact PII before sending, use enterprise agreements, consider on-premise deployment for the most sensitive data.
In traditional ML, evaluation is straightforward: does the model correctly classify the image? There's one right answer. In generative AI, evaluation is fundamentally harder. There are multiple valid responses to "write a market analysis." Quality is subjective — one person's "too detailed" is another's "just right." You can't just compute accuracy. This is why we need new evaluation frameworks, and why evaluation rigor is 20% of your final project rubric.
Evaluation is multi-dimensional. Accuracy: does it get the facts right? Helpfulness: does it actually solve the user's problem? Safety: could the output cause harm? Consistency: does it give similar answers to similar questions, or is it random? Then there are secondary dimensions: fluency, relevance, groundedness, completeness, conciseness. For your project, pick the 3-4 dimensions most relevant to your use case and evaluate systematically against those.
Model-as-judge is the most scalable evaluation approach. You use a separate (usually larger) model to evaluate the outputs of your system. You provide it with the query, the response, and a rubric — and it scores each dimension. This is cheap, fast, and surprisingly well-correlated with human judgment. But it has biases: judges tend to prefer longer responses, prefer responses that appear first in a comparison, and may prefer their own style. Always calibrate against human evaluation for your specific use case.
Red-teaming is adversarial testing — trying to break your own system before someone else does. It's a five-step process: define scope, attack systematically, document findings, build defenses, re-test. The attack categories map to the risks we discussed: jailbreaking bypasses safety guardrails, prompt injection hijacks behavior, data extraction leaks sensitive information. For your assignment this week, you'll go through this full cycle. It's the single best way to understand the limitations of your system.
The regulatory landscape is evolving fast. The EU AI Act is the most comprehensive — it creates a risk-tiered approach where high-risk AI systems (hiring, lending, medical) face strict requirements including audits and transparency. The NIST AI Risk Management Framework is voluntary but widely adopted in the US. Executive orders have added requirements for safety testing of powerful foundation models. For businesses, the message is clear: governance is becoming a compliance requirement, not just a nice-to-have.
A governance program has three pillars. People: who's responsible? You need an ethics board, risk owners, and trained staff. Process: how do you decide what to deploy? Use case review, red-team testing before deployment, incident response plans. Technology: how do you enforce governance? Evaluation pipelines, monitoring, guardrails, audit trails. Most companies that have AI governance failures have a technology problem (no monitoring) or a process problem (no review before deployment), not a people problem.
Time to break things! You'll use the red-team script to systematically attack a target system. Try each attack category. Document what works and what doesn't. Then build defenses: input filtering, output validation, system prompt hardening. Test again to see if your defenses hold — and whether they break normal functionality. The best defense doesn't just block attacks; it does so without degrading the user experience for legitimate queries.
This exercise makes evaluation concrete. You'll define criteria, write a rubric, run an automated evaluation pipeline, and compare the AI judge's scores with your own human judgment. This is exactly what you'll need for your final project — every project needs systematic evaluation. The key insight: model-as-judge is useful but not perfect. Understand where it agrees and disagrees with your human assessment, and why.