Appearance
Monitor Phase
In the monitor phase, governance continues after the implementation. You connect Sources, start tests on a schedule, and keep the Evidence and the risk values current when the AI System changes.
ScopeSet the AI System and the governance scope
ImplementDo the Controls and attach Evidence
MonitorKeep the governance current
AuditPrepare the internal audit and the exports
Outcome
After this phase, you have:
- Sources that give Modulos the signals of the AI System in production
- tests that examine the conditions that are important to you, on a schedule
- a remediation loop that connects each Failed result to the governance work
Time to first value: 60–120 minutes
Prerequisites: an AI System and the AI System Owner role. To add tests, the Editor role is sufficient.
Path at a glance
1
Connect Sources
Add the Sources that give metrics to the AI System
2
Add tests
Change each governance question into a condition
3
Set schedules
Start the tests on a schedule and examine the results
4
Correct the work
Connect failed results to Controls, Evidence, and risk
Step 1: Connect Sources for live signals
Goal: give the tests the signals of the AI System.
Where in Modulos
AI System → Settings → Sources
Do this
- Select Add Source and add a Source that gives metrics, for example Prometheus, Datadog, Azure, or AWS.
- Make sure that the Connection column shows Healthy for the Source.
- If a Source type is not available, an Organization Admin can turn it on in
Organization → Source & Connectors.
If your organization activates Modulos Client, each AI System shows it with its Sources. Your code sends the metrics to Modulos through it. See Sources.
You are done when
- the tests can read metrics from the Source
Step 2: Add tests as governance signals
Goal: write each governance question as a condition that Modulos can examine.
Where in Modulos
AI System → Runtime Inspection
Do this
- Select New Test and make a Metric Test for each condition that is important to you.
- Select the Source, the metric, the operator, and the value of the condition.
- Open the test and select Associate to a Control to link the test to the Controls that it monitors.
- For a check that an AI agent does from instructions in natural language, make an Agent Test. See Agent Tests.
You are done when
- each test has a clear purpose and a condition that Modulos can measure
Step 3: Start tests on a schedule and monitor the results
Goal: get test results continuously.
Where in Modulos
AI System → Runtime Inspection → select a test → Edit → Scheduleto set the schedule of a Metric TestAI System → Runtime Inspection → Create Jobto start many tests, or Agent Tests, on one scheduleAI System → Jobsto see the Jobs that start the scheduled testsAI System → Runtime Inspection → select a test → Resultsto see the results
Do this
- In the Schedule section of the test, select Schedule Testing.
- Set a Frequency (Daily, Weekly, Monthly, or Quarterly) that agrees with the risk.
- Set the Time, and select Save.
- Examine the results on the Results tab of each test.
Modulos starts each scheduled test as a Job. For the schedule rules, see Tests & Schedules and Jobs.
You are done when
- each test gives a result on its schedule: Passed, Failed, or Error
Step 4: Remediate and keep governance current
Goal: connect the test results to the governance work.
Where in Modulos
AI System → ControlsandAI System → Evidencefor the remediation workAI System → Risksto quantify the risks again when the AI System changes
Do this
- When a test fails, find the cause and correct it.
- Attach the new Evidence to the Control that the test monitors.
- Set the Control to Executed again.
- If the AI System changes, quantify the affected threats again with Recalculate.
- Add a comment on the Control when Comments and Logs does not tell the cause of a change.
A Failed result, and an Error result of a Metric Test, set each mapped Executed Control to Not Executed. For the rules, see Results & Remediation. Modulos logs each status change in Comments and Logs.
You are done when
- each Failed result goes back into the Controls, the Evidence, and the risk values
Next
Audit phase
Prepare the AI System for an internal audit
Engineer & Integrations path
The path for the persons who connect Sources, Connectors, and API tokens