Skip to content

Monitor Phase ​

In the monitor phase, governance continues after the implementation. You connect Sources, start tests on a schedule, and keep the Evidence and the risk values current when the AI System changes.

ScopeSet the AI System and the governance scope
ImplementDo the Controls and attach Evidence
MonitorKeep the governance current
AuditPrepare the internal audit and the exports

Outcome ​

After this phase, you have:

  • Sources that give Modulos the signals of the AI System in production
  • tests that examine the conditions that are important to you, on a schedule
  • a remediation loop that connects each Failed result to the governance work

Time to first value: 60–120 minutes
Prerequisites: an AI System and the AI System Owner role. To add tests, the Editor role is sufficient.

Path at a glance ​

Step 1: Connect Sources for live signals ​

Goal: give the tests the signals of the AI System.

Where in Modulos

  • AI System → Settings → Sources

Do this

  • Select Add Source and add a Source that gives metrics, for example Prometheus, Datadog, Azure, or AWS.
  • Make sure that the Connection column shows Healthy for the Source.
  • If a Source type is not available, an Organization Admin can turn it on in Organization → Source & Connectors.

If your organization activates Modulos Client, each AI System shows it with its Sources. Your code sends the metrics to Modulos through it. See Sources.

You are done when

  • the tests can read metrics from the Source

Step 2: Add tests as governance signals ​

Goal: write each governance question as a condition that Modulos can examine.

Where in Modulos

  • AI System → Runtime Inspection

Do this

  • Select New Test and make a Metric Test for each condition that is important to you.
  • Select the Source, the metric, the operator, and the value of the condition.
  • Open the test and select Associate to a Control to link the test to the Controls that it monitors.
  • For a check that an AI agent does from instructions in natural language, make an Agent Test. See Agent Tests.

You are done when

  • each test has a clear purpose and a condition that Modulos can measure

Step 3: Start tests on a schedule and monitor the results ​

Goal: get test results continuously.

Where in Modulos

  • AI System → Runtime Inspection → select a test → Edit → Schedule to set the schedule of a Metric Test
  • AI System → Runtime Inspection → Create Job to start many tests, or Agent Tests, on one schedule
  • AI System → Jobs to see the Jobs that start the scheduled tests
  • AI System → Runtime Inspection → select a test → Results to see the results

Do this

  • In the Schedule section of the test, select Schedule Testing.
  • Set a Frequency (Daily, Weekly, Monthly, or Quarterly) that agrees with the risk.
  • Set the Time, and select Save.
  • Examine the results on the Results tab of each test.

Modulos starts each scheduled test as a Job. For the schedule rules, see Tests & Schedules and Jobs.

You are done when

  • each test gives a result on its schedule: Passed, Failed, or Error

Step 4: Remediate and keep governance current ​

Goal: connect the test results to the governance work.

Where in Modulos

  • AI System → Controls and AI System → Evidence for the remediation work
  • AI System → Risks to quantify the risks again when the AI System changes

Do this

  • When a test fails, find the cause and correct it.
  • Attach the new Evidence to the Control that the test monitors.
  • Set the Control to Executed again.
  • If the AI System changes, quantify the affected threats again with Recalculate.
  • Add a comment on the Control when Comments and Logs does not tell the cause of a change.

A Failed result, and an Error result of a Metric Test, set each mapped Executed Control to Not Executed. For the rules, see Results & Remediation. Modulos logs each status change in Comments and Logs.

You are done when

  • each Failed result goes back into the Controls, the Evidence, and the risk values

Next ​