Alberto D. Mendoza

I had to right a lot of wrongs before this series could continue. This is the second part of my correction journey. The ontology has been corrected and is now released as v2-beta.

In the acknowledgments of What Enforcement Costs I wrote that Vladimir Alexiev “caught a metamodel mistake I’d carried through years of working with ArchiMate, and it’s going to leave a mark.” That was true, and it was the smallest possible way of saying it.

The mistake was not in the ontology, and it was not a rule that fired wrong. It was in how I read the specification’s normative tables, and because everything else in this series is built on top of those tables, every layer inherited the loss.

It takes one sentence to state. Appendix B’s relationship tables are case-sensitive. Uppercase means the relationship may be drawn. Lowercase means it may only ever be the conclusion of a derivation. I read them as a flat permitted / not-permitted matrix, and so did the machine-readable table I built everything on. The relationships.xml I started from, by way of Archi, holds 11,569 entries. Not one of them is uppercase. There was no case for me to lose, because the file never carried any, and I published it that way in post 5 as a 62 x 62 grid of letters.

In my defense, the official 3.2 specs nowhere there is mention of upper case vs lower case on the relationship matrix. However, the newly released specs for 4.0 does state that upper case denotes a direct relationship, lower case denotes a derived relationship. 

One question from Vlado about the B.2 matrix turned my world upside down.

What the case distinction is worth

Re-encoding the tables with case preserved turns one number into two. There are 10,610 permitted (element, relationship, element) combinations across 61 concepts and 3,721 concept pairs. (The grid lost a row and a column on the way. Junction came out, for reasons I come back to below.) Of those:

Forty-five percent of what the specification permits is not something you are allowed to draw. That is not a rounding detail. Look at it per relationship type and it gets sharper:

Nearly nine out of ten permitted serving, flow and triggering combinations are ones you may not draw. Those are the relationships a modeller reaches for most. They are also the ones a derivation engine produces in bulk. A tool that treats the matrix as flat accepts all 4,788 of the relationships the specification reserves for derivation.

Two predicates now carry the distinction, both generated from the tables:

# May a modeller draw this? including as a derived relationship?
ASK { archimate:ApplicationComponent archimate:permitsRealization archimate:ApplicationProcess }
# true
ASK { archimate:ApplicationComponent archimate:permitsDirectRealization archimate:ApplicationProcess }
# false

Every permitsDirect triple is also a permits triple. The gap between the two is the thing I had been unable to express.

Having the matrix as queryable RDF changes what a derivation rule can do. The rule looks up whether the specification permits the relationship it is about to produce, between those two concept types, and declines to fire when it does not. Viktor Senderov had told me the restriction rules were the wall, because forward chaining has no negation and a restriction reads as “R holds unless X.” Requiring the permission as a premise says the same thing without the unless. The matrix is that premise.

How it broke derivation

It broke two things. The second is the one that cost me.

The first is that validation came out too permissive. The Level 2 and Level 3 files in post 5 were generated from the uncased tables, so they enforce permitted rather than permitted directly. A validator built that way accepts relationships the specification reserves for derivation. I come back to this at the end, because it is still true today, and because the reason it is still true is not the reason I expected.

The way that cost me more is that I had built a conformance harness and it was measuring the wrong thing.

The cased tables are an oracle. Feed a ruleset the uppercase half, the relationships a modeller actually drew. Run the derivation rules. Check whether the lowercase half comes out. I built that, drove conformance up to 98.2%, and felt good about it.

The number was meaningless, because the lowercase half is not one thing. It merges relationships reachable by a valid derivation (DR1–DR8, which the specification says you may assert) with relationships reachable only by a potential derivation (PDR1–PDR12, which B.3 says “might be relevant but may also be wrong… It is up to the modeler to decide”). Scored against the whole lowercase half, a ruleset is rewarded for producing potential derivations. It gets points for asserting precisely what the specification says you must not assert without a modeller’s judgement.

Several of my rules were written generically, over a relationship category rather than the specific relationship the specification names. A rule written over “dynamic relationships” covers both flow and triggering. The specification’s rule covers one of them. So each generic rule silently included its potential twin, the oracle scored the extra output as correct, and conformance went up.

The worst case was transitivity. Written over the category instead of triggering alone, one rule materialized flow transitivity across the whole model. On a 2,904-relationship model that turned 193 asserted flows into 53,507 derived ones. Every single one was permitted by the matrix, so precision stayed at 100% and the oracle reported nothing wrong.

Precision against a permitted-set is not the same as being right. That sentence cost me about three weeks.

Scope each rule to the subject the specification actually gives it, and the lowercase half accounts for itself: every entry is reachable either by a valid rule or by a potential one, with nothing left over. Valid derivations get materialized. Potential ones get computed on demand and offered as suggestions, never stored. That division is now the architecture, and it falls directly out of reading the tables properly.

The constraints I deleted

The other half of this cleanup was subtraction. Seven SHACL shapes were sitting in the validation files marked sh:deactivated true. Each had been switched off at some point, again, because they were either expensive or flatly wrong.

They are gone. The reasons are worth separating, because they are not the same reason.

Three were superseded by the cased tables. CoreServiceRealizesInternalBehavior, CoreBehaviorRealizesStrategyBehavior and CoreStructureRealizesResource were all procedural checks on the direction of realization: SPARQL walking the class hierarchy to test whether a particular realization pointed the wrong way. archimate:permitsDirectRealization now states the same thing declaratively, for every type pair at once, and you can query it rather than only fail against it. A lookup beat three hand-written traversals.

One could never fire. DerivationDepthLimit was meant to catch derivation chains more than six hops deep. Its premise matched zero statements in any repository, because it counted provenance sources on a single statement rather than depth along a chain. It had been deactivated rather than fixed, which is how a shape that cannot work survives for months looking like coverage.

Three were advisory shapes that had been switched off because, as the last post explained at length, sh:Warning in RDF4J’s ShaclSail still blocks the transaction. There is no advisory mode. A shape you want to be advisory has to be deactivated, and a deactivated shape is a comment that still costs a parse.

Deleting the depth-limit shape left a real gap. A broken check should be rewritten, not switched off. Specialization now has proper validation: cycle detection, and a chain-depth check that measures depth. Both are written against Appendix B rather than against a guess.

Junction, and why the fix was not a table row

Junction used to have a row and a column in my tables, carrying all eleven relationship letters. It is now absent, and that is deliberate.

A junction is a connector, not an element. It merges or splits relationships of one type, and which type is decided per diagram, not per concept pair. Forced to assign a case to it in a static table, the encoding had to say “all eleven, directly assertable,” which is a statement about what you may draw on a canvas and not a statement about which relationships are permitted between two concept types. Those are different claims and only the second belongs in a permission matrix.

Taking it out had a consequence I did not anticipate. With the permitted matrix acting as a guard on derivation, a junction endpoint has no row, so the guard never matches, so every derivation across a junction silently failed to fire. Four dynamic chains in my test model were deriving nothing at all, and nothing anywhere reported a problem.

Junction transparency fixed it. The rule treats a chain through a junction as equivalent to a direct relationship between the real endpoints, taking the relationship type from the legs. Homogeneity comes for free: the rule binds one variable across both hops, so a junction cannot mix types by construction. On ArchiSurance that single rule accounts for roughly half of everything derived.

Other small things, which were not small

The rest of the release is unglamorous but they are the kind of thing that hurts later.

@base to @prefix. Every file used @base with <#TermName> local references. It works, and it makes every term IRI depend on how the file was loaded. All files now declare an explicit prefix and every term carries rdfs:isDefinedBy. Terms resolve to the same IRI regardless of where the file came from.

The Model / Concept / Aspect / Layer hierarchy was wrong in a way that only showed up when you queried across it, and archimate:hasConcept did not exist at all, so there was no way to ask what a model contains.

Multilanguage Name and Documentation. Originally, That says an element has at most one name. The Exchange format allows a name per language, so any multi-language model was formally inconsistent the moment it loaded. The constraint is gone; names may now be language-tagged and plural, which is what the format always intended. Identity remains as the only owl:FunctionalProperty.

Directed Associations. for association relationships, which ArchiMate added later and I had never encoded.

Profile is now distinct from Specialization. Marking a class as belonging to a profile and specializing a concept are different acts, and they had been conflated.

The relationship-strength ordering moved out of the ontology. It is an input to a derivation procedure, not part of the language. No diagram, no exchange file and no conformance statement depends on it. It now lives with the rules that consume it.

The SKOS vocabulary is complete and published, with Spanish labels, as HTML alongside the Turtle.

Where derived relationships live

Validation has not been rebuilt from the cased tables. It still enforces permitted rather than permitted directly, so as I write this it accepts 799 type-pair combinations as drawable that Appendix B sanctions only as derivations. 582 of those are realization and 184 are access. Six more are not permitted at all.

I assumed that was a chore. Teach the generator to emit the shapes against permitsDirect instead of permits, regenerate, done. It is not a chore, and working out why is the most useful thing this correction turned up.

RDF4J’s ShaclSail validates on commit, and a violation rejects the entire transaction. Derived relationships are written into the same graph as everything else. So the moment validation enforces permitsDirect, the INSERT that materializes derivation fails. The validator rejects, in bulk, exactly the relationships the rules exist to produce. Every one of the 15,310 is by definition a lowercase entry. That is what lowercase means.

Which means the defect had impact. The only reason derivation can write today is that validation is too permissive. Fix the validator in isolation and you break the engine.

While the tables were flat, the design never had to say where derived relationships live or what vouches for them. Now it does.

I think the answer is that they do not belong in the store SHACL guards. Validation and derivation are two different assurance mechanisms for two different kinds of fact. SHACL governs what a modeller asserts. Its job is Appendix B’s matrix, and the matrix states what a modeller may draw, which is permitsDirect and nothing else. Derived relationships are machine output, and what vouches for them is the conformance oracle: run the cased tables through the rules once, offline, and check that the uppercase half reproduces the lowercase half. That is a stronger guarantee than a shape can give, and it is paid for at build time instead of on every commit. Re-checking it on every write is the exact cost profile the last post spent four thousand words warning about.

So validation files should enforce permitsDirect, and the order matters: derived output has to leave the validated write path first. The cleanest way is a separate analysis store. The modelling repository is SHACL-backed and holds what modellers drew. The analysis graph holds asserted plus derived relationships and runs no validator. Queries go to the second. Nothing needs provenance-aware shapes, and no shape has to test for archimate:confidence.

I have not done it. The tables are right, the axioms generated from them are right, the derivation rules are scoped to the subjects the specification gives them, and the validation layer is knowingly too permissive by 799. That is deliberate for now. Tightening it before the boundary is drawn would take the engine down with it.

Fixing the tables exposed a line the architecture had never been forced to draw. Drawing it is the next piece of work.

Beta v2 is out

All of it, thirty commits, is released and now tagged as v2-beta. Keeping the original case-insensitive relationship.xml work as v1-beta. 11,569 relationship entries, every one of them lowercase. v2-beta ships 5,822 uppercase and 4,788 lowercase (with Junction removed). Everything else in this post follows from that.

The axioms and the conformance fixtures are generated from those tables by a script that ships in the repository, and the derivation rules are scoped to the subjects the specification gives them.

Clone the repository, run the generator, and you get the same 10,610 permitted combinations and the same split between them. The encoding is reproducible rather than asserted, which is the only way I know to make a claim like “forty-five percent of the specification is not drawable” checkable by someone who does not trust me.

When will there be a Final release? ArchiMate is ultimately a visual diagram, with views and object and relationships on those views. That is still a lot of work that needs careful definition.

Next Post

Querying the graph: SPARQL patterns, impact analysis, and what 600 hand-drawn relationships actually entail.

The ontology is independent and is not an official publication of The Open Group. ArchiMate is a registered trademark of The Open Group.