As to (1): the protocol occasionally specifies what you do with a payload but does not specify making that result visible.

But if you've ever studied testing you know that, in general, you only want to test public behavior.

This automatically makes testing difficult, even if you have completely set up permissions: there's no requirement of visibility. So it is always possible to be conformant (or not) but have no way to prove that.

Exacerbated by a lack of client interface.

2/

Attempting to address this is challenging at best because it requires specifying that which AP refuses to specify: that side effects be visible to _someone_.

It requires a degree of consistency in interface that we simply do not have and no extant projects work within, and not having a client interface means that your only options for testing it involve things like the mastodon client API, which could be a spec unto itself.

Also no requirement for things you need to even get your bearings

3/

In just base AP you have no webfinger. No nodeinfo. There's no extension registry to know what you are looking at in payload responses. Nothing that allows you to orient yourself as to what server you are looking at, nor what that server's behaviors or quirks may be. There's no way to know how it handles… well, anything, really, but more on that in part (2). You can't assume oauth, or http signatures.

So already in order to test conformance you require a lot of Things that aren't AP.

4/

This brings us to (2): the lack of specified side effects.

If I hand you a Flag, what are you supposed to do with it?

How can I prove you did _anything_ with it?

Simply implementing AP is not "enough" to talk to the broader fediverse and it shows. There are myriad side effects that you need to handle to do this (or at least do this ethically) and none of them are specified behaviors.

AP is "simple" in the sense of just implementing the spec, but that doesn't do what you want.

5/

Finally, there's (3): the payload's somewhat contradictory specification.

Absolutely no one today implements AP fully. No one.

* https://www.stevebate.net/nothin-implements-activitypub/
* https://funfedi.dev/support_tables/generated/score/

There is simply too much complexity in what payloads can come across the wire, and you can choose either to say:

1. You must implement a _specific_ form to be conformant.
2. You must support every form to be compliant and lack of support for certain structures reduces conformance.

But which is it?

6/

Some of these individual sub problems are tractable at the level of the SWICG either through advisement or by chartering a new group, but all of them together make the overall problem essentially intractable by nature.

To get a full conformance test we first need to define "conformance," and there are no conformant systems today on which to base such a statement, no governance capable of marking such a point, and at best limited incentives for the SWICG to acknowledge this or deal with it.

7/

A project like mastodon could define their own conformance profile, of course, but doing so is _very_ challenging, keeping it up to date is also _very_ challenging, and they have a significant amount of tech debt to overcome already.

People have tackled sub-problems in this domain as well, but always need to find the tradeoff points between

1. A usable conformance suite
2. Anyone implementing said conformance suite even to the degree that the suite is meaningful.

8/

Anyways, I don't know what the solution is from here. We still need a conformance suite. We desperately need one in a lot of ways for what the fediverse wants to be.

But without SWICG or a group like SWF owning it then it will simply not happen. In both of those cases they also have to acknowledge certain realities that they have been loathe to acknowledge.

The problem is not, IMO, a tractable one…

…and yet it is killing us.

9/9

A protocol where four of the five authors are queer resists conformance tests, and in fact makes it hard to even define "conformance" ... just a coincidence? I'm sure Os Keyes would have something interesting to say about this!

I also don't know what the solution is. Even though the software engineer in me shudders at the idea, is it a useful thought experiment to abandon the idea of conformance tests and explore what other sociotechnical approaches could help?

@hrefna@hachyderm.io

@jdp23 @hrefna

> A project like mastodon could define their own conformance profile, of course

there's an apparent lack of will in doing so. i've actually approached them about taking steps toward making these definitions available, but no bites.

i also think there's a political convenience in saying it's "activitypub" instead of "the mastodon protocol". everyone wants to be "activitypub" because that's what has the w3c stamp of legitimacy.