The n-Category Café

FYI, Motivating coherence-driven inference via sheaves

John, I’d be interested to hear you expand on the last paragraph.

All AI developments are at least potentially dual use, in that a technique intended to promote safety may end up promoting capabilities including capabilities to do dangerous things. RLHF is often cited as an example of this.

Still, if the aim to promote safety surely on balance you’re more likely to than someone trying to promote capabilities. So sure you’re only promoting safety in expected value, but most things in life are like that.

Thx for sharing this development. I am still looking into materials, but they are giving me impression, that more are less they accept NNs as black box things that can be trained on synthetic data (provided by the ethical and normative world models) and whose output can be filtered by the world models. Though, I feel that this control is not enough, NNs can be deceptive and their optimization can require some kind of microscopic theory of NNs.

I guess, there are open questions what happens in NNs? Whether some specific algorithms live in them (as categorical deep learning approaches), or maybe full theories with lot of algorithms live in them, or maybe they (NNs) are just models or embeddings of models of some theories. In the last case - knowing - that toposes host every model of type theories (even HOL, HoTT), NNs (or their Euclidean? spaces of activations of neuronal layers) as the embeddings or some other kind of translation of toposes.

“Toposes and stacks of deep neural networs” was promising article in this direction, but, sadly, it has not taken off follow ups.

It may be quite impossible to get some results from statement “theory lives in NN”, but theorems in the form “if theory (e.g. as contextual category) is having such and such invariants and particular NN architecture is having such and such invariants, then 1) the learning rate is bounded by… some formula on them; 2) the output if having such and such invariants and they can be used for checking compliance wrt norms (again some theory with such and such invariants)” could be attained, I guess.

The model theory/neural network interface or embedding in activation spaces can be the hardest thing, because Euclidean spaces are not capable to keep toposes, toposes can not be embedded in Euclidean spaces. Actually, it is quite funny, because one expects the neural networks to express everything, and yet - no logic, that is living in toposes, can not be embedded inside them.

So, such kind of microscopic theory of NNs could be the real solution to the explainability, manageability and controllability of NNs. It could be the complete solution to the safety problem, so, Prof. Baez could join in such efforts without any reservations. Yet, of course, if such explainability is achieved for NNs then it can be applied to human brains as well and, huh, I guess there could be some benefits and lot of problems as well.

In 1954, Frederick Brown wrote a science fiction story titled “The Answer”. It’s so short I can reproduce it here.

“Dwan Ev ceremoniously soldered the final connection with gold. The eyes of a dozen television cameras watched him and the subether bore throughout the universe a dozen pictures of what he was doing.

He straightened and nodded to Dwar Reyn, then moved to a position beside the switch that would complete the contact when he threw it. The switch that would connect, all at once, all of the monster computing machines of all the populated planets in the universe – ninety-six billion planets – into the supercircuit that would connect them all into one supercalculator, one cybernetics machine that would combine all the knowledge of all the galaxies.

Dwar Reyn spoke briefly to the watching and listening trillions. Then after a moment’s silence he said, “Now, Dwar Ev.”

Dwar Ev threw the switch. There was a mighty hum, the surge of power from ninety-six billion planets. Lights flashed and quieted along the miles-long panel.

Dwar Ev stepped back and drew a deep breath. “The honor of asking the first question is yours, Dwar Reyn.”

“Thank you,” said Dwar Reyn. “It shall be a question which no single cybernetics machine has been able to answer.”

He turned to face the machine. “Is there a God?”

The mighty voice answered without hesitation, without the clicking of a single relay. “Yes, now there is a God.”

Sudden fear flashed on the face of Dwar Ev. He leaped to grab the switch. A bolt of lightning from the cloudless sky struck him down and fused the switch shut.”

This is what many people are terrified about when it comes to AI but this is utterly ridiculous. AI is not going to turn into a god that will enslave humanity. AI has no trace of consciousness what’s so ever. it has no more consciousness than a chair in your living room. The problem is people mistaking AI for a person, when it not a person at all. It is easy to fool people into thinking they are talking to a person. It is the very people who are raising the alarm bells about AI who are the ones being most fooled. This problem will get worse with increasingly realistic AI generated video avatars, and lifelike humanoid androids. So what do we do? You try to educate the public that this thing is not a person. There are things we should be worried about. We should not allow military drones to use lethal force without a human signing off on it. People in self-driving cars need to pay attention,and be ready to take control at any time. There is a real problem in education where students use AI to cheat. Does that mean we can’t have homework because the teacher can’t know whether the student used AI? This is a problem in physics education, which you can read about here.

https://physics.aps.org/articles/v18/147