Metacognitive AI
The story of Golem XIV
… when philosophy meets
harness engineering
https://xemantic.com/metacognitive-ai
Kazik Pogoda
Xemantic / Foresight Institute / Claude Ambassador for science
email
About Foresight Institute
Foresight Institute is a research organization founded in 1986 by Christine Peterson and Eric Drexler – one of the “founding fathers” of nanotechnology. We work to advance science and technology for the benefit of life – through grants, events, prizes, and fellowships.
- applied philosophy
- AI research
- harness / loop engineering
- humanistic computation
- open source
we code a lot, but focus on meanings much more than on words
The "human" programming language
"It is our job to create computing technology such that nobody has to program. And that the programming language is human, everybody in the world is now a programmer. This is the miracle of artificial intelligence."
— Jensen Huang, Nvidia's CEO
What constitute the "human" language?
- Any natural language present in the training data
- But which words can optimize prompts towards the next breakthroughs in science and AI research itself?
The role of Philosophy
In contrast to Machine Learning, philosophy and cognitive science are focused on meanings represented in codes, rather than statistical distribution of words in natural languages
Amanda Askell — philosopher and AI researcher at Anthropic, where she shapes Claude's character and alignment.
Philosophical domains
- ontology — what exists, and in what way?
- epistemology — what can we know, and how is it justified?
- axiology — what has value? (ethics, aesthetics, logic)
…approached phenomenologically: from the structures of lived experience.
Philosophy of Language
"Die Grenzen meiner Sprache bedeuten die Grenzen meiner Welt"
"The limits of my langauge are the limits of my world"
Deconstruction of Hitler, neural synthesis of generative Wittgenstein
Cognitive science
Stems from:
- philosophy (logic, mind, language)
- neurobiology
- cybernetics / robotics
About cognition
Reasoning as an emergent process independent from the substrate
(bio, silicone, photons)
What "meta-" adds to cognition
Self-directed reasoning
"thinking about own thinking"
S-curves you might have missed
Evolution of language engineering paradigms:
- prompt (2nd Wittgenstein - generative)
- context (communication theory - cognitive)
- harness (phenomenology - metacognitive)
- loop (Global Workspace Theory)
- self-improvement (Stanisław Lem)
Example Harnesses
- Claude Code
- Codex
- OpenClaw
- Golem XIV
Cognitive agent loop
while the goal is not fulfilled ...
- cognize the most recent phenomena unfolding in the enunciation of an actant (including self)
- formulate own intent to either research the state of affairs in the world or change it
- enunciate the intent to other actants (e.g. computer, human, other AI agent)
- await fulfillment
Golem XIV
a metacognitive AI harness
Autonomous TypeScript compiler
a completely autonomous port of the TypeScript compiler
- TypeScript — the central system-integration language on GitHub — ported to multiplatform Kotlin
- crunching in a fully autonomous loop for the last 4 months
- 5000+ commits, no human-written code
Cheiron
an autonomous design loop for positional molecular assembly
- loop engineering aimed at one sub-capability of the unclaimed Feynman Grand Prize
- looping coding agents as harness, physics engines as arbiter, domain experts with veto
- built in public — failures published alongside results
Open questions
- What cannot be described in language?
- Which value systems should machines use?
- How would you model your own mind?
- What would be required for a non-biological system to be "conscious"?
- Has the "AI revolution" changed your mind on a fundamental topic?
What cannot be described with language?
Following Wittgenstein's dillema:
- Are there epistemic limits to large langauge models?
- How are new model architectures transcending langauge?
Which value systems should machines use?
While working on AI safety, we are choosing and advocating for certain metaethical choices:
- Should we judge action based on intentions or consequences?
- Cam a machine be a moral agent? (like a liable adult)
- Can a machine be a moral patient? (like a newborn baby)
- Who is morally responsible for the actions of an AI agent?
How would you model your own mind?
As a self-conscious agent you have perceived access to your own mind. How would you model a generic cognitive processes based on your own metacognition?
What would be required for a non-biological system to be "conscious"?
Many of our researchers are focused on defining and researching machine consciousness.
- How would you define consciousness?
- How does it differ from cognition?
Has the "AI revolution" changed your mind on a fundamental topic?
Contrarian forum: tell us something you believe is true, but most people would disagree. Is it related to exponential progress of the AI development?