现代神经网络通常被描述为过度参数化，因为它们所包含的可训练参数远多于传统统计模型，甚至常常多于训练样本的数量。然而，这些模型既可以高度拟合训练数据，同时又具有良好的泛化能力。这个表面上的矛盾，推动了大量关于良性过拟合、双降、隐式偏置、特征学习以及训练动力学的研究。但到目前为止，仍然没有一个令人信服的理论，能够从根本上清楚解释过度参数化。

然而，这一讨论建立在一个很少被认真审视的假设之上：当我们称一个模型为过度参数化时，实际上已经默认我们大致知道这个学习问题需要多少自由度。但在大多数情况下，我们并不知道。大量参数只能说明我们为学习提供了多大的计算自由度，并不能告诉我们这个问题本身究竟需要多少自由度。

学习是一个反问题

其根本原因很简单：学习是一个反问题。 我们从数据和边界条件出发，试图恢复事先未知的关系。因此，需要学习的结构本身就是未知解的一部分。

出于同样的原因，我们也不知道这个结构究竟需要多少自由度。在学习开始之前，我们并不知道这些关系的完整复杂性、它们之间如何相互作用、哪里存在不连续，也不知道需要多少自由度才能充分表示这些关系。

因此，神经网络通常会被有意赋予很大的参数预算。自由度太少，可能使所需要的结构根本无法形成；而最低需要多少自由度，通常又无法事先确定。从这个意义上说，所谓的过度参数化，在很大程度上只是为了求解一个未知反问题而预留的实际安全余量。

学什么很重要

所需的自由度不仅取决于数据量，还取决于我们想从数据中学什么。我们想学什么，决定了学习空间；单靠数据本身并不能决定学习空间。这一区别可参考关于学习空间的数学思考。

ImageNet 就是一个简单的例子。同一组图像数据，可以只用来学习一个粗粒度的区分，例如室内与室外；也可以用来区分大约两万个类别。底层图像本身也许大体相同，但这两个任务所定义的学习空间却非常不同，因此所需的自由度也可能非常不同。

所需能力不能单靠数据集的大小来推断。我们要求模型学习什么，决定了学习空间；而学习空间又决定了可能需要多少自由度。 由于学习是一个反问题，这个需求在学习开始之前通常是未知的。

机会

这里的迷思并不是神经网络很大，而是我们似乎已经知道它包含了太多参数。在学习开始之前，我们并不知道这个学习问题究竟需要多少自由度，因此，提供较大的参数预算本身就是一种合理的设计选择。

真正的机会出现在训练完成之后。一旦学习完成，原本未知的结构已经形成，此时问题就从“学习需要多少自由度”，转变为“为了保留已经学到的内容，还需要多少自由度”。

过去的大多数工作主要通过剪枝、量化、低秩近似或其他压缩方法来解决这一问题，但效果并不一致。这些方法通常作用于权重、矩阵、层或统计重要性，却很少明确考虑：压缩之后，已经学习到的流形及其同调结构是否仍然被保留下来。

从深度流形的角度看，这可能才是更重要的问题。如果训练形成的是一个相互连接的分片流形，那么参数减少就不应该只是删除看起来较小或冗余的权重，而应该尽可能保留已学习流形中的关系、连通性和同调结构。

因此，真正的机会也许并不是在学习阶段避免过度参数化，而是先用足够的自由度完成学习，再在保留已形成结构的前提下减少参数。

This article is now listed under “Myth Series” in Deep Manifold, Two Years Later: 2024–2026.

Modern neural networks are commonly described as overparameterized because they contain far more trainable parameters than traditional statistical models, and often more parameters than training examples. Yet these models can fit their training data closely while still generalizing well. This apparent contradiction has motivated extensive research into benign overfitting, double descent, implicit bias, feature learning, and training dynamics. So far, however, there is still no convincing theory that explains overparameterization clearly and fundamentally.

The discussion, however, rests on an assumption that is rarely examined: calling a model overparameterized implies that we already know approximately how many degrees of freedom the learning problem requires. In most cases, we do not. A large parameter count tells us how much computational freedom has been made available to learning, but it does not tell us how much freedom the problem itself requires.

Learning Is an Inverse Problem

The reason is fundamental. Learning is an inverse problem. We begin with data and boundary conditions and attempt to recover relationships that are not known beforehand. The structure to be learned is therefore part of the unknown solution.

Its required degrees of freedom are unknown for the same reason. Before learning, we do not know the full complexity of the relationships, their interactions, their discontinuities, or the amount of freedom needed to represent them adequately.

Neural networks are therefore given large parameter budgets deliberately. Too little freedom may prevent the required structure from forming, while the minimum sufficient amount cannot generally be determined in advance. What is called overparameterization is, in this sense, largely a practical safety margin for solving an unknown inverse problem.

What to Learn Matters

The required degrees of freedom depend not only on the amount of data, but also on what we want to learn from the data. What we want to learn defines the learning space; the data alone does not. This distinction is discussed in Mathematical Considerations for Learning Space.

ImageNet provides a simple example. The same collection of images could be used to learn a coarse distinction such as indoor versus outdoor, or to distinguish among roughly twenty thousand categories. The underlying images may be largely the same, but the two tasks define very different learning spaces and therefore may require very different degrees of freedom.

The relevant capacity cannot be inferred from dataset size alone. What we ask the model to learn determines the learning space, and the learning space determines how much freedom may be required. Because learning is an inverse problem, that requirement is generally unknown before learning begins.

The Opportunity

The myth is not that neural networks are large. The myth is that we already know they contain too many parameters. Before learning, we do not know how many degrees of freedom the learning problem requires, so a large parameter budget is a reasonable design choice.

The opportunity appears after training. Once learning is complete, the unknown structure has already formed, and the question changes from how much freedom is needed to learn to how much of that freedom is still needed to preserve what has been learned.

Most previous efforts have approached this through pruning, quantization, low-rank approximation, or other forms of compression, with mixed results. These methods usually operate on weights, matrices, layers, or statistical importance, but they do not explicitly ask whether the learned manifold and its homological structure are being preserved.

From the Deep Manifold perspective, this may be the more important question. If training forms a connected piecewise manifold, parameter reduction should not simply remove weights that appear small or redundant. It should preserve the relationships, connectivity, and homology of the learned manifold.

The real opportunity, therefore, may not be to avoid overparameterization during learning, but to learn with sufficient freedom first, then reduce parameters while preserving the structure that learning has formed.

This article is now listed under “Myth Series” in Deep Manifold, Two Years Later: 2024–2026.

Differential equations may be among the most important equations for describing the real world. Mathematically, differential equations can be studied through both algebra and geometry. Compared with algebra, geometry offers greater breadth, depth, and flexibility in describing space, change, and the relationship between local and global structure. Shiing-Shen Chern brought modern differential geometry to a new level, while Shing-Tung Yau, in proving the Calabi Conjecture, developed the study of nonlinear differential equations on geometric spaces, what later became known as geometric analysis, pushing differential geometry further into the nonlinear world.

But neither Chern nor Yau focused on the practical question of how to solve differential equations. If a differential equation cannot be solved, it cannot solve a real-world problem; in that sense, the differential equation itself can become a form of “useless beauty.”

Mathematics Becomes Computable

With the emergence of computers, numerical computation began to make it possible to actually solve differential equations. Methods such as the finite element method made large classes of linear and nonlinear problems computable. For a time, some even believed that discrete mathematics and numerical computation could eventually replace much of traditional mathematics. But it soon became clear that when confronted with higher-order nonlinearity, discontinuity, large deformation, and complex boundaries, more computational power alone was not enough. Problems often became impossible to compute, computable but inaccurate, or accurate but prohibitively expensive. This is why increasingly powerful supercomputers could not automatically solve increasingly complex mathematical problems: the computing power was there, but the mathematics had not caught up.

Gen-Hua Shi later developed DDA and the Numerical Manifold Method, reconnecting mathematics with actual problem solving and bringing higher-order nonlinearity, discontinuity, and complex geometry into a computable framework. The word numerical in “Numerical Manifold” may make it sound less purely mathematical than “differential geometry,” but it emphasizes something fundamental: mathematics must not only provide structures and equations; it must also help us obtain solutions.

For numerical computation, good mathematics should ultimately help achieve three things:

• Accurate: the solution must be reliable;

• Fast: computation and convergence must be efficient;

• Efficient: fewer computational resources should be required.

AI is now encountering essentially the same problem. An AI answer is an AI solution, and a neural network itself is a form of learnable numerical computation. If it is numerical computation, there must be mathematics behind it. Therefore, understanding the mathematics of AI requires not only mathematicians, but also people who deeply understand numerical computation. What makes Gen-Hua particularly unusual is that he combines a strong mathematical foundation with decades of work in numerical computation and real engineering problems.

Fixed-Point Class Theory and AI

Gen-Hua spent many of his early years in the Department of Mathematics at Peking University studying fixed-point class theory. Fixed-point theory is concerned not only with individual fixed points, but more fundamentally with the existence, uniqueness, and stability of solutions. All three questions are important for AI: existence and uniqueness are related to why AI may produce multiple possible answers, or even hallucinations, while stability is directly related to the stability of model training and inference.

Fixed-point theory studies points that remain invariant under the action of a vector field or mapping. An AI model also defines a field through its weights and learned transformations; we call this the neural fixed-point field. If we can understand the structure of this neural fixed-point field, we may be able to substantially improve inference efficiency. In Deep Manifold, inference is fundamentally an integration process, and one reason current inference remains inefficient is that the mathematics of inference itself has not yet been fully understood.

The Mathematical Direction of Deep Manifold

Shing-Tung Yau, through the Calabi conjecture, pushed differential geometry into the nonlinear regime. Professor Gen-Hua Shi, driven by problems in rock engineering, pushed geometry and numerical computation further into higher-order nonlinearity and discontinuity. Today, AI raises a new question: if equations, mappings, and geometry themselves must be learned from data, how should mathematics study them?

Deep Manifold hopes to push further:

• from differential geometry to learnable geometry,

• from fixed-point class theory to a learnable fixed-point field,

• from numerical computation to learnable numerical computation,

and, ultimately, to explore one of the mathematical dreams Gen-Hua has spoken of: the unification of forward and inverse problems.

• When Mathematical Beauty Becomes Useful in AI

• The Basic Mathematics Overlooked in AI “When mathematics is but ill understood, ’tis a fearful thing indeed.”

This article is now listed under Architecture in Deep Manifold, Two Years Later: 2024–2026.

微分方程可能是描述真实世界最重要的一类方程。数学上研究微分方程，有代数，也有几何。相比代数，几何对空间、变化、局部与整体的描述更有广度、深度和变化性。陈省身把现代微分几何推进到新的高度；丘成桐为了证明卡拉比猜想，发展了几何上的非线性微分方程，也就是后来所谓的几何分析，把微分几何进一步推进到非线性世界。但陈和丘都没有讲如果解“微分方程“, 微分方程没有解, 就解决不了实际问题, 微分方程本身成了“ 无用美“.

数学走向可计算

计算机出现以后，数值计算开始真正解决微分方程。有限元等方法使大量线性和非线性问题可以计算，一度甚至有人认为，离散数学和数值计算最终可以代替很多传统数学。但很快大家发现，面对高阶非线性、不连续、大变形和复杂边界，仅仅增加算力并不够，常常会出现“算不动，算动了算不准，算准了又算不起”。这也是为什么超级计算机有了越来越大的算力，仍然不能自动解决复杂数学问题：算力有了，数学没有跟上。

石根华老师后来发展 DDA 和数值流形方法，就是把数学与实际求解重新连起来，使高阶非线性、不连续和复杂几何进入可以计算的框架。数值流形这个名字里特意加上“数值”二字，看起来似乎没有“微分几何”那么纯数学，但恰恰强调了一件重要的事情：数学不仅要给出结构和方程，还必须帮助我们得到解。

对数值计算来说，好的数学最终要帮助计算做到三个字：

• 准：解得可靠；

• 快：计算和收敛速度高；

• 省：使用更少的算力和资源。

AI其实重新遇到了同样的问题。AI的回答就是AI的解，而神经网络本身就是一种可学习的数值计算。既然是数值计算，它背后一定有数学。所以真正把AI的数学搞清楚，不能只有数学家，还需要真正理解数值计算的人。石老师非常特别的一点，就是他既有深厚的数学基础，又长期做数值计算和工程问题。

不动点类理论与AI

石老师早年在北大数学系长期研究不动点类理论。不动点理论关心的不只是一个点，更重要的是解的存在性、唯一性和稳定性。这三个问题对AI都很重要：存在性和唯一性与AI为什么会产生多个可能答案、甚至幻觉有关；稳定性与模型训练和推理的稳定有关。

不动点理论研究的对象是向量场（或映射）在其作用下不变的点 ，AI模型也有一个由权重构成的场，我们可以把它称为神经不动点场。如果能够把模型的神经不动点场研究清楚，就可能提高模型的推理速度。推理本质上是一个积分过程，而当前推理效率较低，一个重要原因就是推理的数学还没有被真正搞清楚。

从正问题到反问题

这里还有一个比传统微分方程更深的问题。微分几何和传统不动点理论主要研究的是正问题：给定方程、映射和结构，再去研究它的解。但学习本身是一个反问题。我们从数据和观测出发，反过来学习权重、映射和内部结构，而反问题在数学上天然可能是病态的。

神经网络真正特殊的地方，就在于它把正问题和反问题放到了同一个系统里：

• 训练是反问题：从数据中学习方程、关系和结构；

• 推理是正问题：在已经学习出来的结构上求解。

因此，深度流形未来真正希望研究的问题，是能不能把微分几何、不动点类理论和反问题数学进一步统一起来，研究AI解的存在性、唯一性和稳定性。

深度流形的数学方向

丘成桐因为卡拉比猜想，把微分几何推进到非线性；石根华老师因为岩石工程，把几何和数值计算推进到高阶非线性与不连续。今天，AI又提出了新的问题：如果方程、映射和几何本身都需要从数据中学习出来，数学应该怎样研究它们？ 深度流形希望进一步

• 把微分几何推进到可学习几何，

• 把不动点类理论推进到可学习的不动点场，

• 把数值计算推进到可学习的数值计算，

并尝试实现石老师所说的一个数学梦想：正问题与反问题的统一。

• 当数学之美在 AI 中变得有用

• AI 中被忽视的基础数学 (数学若其理未明，诚为可畏之事)

This article is now listed under Architecture in Deep Manifold, Two Years Later: 2024–2026.

数学的无用美和方程的解

有人把数学称为“无用的美”，这句话有它的道理。现实世界纷繁复杂、瑰丽多姿，既有结构，也有变化。江泽涵, 石根华老师的老师, 在上世纪五十年代末曾讲过一句很有意思的话：“通过微分方程的窗子，数学家看到现实世界的光。”这句话点出了一个很深的问题：如果我们希望用数学描述一个不断变化的真实世界，微分方程可能是最重要的语言之一。 相对于代数，几何提供了更丰富的结构。

• 把几何结构连接起来，可以从局部走向整体，于是有了广度；

• 把几何结构叠加起来，可以表达越来越复杂的关系，于是有了深度；

• 而几何中的曲率、变形与连接，又天然适合描述一个动态世界。

陈省身 所发展的现代微分几何，把几何、拓扑和微分方程更加紧密地联系起来；丘成桐进一步推进了非线性微分方程与几何结构之间的关系。到了石根华这里，问题又继续向高阶非线性、多层覆盖和不连续结构推进。

但是，再美的数学，如果只有描述而没有解，在现实世界中仍然是不完整的。知道一个系统应该由什么微分方程来描述，与真正把这个方程算出来，是两个不同的问题。很多数学之所以长期被称为“无用之美”，并不是因为它不能描述现实，而是因为我们没有能力把这些复杂方程真正解出来。

数值计算作为数学问题的求解方法

计算机的发明改变了这种局面。伯克利及其他地方的早期研究推动了有限元方法的发展，用离散化物理问题的数值近似来取代对解析解的依赖。后来，O. C. Zienkiewicz、Y. K. Cheung 等人进一步发展了有限元的数学基础，使其逐渐成为求解物理微分方程的一般数值框架。

随着超级计算机的发展，人们一度对数值计算充满期待。似乎只要网格足够细、计算机足够快，越来越复杂的现实问题最终都能够被算出来。甚至有人相信，离散化和计算能力的发展可能会使很多传统数学分析变得不再重要。

真正处理复杂问题的人很快发现，仅仅增加计算量并不够。高阶非线性、不连续、大变形和复杂边界常常带来一个很现实的结果：不是算不动，就是算不准。上世纪八十年代末，我在认识石根华老师后进入这一领域。他的方法并不只是进一步细化网格，而是重新思考复杂问题应该如何表示。数值流形方法采用多层覆盖，并将数学覆盖与物理覆盖分开，使局部数学描述与物理不连续能够分别处理，再通过计算组合起来。这为传统数值方法难以处理的问题提供了另一条路径。

陈省身很早就意识到了这种多层覆盖思想的意义。他看到的并不只是一个新的工程计算方法，而是一条可能把几何、微分方程与数值计算真正连接起来的道路。石老师后来讲过他与陈省身之间那段很有名的“给庞加莱磕头对话”。其中真正重要的意思是：即使数值流形最终成为一种强大的数值计算方法，也希望它在成功以后仍然忠诚于数学。

这段“超级计算机撞墙”的历史，其实给了我们一个很重要的教训。数值计算当然能够帮助我们求微分方程的解，但一个解究竟对不对、准不准、稳不稳定、收不收敛、算得快不快，最终仍然离不开数学。所以，在数值计算中，数学并不是“无用之美”。它不仅美，而且非常有用。

数学的有用美和 AI

AI 的出现，把这个故事又往前推进了一大步。AI 之所以如此强大，其中一个很重要的原因，是它在并不完全自觉的情况下重新组合了过去几十年数值计算中的一些重要思想：多层覆盖、数学结构与物理承载的分离、局部计算、迭代过程，以及从局部结构形成整体结构。 但 AI 又在两个方向上有了真正重要的突破。

• 正问题与反问题开始在同一个计算系统中统一。

• 神经网络能够学习任何类型的数据，以及数据之间的任何关系；其无属性和满秩特征极大增加了它描述现实世界的能力。

神经网络本质上是可学习的数值计算。

通过深度流形，我们可以看到数学在 AI 中呈现出一种很特别的状态。它并没有像传统物理模型那样，把每个变量和结构都事先规定得非常严格；但它也不是毫无结构的自由计算。很多时候，它恰恰处在一个微妙的位置：数学用得不多不少，既保留结构，又保留学习的自由。这可能就是 AI 时代数学真正的“有用之美”。

神经网络架构，就是列方程

我们小学做应用题的时候就已经知道，同一道题，方程列得好不好，会直接影响求解过程。一个好的方程也许一两步就可以得到答案，一个不好的方程可能需要绕很多步，而每增加一步，就增加一次误差和出错的机会。在 AI 中，神经网络架构实际上就在做这件事：它决定学习方程怎样被列出来。

Transformer 之所以强大，一个很重要的原因，就是它把这个方程列得相对好。它形成了一个宽广而可学习的计算结构，使数据中的复杂关系能够通过训练逐步建立起来。

Transformer 的方程列得相对好，并不意味着每一部分都已经是最自然、最简洁或最正确的数学表达。数值计算的历史已经告诉我们：一个方法能够收敛，并不自动意味着它背后的数学表达就是最好的。有限元和超级计算机发展的历史也曾提醒我们：当计算撞墙的时候，问题不一定只是计算机还不够快。有时候，真正的问题是方程没有列好。

过去二年我们比较成功地系统地把AI(神经网络和它训练）放在了数学和数值计算框架之下, 下一步是“落地”去验证.

• 深度流形的数学展望

This article is now listed under Architecture in Deep Manifold, Two Years Later: 2024–2026.

The Useless Beauty of Mathematics

Mathematics has sometimes been called “useless beauty,” and there is some truth in that description. The real world is extraordinarily rich and complex, filled with both structure and change. In the late 1950s, Jiang Zehan, the teacher of Gen-Hua Shi, once said something memorable: “Through the window of differential equations, mathematicians see the light of the real world.” The statement points to a deep idea: if we want mathematics to describe a real world that is constantly changing, differential equations may be one of its most important languages.

Compared with algebra, geometry provides a richer structural language.

• By connecting geometric structures, we can move from the local to the global and gain breadth.

• By stacking geometric structures, we can represent increasingly complex relationships and gain depth.

• Curvature, deformation, and connectivity naturally allow geometry to describe a dynamic world.

The modern differential geometry developed by Shiing-Shen Chern brought geometry, topology, and differential equations into much closer contact. Shing-Tung Yau pushed the relationship between nonlinear differential equations and geometric structures much further. With Gen-Hua Shi, the problem continued toward higher-order nonlinearity, multilayer covers, and discontinuous structures.

But even the most beautiful mathematics remains incomplete in the real world if it provides only a description and no solution. Knowing which differential equations describe a system and actually being able to solve those equations are two very different things. Much mathematics has long been regarded as “useless beauty” not because it fails to describe reality, but because we have lacked the ability to solve the complex equations that arise from those descriptions.

Numerical Computation as Mathematical Problem Solving

The invention of the computer changed this situation. Early work at Berkeley and elsewhere advanced the finite element method, replacing the need for analytical solutions with numerical approximations of discretized physical problems. Later, O. C. Zienkiewicz, Y. K. Cheung, and others helped develop its mathematical foundation, turning finite elements into a general framework for solving physical differential equations.

With the rise of supercomputers, expectations for numerical computation became extremely high. It seemed that if the mesh were fine enough and the computer fast enough, increasingly complex real-world problems could eventually be solved. Some even believed that advances in discretization and computational power might make much of traditional mathematical analysis less important.

Those working on genuinely difficult problems soon found that more computation alone was not enough. Higher-order nonlinearity, discontinuity, large deformation, and complex boundaries often led to a simple outcome: either the computation would not run, or the answer would not be accurate. I entered this field in the late 1980s after meeting Gen-Hua Shi. His approach was not merely to refine the mesh, but to rethink how complex problems should be represented. The Numerical Manifold Method used multilayer covers and separated mathematical covers from physical covers, allowing local mathematics and physical discontinuities to be treated independently and then assembled computationally. This offered a different path toward problems that conventional numerical methods struggled to solve.

Shiing-Shen Chern recognized the significance of this multilayer-cover idea very early. What he saw was not merely another engineering computation method, but a possible path for bringing geometry, differential equations, and numerical computation together in a deeper way. Shi later often recalled his famous “kowtow conversation” with Chern. The deeper meaning of that exchange was simple: even if the Numerical Manifold Method eventually became a powerful numerical method, it should remain faithful to mathematics after it succeeded.

The history of “Supercomputers Hitting the Wall” teaches us an important lesson. Numerical computation can certainly help us solve differential equations, but whether a solution is correct, accurate, stable, convergent, and efficient ultimately still depends on mathematics. In numerical computation, therefore, mathematics is not “useless beauty.” It is not only beautiful, but also profoundly useful.

The Useful Beauty of Mathematics and AI

AI pushes this story another major step forward. One important reason AI has become so powerful is that, without necessarily doing so consciously, it recombines several important ideas developed through decades of numerical computation: multilayer covers, separation between mathematical structure and physical realization, local computation, iterative processes, and the construction of global structure from local structures.

But AI introduces two genuinely important advances.

• forward and inverse problems begin to coexist within the same computational system.

• neural networks can learn from any type of data and learn arbitrary relationships among data. Their propertyless and full-rank nature greatly expands their ability to describe the real world.

A neural network is fundamentally learnable numerical computation.

Through Deep Manifold, we can see mathematics taking on a particularly interesting form in AI. Unlike traditional physical models, it does not rigidly prescribe the meaning of every variable and structure in advance. But neither is it structureless free computation. It occupies a subtle middle ground: mathematics is used neither too much nor too little. Enough structure is preserved to make learning coherent, while enough freedom remains for the system to learn.

This may be the true “useful beauty” of mathematics in the age of AI.

Architecture Is the Equation

We learn this lesson as early as elementary-school word problems. For the same problem, how well we formulate the equation directly affects how easily it can be solved. A good equation may require only one or two steps, while a poorly formulated equation may require many detours. Every additional step creates another opportunity for error.

In AI, neural network architecture is doing essentially the same thing: it determines how the learning equation is formulated. One important reason the Transformer is so powerful is that it formulates this equation relatively well. It creates a broad and learnable computational structure in which complex relationships in data can gradually be established through training.

The Transformer formulates the equation relatively well, but that does not mean every part of it is already the most natural, simplest, or mathematically correct formulation. The history of numerical computation teaches us that the fact that a method converges does not automatically mean that the mathematics behind it is the best possible formulation. The history of finite elements and supercomputing gives us another warning: when computation hits a wall, the problem is not necessarily that the computer is still not fast enough. Sometimes the real problem is that the equation was not formulated well.

Over the past two years, we have made good progress in systematically placing AI, neural networks and their training, within a mathematical and numerical computation framework. The next step is to put this framework into practice and validate it.

• The Mathematical Outlook of Deep Manifold

This article is now listed under Architecture in Deep Manifold, Two Years Later: 2024–2026.

Anthropic 关于 Claude 情绪概念的文章，是一个很好的例子：它一方面显示了 LLM 研究已经推进到什么程度，另一方面也说明公众讨论是如何很容易偏离方向的。文章认为，模型内部可以存在对情绪概念的表征，而且这些表征会在功能上影响行为；但同时，它也明确没有声称模型具有主观感受，或具有人类式的内在体验。这一区分很重要。然而，一旦“恐惧”“绝望”“共情”这样的词进入讨论，话题就很容易从“功能性的可学习性”滑向“拟人化解释”。真正的问题，并不是 LLM 是否“有情绪”，而是它到底是一种什么样的计算系统，以及这样的系统如何学会某些形式，而后被人类解释为情绪、意图，甚至确定性.

因为神经网络是无属性的，它并不保留人类自然设定的类别边界，比如行动者、力、对象或情绪。从这个意义上说，行动者、严肃性和情绪 都进入同一个可学习的数值框架。它们通过单标嵌入被离散化，而嵌入向量的每一个元素都可以理解为一个“特征位”，也就是一个特征单位。一旦以这种方式表征，任何东西都可以和任何别的东西建立连接。权重本身并不关心实体类型；它只作用于习得的数值关系。这也是为什么，LLM 更应被理解为一种无属性、可学习的数值计算系统，而不是一个语义性的存在。

从这个角度看，LLM 只有随机不动点, 换句话说，只有各种近似。我们通常所说的“确定性”答案，往往不过是某种稳定近似，而人类乐于把它解释成确定性。情绪因此也不是模型内部某种特殊的内在实体。它只是一个分布在大量特征单位上的习得形式，并在同一个无属性计算空间中，与行动者、行为、因果关系和结果建立连接。所以，当人们说模型具有恐惧、共情或绝望时，很多时候其实是把一种本体性的解释，投射到了一个习得的、可连接的近似之上。那种表象也许是连贯的、有说服力的、甚至在行为上具有意义，但这并不意味着它就是一种内在体验。

这也正是为什么 “下一词预测” 是一个极其高明的数学设计。预训练并不要求把人类逻辑显式地内建到系统之中。相反，它使模型能够发现大量捷径，而这些捷径在功能上很像我们所说的人类直觉。一个无属性、可学习的系统，会自然找到这些捷径，因为特征单位 可以跨越不同领域建立连接，而不受实体类型限制。强化学习往往所做的，不过是强迫模型把这些近似，以人类偏好的 “思维链” 形式展开出来。这当然会让大家更满意，但对 LLM 研究者来说，更深层的问题始终不是模型是否“有情绪”。更深层的问题是，LLM 的可学习性究竟如何产生这些强大、充满捷径、并且以随机不动点形式存在的行为。

真正的问题，不是情绪、意图或角色。真正的问题，是神经网络的可学习性究竟如何运作。从不动点的视角看，基础模型中到处都是捷径。预训练会产生大量捷径，很像人类直觉，并不需要显式的人类逻辑。RL 往往不过是强迫模型把这些捷径以 chain-of-thought 的形式展开出来，这样大家都很满意。

对于像 Anthropic 这样的公司，我原本期待其研究能展现出更高的科学严谨性、学科规范和写作水准，而不是使用这种容易误导读者的术语。作者将这些特征命名为“向量（vector）”, 即便他们辩解称这是为了“描述每个情绪概念的特征而提供的权宜之计（for convenience）” 在数学上也是不严谨的。在严格的数值计算领域，向量并非一个可以随意使用的代称，它是一个正式的数学实体，必须定义在特定的向量空间中，并拥有明确的基（basis）和坐标系才有意义。通过这种所谓的“权宜”命名，作者回避了建立不变量几何结构的必要性，反而暗示了一种模型高维激活空间可能并不具备的线性欧几里得架构。这些模式应当被正式定义为指标（indicator） 即反映特定状态存在的函数映射或信号, 这样才能准确描述其作为复杂概念标记的功能，而不必依赖于“存在稳定底层坐标空间”这一错误的数学假设。

• 关于人工智能对齐的数学思考

• 关于犬牙交错与病态智能的数学思考

• 神经学习的喜剧

This article is now listed under Mechanistic Interpretability & Alignment in Deep Manifold, Two Years Later: 2024–2026.

深度流形 认为，神经网络的表达能力植根于两个基本性质：无属性激活满秩关系变换。神经网络学习的并不是数据本身，而是数据之间的关系。这一观点与范畴论自然一致，因为在范畴论中，关系结构比单个对象的内在身份更为根本。然而，正如 Deep Manifold Part 2: Neural Network Mathematics 第 4.6 节 Data Complexity，以及 单标几何 03：数据复杂性 所讨论的那样，要求一个全局共享的满秩变换来表示异质数据的全部复杂性，既不合理，而且一般而言在数学上也并不恰当。混合专家模型架构正是基于充分的理由来回应这一问题：它放宽了“所有输入都必须经过同一个变换处理”的要求，而是将关系计算分布到多个专家之中。

一种审视并可能改进 MoE 的方式，因此应当从几何出发，更具体地说，从学习所形成的流形结构出发。核心问题不再只是要使用多少专家、如何路由, 单标，或如何平衡专家负载，而是这些由局部学习形成的流形集合，是否仍然能够作为一个连贯的神经网络，保持几何上的连续性、拓扑上的相容性，以及功能上的可重组性。

流形同调与 MoE 的不稳定性

当前的 MoE 架构因此极有可能产生一种犬牙交错的神经不动点场，因为稀疏路由将计算划分开来，专家在不均衡参与下发生专门化，局部几何以半独立的方式演化，而它们的输出随后还必须跨越不连续的路由边界重新组合。这些因素会在局部能力区域之间造成尖锐过渡。这样的犬牙交错性对于**智能体型工作负载（agentic workloads）**尤其成问题，因为这类任务需要跨越许多功能区域的长程、连通轨迹，从而使路径连续性更难维持，并在长时间执行过程中增加不稳定性与误差累积。

这就引入了一种结构性的不稳定来源。路由边界、不均等的专家参与，以及独立演化的专家几何，会逐渐削弱局部流形之间的相容性。从深度流形 的视角看，这种失效可以理解为 流形撕裂：在单个专家内部，局部有效的计算路径可能依然保持完整，但将这些路径连接到更大系统中所需的关系连续性，却开始发生断裂。

DeepSeek V4 展示了如何在这种撕裂扩展为更广泛的架构性失效之前，将其约束在局部层面。正如 关于 DeepSeek V4 的数学思考 所讨论的，SwiGLU Clamping 用于限制极端激活，以及可能引发撕裂的局部曲率尖峰；Anticipatory Routing 提高了专家分配与产生该单标表征的流形状态之间的一致性；而 mHC 则限制了残余不连续性通过残差流的传播。

Kimi K3 表明，随着 MoE 规模不断扩大，重组问题会变得越来越重要。从本质上看，MoE 是一种分布式计算：计算首先被拆分、路由，在局部执行，然后再被带回到共同的计算路径中。用 MapReduce 的语言来说，或者更一般地说，在 拆分–执行–合并 模式中，越来越困难的其实是 Reduce 这一步。挑战不仅在于如何高效地分配计算，更在于如何将彼此独立处理的局部结构重新组合起来，同时避免破坏关系信息、形成瓶颈、 使训练失稳，或者不得不引入越来越多的修正机制。K3 针对这一问题引入了若干相互协调的机制。从深度流形 的视角看，这些类似 Reduce 的机制，可以理解为在恢复分布式计算结果的同时，尽可能保留足够的区域流形同调，从而使更大的模型仍然保持整体连贯性。

再结合极度稀疏路由，这些机制可以被看作一种协调一致的几何防御体系：clamping 控制撕裂发生的前置条件，路由一致性控制撕裂事件本身，而残差约束则控制撕裂的进一步传播。它们本身并不能保证全局流形同调，但能够打断这样一种级联过程：局部几何不匹配原本可能沿着专家之间以及层与层之间不断扩散。

在架构与训练中保持几何弹性

我们提出了联邦流形（Federated Manifold）这一思想，用以应对学习中的数据复杂性；参见 Deep Manifold Part 2: Neural Network Mathematics 第 7.3 节 Mosaic of Small Elastic Models，以及第 7.6 节 Federated Learning。就 MoE 而言，这一思路在概念上更接近于 Sparse MoE 或 Ultra-Sparse MoE。其核心目标是保持模型弹性。从几何角度看，弹性使每一个局部流形能够在学习新关系时进行弯曲、伸展和重组，而不必引发突兀的曲率变化、撕裂局部邻域，或破坏与相邻流形之间的重叠。因此，足够的弹性有助于在允许专家专门化的同时，仍然保持平滑的图册过渡、几何连续性以及同调相容性。

然而，架构只是问题的一半。由于 MoE 在结构上更容易受到路由不连续、专家失衡以及局部几何独立演化的影响，其训练过程同样应当采用分阶段、渐进式的方式，参见 关于强化学习的数学思考。边界条件最好具有弱、对称、渐进且几何相容的性质，使专家专门化能够逐步形成，而不是过早地迫使局部几何发生突变。训练进程应逐步提高任务难度、路由专门化程度和专家差异化，同时持续保持不断演化的流形之间的连通性。从这一角度看，MoE 的稳定性是由架构与训练进程共同决定的。

• 关于神经网络架构的数学思考

• 关于全模态模架构的数学思考

• 关于巨型单体模型的数学思考

• 关于流形联邦的数学思考

• 关于开放式基础模型的数学思考

• 关于自我演化神经网络的数学思考

This article is now listed under Architecture and Mathematical Considerations Series in Deep Manifold, Two Years Later: 2024–2026.

Deep Manifold argues that the expressive power of neural networks is rooted in two fundamental properties: propertyless activations and full-rank relational transformations. What a neural network learns is not the data itself, but the relationships among data, a view that is naturally consistent with category theory, where relational structure is more fundamental than the intrinsic identity of individual objects. However, as discussed in Deep Manifold Part 2: Neural Network Mathematics, Section 4.6, Data Complexity, and Single Token Geometry 03: Data Complexity, it is neither reasonable nor, in general, mathematically appropriate to expect a single globally shared full-rank transformation to represent the full complexity of heterogeneous data. MoE architecture responds to this problem for good reason: it relaxes the requirement that every input be processed through the same transformation and instead distributes relational computation across multiple experts.

One way to examine, and potentially improve, MoE therefore begins with geometry, and more specifically with the manifold structure created by learning. The central question is no longer only how many experts to use, how to route tokens, or how to balance expert loads, but whether the resulting collection of locally learned manifolds remains geometrically continuous, topologically compatible, and functionally recombinable as a coherent neural network.

Manifold Homology and MoE Instability

Current MoE architectures are therefore very likely to produce a jagged neural fixed-point field because sparse routing partitions computation, experts specialize under uneven participation, local geometries evolve semi-independently, and their outputs must later be recombined across discontinuous routing boundaries. These effects can create sharp transitions between locally competent regions. Such jaggedness is especially problematic for agentic workloads, which require long, connected trajectories across many functional regions, making pathway continuity harder to maintain and increasing instability and accumulated error over extended execution.

This introduces a structural source of instability. Routing boundaries, unequal expert participation, and independently evolving expert geometries can gradually weaken compatibility between local manifolds. From the Deep Manifold perspective, this failure can be understood as manifold tearing: locally valid computational pathways may remain intact inside individual experts while the relational continuity required to connect those pathways across the larger model begins to break.

DeepSeek V4 illustrates how such tearing can be constrained at the local level before it propagates into a broader architectural failure. As discussed in Mathematical Considerations for DeepSeek V4, SwiGLU Clamping limits extreme activations and the associated local curvature spikes that can initiate a tear; Anticipatory Routing improves consistency between expert assignment and the manifold state that produced the token representation; and Manifold-Constrained Hyper-Connections restrict the propagation of remaining discontinuities through the residual stream

Kimi K3 shows why the recombination problem becomes increasingly important as MoE scales. At its core, MoE is a form of distributed computation: computation is split, routed, applied locally, and then brought back into a common computational pathway. In the language of MapReduce, and more generally the split–apply–combine pattern, the difficult step is increasingly the Reduce operation. The challenge is not merely to distribute computation efficiently, but to recombine independently processed local structures without destroying relational information, introducing bottlenecks, destabilizing training, or requiring progressively more corrective machinery. K3 introduces several coordinated mechanisms around this problem. From the Deep Manifold perspective, its Reduce-like mechanisms can be interpreted as attempts to restore distributed computation while preserving sufficient regional manifold homology for the larger model to remain coherent.

Together with extremely sparse routing, these mechanisms can be viewed as a coordinated geometric defense: clamping controls the precondition, routing consistency controls the tearing event, and residual constraints control its propagation. They do not establish global manifold homology by themselves, but they interrupt the cascade through which local geometric mismatch can spread across experts and layers.

Preserving Geometric Elasticity in Architecture and Training

We proposed the idea of a Federated Manifold to address data complexity in learning; see Deep Manifold Part 2: Neural Network Mathematics, Section 7.3, Mosaic of Small Elastic Models, and Section 7.6, Federated Learning. In MoE terms, this is conceptually closer to Sparse MoE or Ultra-Sparse MoE. The key objective is to preserve model elasticity. Geometrically, elasticity allows each local manifold to bend, stretch, and reorganize as new relationships are learned without forcing abrupt curvature changes, tearing local neighborhoods, or destroying overlap with neighboring manifolds. Sufficient elasticity therefore helps preserve smooth chart transitions, geometric continuity, and homological compatibility while still allowing experts to specialize.

Architecture, however, is only half of the story. Because MoE is inherently more vulnerable to routing discontinuity, expert imbalance, and independently evolving local geometries, its training process should also be staged and progressive, see Mathematical Considerations for Reinforcement Learning. Boundary conditions should preferably be weak, symmetric, progressive, and geometrically compatible, so that specialization emerges without forcing abrupt local deformation too early. Training progression should gradually increase task difficulty, routing specialization, and expert differentiation while continuously preserving connectivity across the evolving manifold. In this view, MoE stability is jointly determined by architecture and training progression.

• Mathematical Considerations for Neural Network Architecture

• Mathematical Considerations for Giant Monolithic Models

• Mathematical Considerations for Manifold Federation

• Mathematical Considerations for Open-Ended Foundation Models

• Mathematical Considerations for Self-Evolving Neural Networks

• Mathematical Considerations for Omni-Model Architectures

This article is now listed under Architecture and Mathematical Considerations Series in Deep Manifold, Two Years Later: 2024–2026.

除了我们对分片流形、不动点理论、反问题和数值计算的深入理解，以及将这些数学方法应用于现实世界问题的广泛经验之外，还有一个非常简单的原因，一直推动我们研究神经网络数学：每一个对数字的运算都涉及数学。

在训练和推理过程中，会发生数十亿次数值运算。数学就在那里，等待被发现，而我们通过 深度流形 去发现它。

AI 中被忽视的两个数学基础

反问题：在我们 2024 年的论文 Deep Manifold Part 1: Anatomy of Neural Network Manifold 一开始的 第 3.1 节 “Inverse and Forward Enablement” 中，我们就指出，神经网络同时包含正向与反向过程。维基百科对反问题 有较为详细的介绍，其中关于数学与计算的部分指出，“反问题通常是病态的”，这不同于数学中常见的 适定问题。1902 年，法国数学家 Jacques Hadamard 提出了适定问题的三个条件：解必须存在、唯一，并且连续依赖于数据。

认识到学习是一个反问题，会为 AI 的多个领域带来基础性的理解：

1. AI 安全与对齐： 这有助于解释为什么 AI 在本质上呈现犬牙交错性，以及为什么彼此接近的输入可能产生非常不同的行为。病态性并不只是实现层面的缺陷；它根植于学习的反问题本质。

2. 模型压缩： 一旦认识到训练后模型的有效自由度是未知的，量化和剪枝就更容易理解。许多不同的参数配置可能表示本质上相同的已学习函数，因此留下了很大的压缩空间。

3. 优化： 当学习是一个反问题时，学习本质上是反向重构，而不是朝着一个已知解进行优化。这也有助于解释，为什么在超过 20 万篇优化论文之后，神经网络训练仍然缺乏真正的优化基础，正如 神经网络优化的误区 中所讨论的那样。

注意力： 点积只衡量它实际计算的两个向量之间的相似性。在 Transformer 注意力中，Q 和 K 是由原始输入 X 经过不同的线性变换得到的。但线性变换并不一定保持内积或距离，因此 Q 和 K 之间测得的相似性，并不一定就是原来 X 中存在的相似性。也就是说，注意力是在把输入变换到不同表示之后再计算相似性，而不是必然保持 X 原有的几何相似性。

点积确实衡量相似性，但这种相似性只对实际参与计算的向量成立。当原始向量 X 经过线性变换得到 Q 和 K 时，点积 Q · K 衡量的是 Q 和 K 的相似性，而不是 X 的相似性。线性变换通常并不保持点积，因此即使 X 中原本的点积很小，Q · K 也可能很大；反过来，X 中原本的点积很大，Q · K 也可能变小。经过变换后得到的相似性得分，并不一定代表输入 X 原本的相似性得分。

一种常见的说法是，注意力是可学习的。但如果前向计算本身包含一个数学上的问题，为什么一开始要把这个问题引入进去？第二个问题是，既然如此，Transformer 为什么仍然能够工作得如此出色？原因很简单：神经网络是可学习的数值计算。详见下文“数值计算”部分。

深度流形

深度流形并不是一个新的数学分支，而是一个揭示神经网络数学结构的框架。一旦底层数学被理解，许多过去看起来并不清楚的问题，就可以得到更直接的解释。下面两个例子可以说明这一点。

1. 数值计算： 神经网络是可学习的数值计算，而数值方法即使采用并不完美的数学形式，也仍然可能收敛。即使偏微分方程本身是错误的，有限元法也可能得到一个收敛解；收敛并不意味着数学本身就是正确的。这也有助于解释，为什么许多不同的神经网络架构和注意力机制仍然能够工作。参见 关于神经网络架构的数学思考 以及 Deep Manifold Part 2: Neural Network Mathematics 的 第 3 节 Neural Network: Learnable Numerical Computation。

2. 隐空间与维度性： 隐空间和高维性被广泛讨论，但它们究竟存在于哪里，往往并没有被清楚说明。深度流形提出，对于一个给定输入，隐空间与一个不动点场相关，而维度性则存在于激活覆盖之间的关系之中。参见 关于隐藏空间与压缩的误区 和 维度所在之处的误区。

• 什么是深度流形？

• 神经不动点场

• 学习是一个反问题

• 深度流形模型扩展

• 数学家不是这样训练的

• 深度流形不是计划的

This article is now listed under Core in Deep Manifold, Two Years Later: 2024–202

Besides our deep knowledge of piecewise manifolds, fixed-point theory, inverse problems, and numerical computation, and our broad experience applying them to real-world problems, one simple reason has kept us working on neural network mathematics: Every Operation on Numbers Involves Mathematics.

During training and inference, billions of numerical operations take place. Mathematics is there, waiting to be discovered, and we did with Deep Manifold.

Two Mathematical Basics Overlooked in AI

Inverse Problems: At the very beginning of our 2024 paper, Deep Manifold Part 1: Anatomy of Neural Network Manifold, Section 3.1, “Inverse and Forward Enablement,” we point out that a neural network combines forward and inverse processes. Wikipedia provides a detailed discussion of inverse problems. Its section on mathematical and computational aspects explains that “inverse problems are typically ill-posed,” unlike the well-posed problems commonly encountered in mathematics. In 1902, the French mathematician Jacques Hadamard defined the conditions of a well-posed problem: a solution must exist, be unique, and depend continuously on the data.

Recognizing that learning is an inverse problem brings fundamental benefits to several areas of AI:

1. AI Safety and Alignment: It helps explain why AI is inherently jagged and why nearby inputs can produce very different behaviors. Ill-posedness is not merely an implementation defect; it is rooted in the inverse nature of learning.

2. Model Compression: Quantization and pruning become easier to understand once we recognize that the effective degrees of freedom of a trained model are unknown. Many different parameter configurations may represent essentially the same learned function, leaving substantial room for compression.

3. Optimization: When learning is an inverse problem, learning is inverse reconstruction, not optimization toward a known solution. This helps explain why, after more than 200,000 optimization papers, neural-network training still lacks a true optimization foundation, as discussed in The Myth of Neural Network Optimization.

Attention: A dot product measures similarity only between the vectors on which it is actually computed. In Transformer attention, Q and K are produced from the original input X through different linear transformations. But a linear transformation does not necessarily preserve inner products or distances, so the similarity measured between Q and K is not necessarily the similarity that existed in X. Attention therefore computes similarity after transforming the input into different representations, rather than necessarily preserving the original geometric similarity of X.

A dot product measures similarity, but that similarity only holds for the vectors it is actually computed on. When linear transformations are applied to the original vectors X to produce Q and K, the dot product Q · K is a similarity score for Q and K, not for X. Linear transformations generally do not preserve dot products, so Q · K can be large even when X’s own dot product was small, or small even when X’s was large. The similarity score computed after transformation does not necessarily represent the similarity score of the input X.

One argument is that attention is learnable. But if the forward pass contains a mathematical mistake, why introduce that mistake in the first place? A second question is why the Transformer still works so well despite it. The reason is simple: a neural network is learnable numerical computation. See the discussion below under “Numerical Computation.”

Deep Manifold

Deep Manifold is not a new branch of mathematics. It is a framework that reveals the mathematics of neural networks. Once the underlying mathematics is understood, many questions that previously seemed unclear can be explained more directly. The following two examples illustrate this.

1. Numerical Computation: A neural network is learnable numerical computation, and numerical methods can tolerate imperfect formulations while still converging. Even a wrong PDE can produce a converged FEM solution, convergence does not make the mathematics correct. This also helps explain why many different architectures and attention mechanisms can still work. See Mathematical Considerations for Neural Network Architecture and Deep Manifold Part 2: Neural Network Mathematics Section 3 Neural Network: Learnable Numerical Computation

2. Hidden Space and Dimensionality: Hidden space and high dimensionality are widely discussed, but where they actually reside is often left unclear. Deep Manifold proposes that, for a given input, hidden space is associated with a fixed-point field, while dimensionality resides in the relationships among activation covers. See The Myth of Hidden Space and Compression and The Myth of Where Dimensionality Resides.

• What is Deep Manifold ?

• Neural Fixed-Point Field

• Learning Is an Inverse Problem

• Deep Manifold Model Scaling

• Mathematicians Are Not Trained This Way

• Deep Manifold Was Not Planned

This article is now listed under Core in Deep Manifold, Two Years Later: 2024–202

传统 AI 观点通常把训练看作一个优化问题：定义目标函数，选择优化器，然后寻找最佳解。

学习是一个反问题。训练在经典意义上并不存在优化目标；损失函数只是一个边界泛函，为数值迭代提供残差信息。 几何决定可容许的解空间，而训练则在数据约束与边界条件下，对这一几何进行数值遍历并持续重塑。

AdamW 的梯度虽然来自整个网络，但其自适应预条件在很大程度上仍然是逐坐标处理；Muon 则更明确地把权重矩阵视为一个几何算子，利用矩阵层面的谱结构来决定残差如何被校正。

这是一个重要的概念转变，因为一个权重并不是孤立移动的：它作用于一个由连接、堆叠、分片流形组成的系统，而这个系统的几何又在训练过程中不断被重塑。 因此，Muon 支持了一个更广义的深度流形命题：神经网络的残差必须以几何方式进行校正。 但它并没有回答更困难的问题：什么才是正确的几何？一旦几何成为首要问题，仅仅看收敛速度就不再充分。残差校正最终必须由三个标准来判断：几何连续性、同调保真度与功能能力。

几何连续性

Muon 相比逐坐标校正前进了一步，因为它开始考虑一个权重矩阵内部的几何；但局部矩阵几何只是整个神经系统的一部分。一个权重算子在变换某个流形区域的同时，也与其前后以及周围的堆叠流形保持几何耦合；对这个算子的改变，会进一步改变相邻算子所面对的几何，也会影响穿过这些流形的内在路径。

从 Muon 到 Compositional Muon 的发展使这一点越来越清楚：Query 与 Key，或者 Output 与 Value，并不能始终被彼此独立地处理，因为它们的功能几何往往存在于组合关系之中。从深度流形的角度看，合适的残差校正因此不应只是让单个矩阵具有更好的条件数，而应当保持连接的权重流形与激活流形之间的几何连续性。Muon 是从标量坐标迈向算子几何的重要一步，但谱归一化能否在更大的堆叠流形系统中保持这种连续性，目前仍然是一个开放问题。

同调保真度

几何上的规整，并不意味着已经保持了学习到的结构。一个校正可以在局部上非常光滑，在谱上非常均衡，在数值上也十分稳定，却仍然破坏整个堆叠流形的全局组织。在 《单标几何 02：流形撕裂》 中，我们讨论了局部曲率、路由不一致以及不连续传输如何撕裂流形之间的关系，即使每个局部计算本身依然在数值上成立。《关于 DeepSeek V4 的数学思考》 从约束与几何兼容性的角度讨论了同一个问题，而 《关于 Kimi K3 的数学思考》 则揭示了一个互补的问题, 归并远比分散困难：把计算分配给不同专家相对容易，但要把这些局部变换重新组合成一个连贯的整体则困难得多。

从深度流形的角度看，这些都属于同调保真度问题：残差校正不仅要保持局部形状，还应尽可能保持堆叠流形之间那些本质性的重叠、连接、环、内在路径以及不动点组织。换句话说，一个校正完全可能局部平滑、谱上均衡、数值稳定，却在全局结构上是错误的。Muon 目前还没有证明它的谱校正能够保持这种更深层的结构；它可能找到了一个有用的局部几何，却尚未处理整个流形系统的同调关系。

功能能力

最终的检验，并不是残差下降得有多快，而是所形成的几何是否真正保留、甚至扩展了模型能够完成的功能。重尾关联记忆实验之所以重要，是因为它表明 Muon 的校正几何确实会产生真实的功能后果：当占主导地位的奇异方向不再压制较弱方向时，稀有知识可以得到显著更好的学习。但这并不意味着更快的数值收敛就等同于更好的学习。

Muon、等谱优化 以及其他受约束或归一化的训练方法，都可能通过限制、归一化或预条件权重空间中的可容许运动来减少训练步数；然而，残差在激活空间中的重新分布、流形的重新组织、内在路径的形成以及不动点的稳定化，可能运行在完全不同的时间尺度上。参数收敛时间并不等于流形形成时间。 一个系统完全可能在数值上已经变得安静，但其几何结构仍未充分形成。因此，目标不应该是让残差尽可能快地消失，而应该让残差以一种能够构造正确几何的方式逐步消失，从而保持几何连续性、同调保真度，并最终维持功能能力。

Muon 作为几何归一化

从这一视角来看，Muon 可以被理解为一种残差校正的几何归一化。它并不是把权重矩阵中的各个元素主要视为彼此独立的坐标，而是将整个矩阵作为一个几何算子来处理，对其谱结构进行归一化，并在不同奇异方向之间重新分配校正量。因此，Muon 的意义并不只是更快的收敛，而在于它认识到：残差校正本身具有几何结构。

然而，归一化并不等于发现了正确的几何。由于学习是一个反问题，解的几何并不是预先已知的，而是在数值迭代过程中由数据逐步构造出来的。因此，Muon 实际上是在这一构造过程中施加了一种特定的几何偏好。这种偏好可能改善数值条件并提高学习效率，但它是否能够在不断演化的堆叠流形中同时保持几何连续性、同调保真度与功能能力，仍然是一个更根本的问题。

谨慎的观点

最后还必须保持谨慎。我们目前仍然不知道 《神经不动点场》 在训练过程中究竟是如何形成的, 不同区域如何出现、相互作用并逐步稳定，也不知道某些看起来“不规则”的结构究竟是应当消除的缺陷，还是形成后续能力所必需的暂态阶段。从 《数据主义 》 的角度看，学习到的几何最终应当由对数据的忠实性来塑造，而不是由我们预先假定一种几何再强加给模型。Muon 的谱约束可能帮助不动点场形成得更加协调，但也可能抑制后来证明是必要的几何变化；目前我们并不知道答案。

关于Muon的误区，并不是 Muon 没有效果，而是把更快的收敛或更整洁的局部几何，等同于已经找到了神经网络学习的正确几何。 在我们能够更直接地观察和刻画神经不动点场的形成之前，Muon 更适合被看作一个重要的几何假设，而不是最终几何；评价它的标准，也不应只是收敛速度，而应是它能否保持几何连续性、同调保真度与功能能力。

• 神经网络优化的误区

• 关于隐藏空间与压缩的误区

• 维度所在之处的误区

This article is now listed under “Myth Series” in Deep Manifold, Two Years Later: 2024–2026.

The conventional AI view treats training as an optimization problem: define an objective, choose an optimizer, and search for the best solution.

Learning is an inverse problem. Training has no optimization objective in the classical sense; the loss acts only as a boundary functional that provides residual information for numerical iteration. Geometry determines the admissible solution space, while training numerically traverses and continuously reshapes that geometry under data and boundary constraints.

This distinction makes Muon interesting. AdamW’s gradient is produced by the whole network, but its adaptive preconditioning remains largely coordinate-wise; Muon instead treats a weight matrix explicitly as a geometric operator, using its matrix-level spectral structure to determine the residual correction.

This is an important conceptual shift because a weight does not move alone: it acts within a connected, stacked system of piecewise manifolds whose geometry is continually being reshaped during training. Muon therefore supports a broader proposition, neural-network residuals must be corrected geometrically, but it does not answer the harder question of what the correct geometry should be. Once geometry becomes primary, convergence speed alone is no longer sufficient. The correction must ultimately be judged by geometric continuity, homological fidelity, and functional capability.

Geometric Continuity

Muon improves on coordinate-wise correction by recognizing the internal geometry of a weight matrix, but local matrix geometry is only one part of the neural system. A weight operator transforms one manifold region while remaining geometrically coupled to the stacked manifolds around it; changing that operator changes the geometry seen by neighboring operators and the intrinsic pathways passing through them.

The progression from Muon toward compositional variants makes this increasingly explicit: Query and Key, or Output and Value, cannot always be treated independently because their functional geometry emerges through their composition. From the Deep Manifold view, the appropriate residual correction should therefore preserve geometric continuity across connected weight and activation manifolds, rather than merely produce a well-conditioned individual matrix. Muon represents an important move from scalar coordinates toward operator geometry, but whether spectral normalization preserves continuity across the larger stacked manifold system remains an open question.

Homological Fidelity

Geometric regularity alone does not guarantee that learned structure has been preserved. A correction can be locally smooth, spectrally balanced, and numerically stable while still disrupting the global organization of the stacked manifold. In Single Token Geometry 02: Manifold Tearing, we described how local curvature, routing inconsistency, and discontinuous transport can tear manifold relationships even when individual computations remain valid. Mathematical Considerations for DeepSeek V4 examines the same issue through containment and geometric compatibility, while Mathematical Considerations for Kimi K3 highlights the complementary problem of reduction: distributing computation is comparatively easy; recombining local transformations into a coherent whole is much harder.

These are manifestations of what we call homological fidelity: residual correction should preserve the essential overlaps, connections, cycles, intrinsic pathways, and fixed-point organization across stacked manifolds. Muon has not demonstrated that spectral correction preserves this deeper structure. It may have found a useful local geometry without yet addressing the homology of the manifold system as a whole.

Functional Capability

The final test is not whether the residual falls faster, but whether the resulting geometry retains or expands what the model can actually do. The heavy-tailed associative-memory results are important because they show that Muon’s correction geometry can have a real functional consequence: rare knowledge can be learned much better when dominant singular directions no longer overwhelm weaker ones. Yet this does not mean that faster numerical convergence is necessarily better learning.

Muon, Isospectral Optimization, and related approaches can restrict, normalize, or precondition the admissible movement in weight space and thereby reduce the number of training steps, while residual redistribution, manifold reorganization, intrinsic-path formation, and fixed-point stabilization may operate on different timescales. Parameter convergence time is not necessarily manifold formation time. A system can become numerically quiet before its geometric structure has fully developed. The objective should therefore not be to eliminate residuals as quickly as possible, but to let residual correction build a geometry that maintains geometric continuity, preserves homological fidelity, and ultimately sustains functional capability.

Muon as Geometric Normalization

Seen from this perspective, Muon can be understood as a form of geometric normalization of residual correction. Rather than treating the entries of a weight matrix as largely independent coordinates, Muon acts on the matrix as a geometric operator, normalizing its spectral structure and redistributing correction across singular directions. Its significance is therefore not simply faster convergence, but the recognition that residual correction itself has geometry.

Yet normalization is not the same as discovering the correct geometry. Because learning is an inverse problem, the geometry of the solution is not known beforehand; it is progressively constructed from data through numerical iteration. Muon therefore imposes a particular geometric preference on how this construction proceeds. That preference may improve conditioning and learning efficiency, but whether it also preserves geometric continuity, homological fidelity, and functional capability across the evolving stacked manifold remains the more fundamental question.

A Cautious View

A final caution is necessary. We still do not know how the Neural Fixed-Point Field actually develops during training, how different regions form, interact, stabilize, or whether apparent irregularities are temporary stages that are necessary for later capability. From the perspective of Dataualism, the learned geometry should ultimately be shaped by fidelity to data rather than by a geometry imposed in advance. Muon’s spectral constraint may help the field form more coherently, but it may also suppress geometric variation that later proves necessary; at present, we do not know.

The myth of Muon is not that Muon does not work, but that faster convergence or cleaner local geometry tells us that we have found the correct geometry of neural learning. Until the formation of the neural fixed-point field can be observed and characterized more directly, Muon should be treated as an important geometric hypothesis, not the final geometry and judged by geometric continuity, homological fidelity, and functional capability rather than convergence speed alone.

• The Myth of Neural Network Optimization

• The Myth of Hidden Space and Compression

• The Myth of Where Dimensionality Resides

This article is now listed under “Myth Series” in Deep Manifold, Two Years Later: 2024–2026.

GRPO 作为更好的边界条件

GRPO 最初由 DeepSeek 提出，主要是为了让强化学习训练更加实际：通过去掉 PPO 中独立的 价值评价模型 来降低显存开销。然而，它或许无意中触及了一个更重要的问题：更好的数学结构。目前关于 GRPO 的讨论，大多集中于算法、推理性能和计算效率，却很少从数学结构本身解释，为什么 GRPO 的意义可能远远不止节省显存。

从 深度流形 的视角来看，GRPO 实际上为学习提供了一种更好的边界条件。正如 《关于学习空间的数学思考》 以及 《深度流形第二部分：神经网络数学》 第 2.6 节 《神经网络边界条件》 所讨论的，神经网络学习不仅由数据和架构决定，也受到施加在这一逆学习问题上的边界条件所塑造。GRPO 不预先规定一条目标轨迹，也不持续估计中间状态的价值，而是允许多个轨迹在同一个问题下自行展开，再通过相对结果加以区分。因此，它形成了一种 弱、离散、对称且渐进的边界条件：弱，是因为它不规定精确路径；离散，是因为反馈来自采样轨迹；对称，是因为事先没有任何一条轨迹被赋予优先地位；渐进，则是因为反复的相对比较会逐步重塑神经场。

GRPO 作为堆叠分片流形设计

GRPO 还可以被理解为一种堆叠分片流形设计。正如 《单标几何 06：堆叠分片流形》 所讨论的，复杂行为并不需要由一个全局光滑流形来表示。不同的局部流形可以捕捉同一个问题的不同侧面，再通过组合形成更加丰富的表示。GRPO 天然形成了这样的结构：每一条采样轨迹都可以看作同一个任务的一种不同流形实现，捕捉问题的一条不同可能路径或一个不同侧面。多条轨迹组合在一起，使 GRPO 能够更加丰富地表示一个本来就具有非线性、多样性、甚至不连续性的真实世界，而不必把所有可能性压缩成唯一条所谓“正确”的路径。

因此，GRPO 中实际上存在两个层次的几何组合。横向看，不同 rollout 提供了同一个学习问题的不同流形视角，并共同丰富对问题的表示；纵向看，一条轨迹中的连续推理步骤构成一系列局部流形覆盖，而这些覆盖之间必须具有足够的重叠和平滑连接，才能保持路径的连续性。相对比较作用于这些已经组装起来的路径，而不是某一条预先指定的标准答案。从这个意义上说，GRPO 并不仅仅是在“采样多个答案”，而是在同一个边界条件下，让多个流形结构自行形成，再通过最终结果对它们加以区分。

这种堆叠结构很有力量，但它也暴露出一个潜在问题。一条轨迹可能在多个局部覆盖之间一直保持良好的几何一致性，却在某一次转换时突然发生一个急转弯，随后又沿着错误路径继续保持内部一致的推理。GRPO 可以判断最终轨迹表现不好，却未必能够识别几何不连续究竟发生在哪里。反过来，一个中间步骤可能发生了不良转向，后面却偶然恢复，并最终得到正确结果。因此，这个问题不仅仅是传统意义上的信用分配：边界条件作用在整条轨迹上，而真正的缺陷可能只局限于两个局部流形覆盖之间的一次转换。

迈向几何感知的 GRPO

这自然引出了超越轨迹级 GRPO 的一个方向。按照 《深度流形模型扩展》 中提出的原则，强化学习不应该只区分哪一条完整轨迹表现更好，还应该在轨迹展开过程中保持几何连续性、同调保真性和功能能力。如果把每一个推理步骤视为一个局部流形覆盖，那么相邻覆盖之间的重叠与转换本身也应该成为学习问题的一部分。更好的边界条件应该尽可能保持平滑、可行的连接，同时识别轨迹在哪里发生急转弯，进入另一条不同或不稳定的路径。目标并不是规定每一步应该怎样推理，而是在保留 GRPO 弱、离散、对称、渐进这些优点的同时，让边界信息更加具有几何依据，也更加局部化。

这样的设计有可能同时改善 GRPO 的多个方面。如果学习信号能够更加准确地定位，那么当成功或失败只由轨迹中的一小部分造成时，就没有必要反复奖励或压制整条轨迹。原则上，这不仅可能提高性能，还可能减少训练所需的样本数量、生成单标数量以及计算量，因为学习信号能够更精准地作用于真正需要改变的几何结构。这也符合深度流形更广义的模型扩展原则：效率不应该只是来自“少做一些计算”，而应该来自更好地组织学习，在保持几何连续性、同调保真性和功能能力的同时，减少不必要的几何变形与重复学习。

• 关于训练进程的数学思考

• 关于持续学习的数学思考

• 关于强化学习的数学思考

This article is now listed under Training and Mathematical Considerations Series in Deep Manifold, Two Years Later: 2024–2026.

GRPO as Better Boundary Condition

GRPO was proposed by DeepSeek largely to make reinforcement-learning training more practical by removing PPO’s separate critic model and reducing memory cost. Yet it may have stumbled onto something more important: better mathematics. Most discussions of GRPO focus on algorithms, reasoning performance, and computational efficiency, while much less attention has been given to the mathematical structure that may explain why GRPO works well beyond memory saving.

From the Deep Manifold perspective, GRPO introduces a better boundary condition for learning. As discussed in Mathematical Considerations for Learning Space and Section 2.6, “Neural Network Boundary Condition,” of Deep Manifold Part 2: Neural Network Mathematics, learning is shaped not only by data and architecture but also by the boundary conditions imposed on the inverse learning problem.

Rather than prescribing a target trajectory or continuously estimating the value of intermediate states, GRPO allows multiple trajectories to develop under the same problem and differentiates them through relative outcomes. Its boundary is therefore weak, discrete, symmetric, and progressive: weak because it does not dictate an exact path, discrete because feedback comes from sampled trajectories, symmetric because no trajectory is privileged in advance, and progressive because repeated comparisons gradually reshape the neural field.

GRPO as a Stacked Piecewise Manifold Design

GRPO can also be understood as a stacked piecewise manifold design. As discussed in Single Token Geometry 06: Stacked Piecewise Manifold, complex behavior does not need to be represented by one global smooth manifold. Different local manifolds can capture different aspects of the same problem, and their assembly can produce a richer representation. GRPO naturally creates such a structure: each sampled trajectory can be viewed as a different manifold realization of the same task, capturing a different possible pathway or aspect of the problem. Together, these trajectories give GRPO a richer way to represent a real world that is nonlinear, diverse, and often discontinuous, rather than compressing it into one supposedly correct path

.This produces two levels of geometric composition. Across trajectories, different rollouts provide different manifold views of the same learning problem and collectively enrich its representation; along each trajectory, successive reasoning steps form local manifold covers that must overlap and connect smoothly enough to preserve continuity. Relative comparison then acts on these assembled pathways rather than on one prescribed solution. In this sense, GRPO is not simply sampling multiple answers—it allows multiple manifold constructions to form under the same boundary condition and then differentiates them through their outcomes.

This stacked structure is powerful, but it also exposes a potential weakness. A trajectory may remain geometrically consistent through several local covers and then make a sharp turn at one transition, after which the reasoning can continue coherently along the wrong pathway. GRPO can determine that the completed trajectory performed poorly, but it does not necessarily identify where that geometric discontinuity occurred. Conversely, a poor intermediate turn may later recover and still receive a positive final outcome. The issue is therefore not only conventional credit assignment: the boundary condition is applied at the trajectory level, while the actual defect may be localized in the transition between two manifold covers.

Toward Geometrically Informed GRPO

This suggests a natural direction beyond trajectory-level GRPO. Following the principles of Deep Manifold Model Scaling, reinforcement learning should not only distinguish which completed trajectories perform better, but also preserve geometric continuity, homological fidelity, and functional capability as those trajectories develop. If each reasoning step is treated as a local manifold cover, then the overlap and transition between successive covers become part of the learning problem itself. A better boundary condition should preserve smooth and viable transitions while identifying where a trajectory makes a sharp turn into a different or unstable pathway. The goal is not to prescribe every reasoning step, but to retain GRPO’s weak, discrete, symmetric, and progressive character while making its boundary information more geometrically informed and locally meaningful.

Such a formulation could potentially improve GRPO along several dimensions at once. Better localization of the learning signal should reduce the need to repeatedly reward or suppress an entire trajectory when only a small part of it is responsible for success or failure. In principle, this could improve performance while reducing samples, generated tokens, and training compute, because learning would be directed more precisely toward the geometry that actually needs to change. This follows the broader Deep Manifold scaling principle: efficiency should not come merely from doing less computation, but from organizing learning so that geometric continuity, homological fidelity, and functional capability are preserved while unnecessary geometric deformation and repeated relearning are reduced.

• Mathematical Considerations for Training Progression

• Mathematical Considerations for Continual Learning

• Mathematical Considerations for Reinforcement Learning

This article is now listed under Training and Mathematical Considerations Series in Deep Manifold, Two Years Later: 2024–2026.

模型原生智能（Model-Native Intelligence），是指人类只向模型提供一个任务，而模型自己构建完成这个任务所需要的一切，包括规划、记忆、工具、验证、重试、智能体以及执行框架。定义的关键非常简单：执行框架由模型自己构建，而不是由人类预先设计后交给模型。

近期 IMO 级别的问题求解以及一些新的数学证明，展示了非常强的 AI 能力，但它们并不必然代表模型原生智能。即使执行过程中没有人类介入，模型仍然可能运行在一个由人类预先设计好的系统之中，例如推理流程、智能体或者执行框架。此时，模型仍然是在沿着人类事先构建的计算路径解决问题，而不是自己形成这条路径。

因此，定义可以非常直接：人类只提供任务；模型自己构建智能体和执行框架，并在没有进一步人类干预的情况下完成任务。 模型原生并不意味着只能依赖模型权重，也不意味着不能使用智能体、工具或执行框架。区别只在于：这些结构是否由模型自己产生。

在 《大语言模型能否带着信念与勇气完成跳跃?》中，我们讨论过，信念与勇气可能是科学突破所需要的一类能力。我们当然可以把类似信念与勇气的行为编码进执行框架，但那仍然是人类干预。要测试模型自身是否真正具备这种能力，就要看这些行为是否源自模型本身。信念与勇气只是一个例子。也正因为如此，我们需要开始认真讨论模型原生智能：真正要测量的，不只是模型在人类搭建的系统中能够做到什么，而是模型自身能够产生什么样的智能能力。

Model-native intelligence means that a human gives the model a task, and the model itself constructs whatever is needed to complete it: planning, memory, tools, verification, retries, agents, and harness. The defining point is simple: the model builds its own harness rather than receiving a harness designed by humans.

Recent IMO-level results and mathematical proofs show extraordinary AI capability, but they do not necessarily demonstrate model-native intelligence. Even when there is no human intervention during execution, the model may still operate under a human-predefined system, such as a reasoning or agent harness. In that case, the model is functioning as a smart tool within a computational pathway humans designed in advance.

The definition is therefore straightforward: human provides the task; the model builds its own agent and harness and finishes the task without further human intervention. Model-native does not mean weights alone. Agents and harnesses can still exist—the difference is that they are created by the model itself.

In Can LLMs Jump with Conviction and Courage?, conviction and courage were discussed as one example of capabilities that may matter for genuine scientific discovery. We can encode behaviors resembling conviction and courage into a harness, but that only shows what a human-designed system can produce. To test the model’s own intelligence, we need to ask whether such capabilities originate from the model itself. Conviction and courage are only one example; this is why it is important to begin discussing model-native intelligence as a distinct measure of AI capability.

This article is now listed under Architecture in Deep Manifold, Two Years Later: 2024–2026.

归纳与演绎：大语言模型已经具备的能力

Tom Zahavy 与 Yong Zheng-Xin 有一个重要的共同出发点：大语言模型已经具备归纳与演绎能力。Zahavy 将现代生成式 AI 描述为已经非常擅长通过统计学习进行归纳，并且正越来越擅长通过形式推理和定理证明进行演绎。Yong 也明确接受这一前提。两人的分歧出现在更后面：溯因，以及科学跳跃究竟需要什么。

从深度流形的角度看，归纳和演绎并不应该令人意外，因为神经网络从根本上就是为学习而设计的。正如 《学习是一个反问题》 中所讨论的，学习是一个反问题：神经网络从观测与边界条件出发，通过不断迭代，重建未知的关系结构。它非同寻常的学习能力，尤其根植于无属性激活与足够满秩的变换。无属性的数值本身并不预先属于语言、图像、方程、物理测量或任何既定概念；而足够满秩的变换，则保留了足够多彼此独立的方向，使各种关系能够形成、分离、组合与传播。

从范畴论来看，这一点更加清楚。正如 《关于范畴论视角下迁移学习的数学思考》 和 《关于专家数据迁移的数学思考》 所讨论的，真正重要的并不只是对象本身，而是对象之间的关系与变换。不同领域中的对象可能完全不同，但它们之间的关系结构与复合结构仍然可以被映射、比较，并在一定条件下得到保持。神经网络尤其适合这种观点，因为其内部无属性的数值并不携带固定的领域身份。它可以首先学习类似态射的关系，而意义则从这些关系如何复合之中逐渐形成。

从这个角度看，归纳是在构建关系结构，而演绎是在已有结构中进行复合与沿关系路径推进。归纳学习哪些对象、观测或变换彼此相关；演绎则沿着这些已经学得的关系，推出并未被直接给出的结果。范畴论为这两种过程提供了一种自然的数学语言：对象、态射、复合，以及保持结构的映射。

这些能力的涌现，也与 《关于复杂性与涌现的数学思考》中的讨论一致。真实世界的数据包含高阶非线性、不连续性、递归、上下文依赖、复合关系，以及极其庞大的关系复杂性。无属性消除了表征上的先验限制，满秩结构保留了关系自由度，而堆叠流形则把这种复杂性分散到可学习的局部结构之中。当复杂性、容量与学习达到足够程度时，那些从未被逐项编程进去的能力就可能涌现出来。因此，归纳与演绎并不是大语言模型后来神秘获得的附加能力，而是一个足够强大的关系学习机器所自然产生的结果。

这也构成了 Zahavy 与 Yong 的共同基础：大语言模型能够学习关系、连接关系、复合关系，并沿着这些关系进行推理。 真正更困难的问题，是它能否超越已经建立的关系，提出一个足以改变整体结构的新连接。也正是在这里，两人的讨论从归纳与演绎进入了溯因, 以及“跳跃”本身。

溯因：从完整关系空间到有序结构

Tom Zahavy 在 LLMs Can’t Jump 与 Yong Zheng-Xin 在 LLM Can “Jump” 中，都把溯因置于科学发现讨论的核心。从深度流形的角度看，溯因可以被理解为：在正确的关系及其顺序尚未确定之前，对一个完整关系空间进行搜索。概念、观测、数学结构与物理现象之间，可能存在许多不同方向的潜在关系；科学问题在于发现其中哪些连接是真实的。

但提出一种关系，只是开始。扰动使结构产生秩序。 正如 《关于因果推断的数学思考》中所讨论的，干预会改变边界条件，暴露出不同的可能路径，并揭示当系统受到扰动时，哪些关系仍然能够保持稳定。因此，因果结构可以被理解为一种偏序结构：它是从那些在扰动下仍然稳定存在的关系中重建出来的。

爱因斯坦关于引力与加速度等价的思想很好地说明了这一过程。溯因式跳跃打开了一个此前并未被认识到的关系:引力 ↔ 加速度。随后，思想实验不断扰动物理条件：自由落体、加速运动、时钟、光，以及不同的参考系。在这些扰动下仍然保持成立的关系，逐渐形成了结构与秩序。

从这个意义上说，溯因打开完整关系空间；扰动对它进行检验，并使秩序涌现出来。

信念与勇气

研究是残酷而冷峻的。在提出问题、检验和探索的过程中，并不存在所谓“错误的研究”；只有能够经受现实检验的结果，以及无法经受现实检验的结果。在前沿研究中，失败占据绝大多数。我有时把它看作一场 95% 都会输的游戏：绝大多数想法、假设、实验和数学路径，最终都走不通。因此，研究者其实是一种高风险职业, 这种风险未必来自身体，而更多来自智识、职业生涯和个人层面。正如我在 《辛克維奇时刻，历史正在重演？》中所讨论的，即使是一个领域中的巨匠，也可能花费数十年建立一个极其优美的数学方向，最后却发现，当问题进入更高一级的复杂性时，其底层结构无法继续成立。研究没有义务保护声望、资历、投入或信念。它只回答一个问题：什么能够经受住现实。

因此，伟大的研究还需要一些归纳、演绎，甚至溯因都无法完全描述的东西：信念与勇气。一个新的关系可以通过溯因被提出，它的后果可以通过演绎被推出，证据可以通过归纳不断积累。但在这些过程能够提供足够的确定性之前，仍然需要有人相信：一条看起来可能性很低的路径，值得继续走下去。而这往往意味着多年孤独、怀疑、拒绝、嘲讽，甚至失去同事、机构支持或职业地位。

Alfred Wegener 是最清楚的例子之一。他提出的大陆漂移假说，把地质、化石、地理和古气候证据连接起来，形成了一幅完全不同的地球图景，但当时大部分地质学界并不接受这一理论。Wegener 无法给出一个令人信服的大陆运动机制，但他仍然坚持，因为这些观测之间的关系过于一致，无法轻易忽略。他去世几十年之后，板块构造理论才提供了缺失的机制，使大陆漂移最终成为现代地球科学的基础。他的故事说明，信念并不等于确定性：一个人可以坚定地相信一种结构，同时承认其中一部分解释仍然缺失。

Barry Marshall 和 Robin Warren 提供了另一个完全不同的例子。他们提出，Helicobacter pylori（幽门螺杆菌）是胃炎和消化性溃疡的重要原因，这直接挑战了当时占主流的观点: 溃疡主要与压力、生活方式和胃酸过多有关。在经历批评以及论文发表困难之后，Marshall 亲自吞服了这种细菌，随后患上胃炎，以极具冲击力的方式证明这种细菌确实可以感染健康人的胃。他们的工作最终颠覆了医学界长期以来的认知，并获得了 2005 年诺贝尔生理学或医学奖。

这些例子揭示了“大语言模型跳跃”背后一个更深的问题。即使一个智能系统发现了新的关系连接，问题也并没有结束。科学发现发生在不确定性之中，而新的路径通常拥有更少的证据、更少的社会支持，并且往往也没有既有理论那么方便直接的解释能力。一个模型可以生成一万个假设，并对它们进行排序；它可以检验这些假设，淘汰其中绝大多数，然后再生成另外一万个。但这仍然不等同于信念。

信念，是当证据仍然不完整时，依然沿着一条微弱却在结构上极具说服力的路径继续前进。勇气，则是接受这样一种可能：这条路径也许会摧毁自己当前所依赖的假设、声望，甚至毕生建立起来的整个研究体系。

这提出了一个比“大语言模型能否进行归纳、演绎或溯因”更困难的问题。也许大语言模型能够跳跃。也许相互连接的知识、扰动、世界模型，以及足够丰富的关系学习，都能够让这种跳跃越来越可能发生。但科学史告诉我们，最伟大的跳跃，往往发生在既有世界赋予这条新路径的概率仍然非常低的时候。

大语言模型能否识别这样的路径，并且坚持走下去, 不是因为它立即获得了更高的奖励，不是因为共识支持它，也不是因为某个基准告诉它“你正在接近正确答案”，而仅仅是因为其底层结构给了它足够的理由继续前进？

也许正是在这里，科学智能最终必须面对信念与勇气。

Rudyard Kipling 在 If— 中，以极其精炼的方式表达了这种平衡：

“当所有人都怀疑你时，你仍能相信自己，
但同时，也能理解他们为何怀疑。”

目前，终端用户首先通过模型的执行框架层访问模型，而不是直接访问模型本身。我们当然可以在执行框架中实现“信念与勇气”，强制模型完成“跳跃”，但这并不是模型原生智能。参见《 模型原生智能 》

• 单标几何 03：数据复杂性

• 关于学习复杂性的数学思考

• 关于犬牙交错与病态智能的数学思考

• 单标几何 07:注意力

This article is now listed under Scientific and Engineering Intelligence and Mathematical Considerations Series in Deep Manifold, Two Years Later: 2024–2026.

Induction and Deduction: What LLMs Already Have

Tom Zahavy and Yong Zheng-Xin begin from an important point of agreement: LLMs are already capable of induction and deduction. Zahavy describes modern generative AI as highly capable of induction through statistical learning and increasingly capable of deduction through formal reasoning and theorem proving. Yong explicitly accepts this premise. Their disagreement begins later, with abduction and the nature of the scientific jump.

From the Deep Manifold perspective, induction and deduction should not be surprising capabilities because a neural network is fundamentally designed to learn. As discussed in Learning Is an Inverse Problem, learning is an inverse problem: from observations and boundary conditions, the network iteratively reconstructs an unknown relational structure. Its extraordinary learning power is rooted particularly in propertyless activations and sufficiently full-rank transformations. Propertyless numerical quantities are not intrinsically tied to language, images, equations, physical measurements, or predefined concepts, while sufficiently full-rank transformations preserve many independent directions in which relationships can form.

This becomes even clearer through category theory. As discussed in Mathematical Considerations for Transfer Learning through Category Theory and Mathematical Considerations for Expert Data Transfer, what matters is not simply the objects themselves but the relationships and transformations among them. Objects may differ radically across domains, while relational and compositional structures can still be mapped, compared, and sometimes preserved. Neural networks are unusually compatible with this view because their propertyless internal quantities do not carry fixed domain identities. They can learn morphism-like relationships first and allow meaning to emerge from how those relationships compose.

Seen this way, induction constructs relational structure, while deduction composes and traverses the structure already available. Induction learns that certain objects, observations, or transformations belong together; deduction follows those learned relationships to consequences that were not explicitly given. Category theory provides a natural mathematical language for both: objects, morphisms, composition, and structure-preserving mappings.

The emergence of these capabilities is also consistent with Mathematical Considerations for Complexity and Emergence. Real-world data contain high-order nonlinearity, discontinuity, recursion, context dependence, composition, and enormous relational complexity. Propertylessness removes representational barriers, full-rank structure preserves relational freedom, and stacked manifolds distribute this complexity across learnable local structures. Under sufficient complexity, capacity, and learning, capabilities that were never individually programmed can emerge. Induction and deduction are therefore not mysterious additions to LLMs; they are reasonable consequences of a sufficiently powerful relational learning machine.

This supports the common ground between Zahavy and Yong. LLMs can learn relationships, connect them, compose them, and reason through them. The harder question is whether they can go beyond the relationships already established and introduce a connection that changes the structure itself. That is where their debate moves from induction and deduction to abduction and to the jump.

Abduction: From Full Relational Space to Order

Both Tom Zahavy in LLMs Can’t Jump and Yong Zheng-Xin in LLM Can “Jump” place abduction at the center of scientific discovery. From the Deep Manifold perspective, abduction can be viewed as searching a full relational space before the correct relationships and their order are known. Concepts, observations, mathematical structures, and physical phenomena may potentially relate in many directions; the scientific problem is to discover which connections are real.

But a proposed relationship is only the beginning. Perturbation creates order. As discussed in Mathematical Considerations for Causal Inference, interventions change boundary conditions, expose alternative pathways, and reveal which relationships remain stable as the system is perturbed. Causal structure can therefore emerge as a partial order reconstructed from relations that survive these changes.

Einstein’s equivalence of gravity and acceleration illustrates this process. The abductive jump opened a previously unrecognized relation: gravity ↔ acceleration. Thought experiments then perturbed the physical setting: free fall, acceleration, clocks, light, and different frames of reference. What survived those perturbations gradually acquired structure and order. In this sense, abduction opens the full relational space; perturbation tests it and makes order emerge.

Conviction and Courage

Research is brutal and cold-hearted. There is no such thing as “wrong” research in the act of asking, testing, and searching; there are outcomes that survive reality and outcomes that do not. In frontier research, failure dominates. I sometimes think of it as a 95% losing game: most ideas, hypotheses, experiments, and mathematical paths eventually lead nowhere. Being a researcher is therefore a high-hazard occupation, not necessarily physically, but intellectually, professionally, and personally. As I discussed in Zienkiewicz Moment, History Repeating Itself?, even a giant of a field can spend decades building a beautiful mathematical direction only to discover that the underlying structure does not survive the next level of complexity. Research has no obligation to protect reputation, seniority, effort, or belief. It only answers to what survives.

Great research therefore comes with something that induction, deduction, and even abduction do not fully describe: conviction and courage. A new relationship can be proposed abductively. Its consequences can be deduced. Evidence can be accumulated inductively. But before any of those processes provide sufficient reassurance, someone still has to believe that an improbable path is worth following. That often means years of loneliness, doubt, rejection, ridicule, and sometimes the loss of colleagues, institutional support, or professional standing.

Alfred Wegener is one of the clearest examples. His continental-drift hypothesis connected geological, fossil, geographic, and paleoclimatic evidence into a radically different picture of Earth, yet much of the geological establishment rejected it. Wegener lacked a convincing mechanism for continental motion, but he continued because the relationships across those observations were too coherent to dismiss. He died decades before plate tectonics supplied the mechanism that transformed continental drift into modern Earth science. His story shows that conviction does not mean certainty: one can believe strongly in a structure while recognizing that part of the explanation is still missing.

Barry Marshall and Robin Warren provide a different example. Their proposal that Helicobacter pylori caused gastritis and peptic ulcers contradicted the prevailing view that ulcers were primarily associated with stress, lifestyle, and excess acid. After facing criticism and difficulty publishing their work, Marshall experimented on himself by ingesting the bacterium, developing gastritis and providing dramatic evidence that it could infect a healthy stomach. Their work overturned medical doctrine and earned them the 2005 Nobel Prize in Physiology or Medicine.

These examples expose something deeper about the “LLM jump.” The problem may not end when an intelligence discovers a new relational connection. Scientific discovery happens under uncertainty, where the new path usually has less evidence, less social support, and often less immediate explanatory convenience than the established one. A model can generate ten thousand hypotheses and rank them. It can test them, reject most of them, and generate ten thousand more. But that is not yet the same as conviction.

Conviction is continuing along a weak but structurally compelling path when the evidence is incomplete. Courage is accepting the possibility that the path may destroy the very assumptions, reputation, or body of work on which one currently stands.

This creates a harder question than whether LLMs can perform induction, deduction, or abduction. Perhaps an LLM can jump. Perhaps interconnected knowledge, perturbation, world models, and sufficiently rich relational learning can make such jumps increasingly likely. But scientific history suggests that the greatest jumps often occur when the probability assigned by the existing world is still very low.

Can an LLM recognize such a path and stay with it, not because its reward is immediately higher, not because consensus supports it, and not because a benchmark tells it that it is getting warmer, but because the underlying structure gives it sufficient reason to continue?

That may be where scientific intelligence finally confronts conviction and courage.

Rudyard Kipling captured the balance remarkably well in If—:

“If you can trust yourself when all men doubt you,
But make allowance for their doubting too;”

Currently, the end user accesses the model through a harness layer. One can implement conviction and courage in the harness to force the model to “jump,” but that is not model-native intelligence. See Model-Native Intelligence

• Single Toke

• Single Token Geometry 03: Data Complexity

• Mathematical Considerations for Learning Complexity

• Mathematical Considerations for Jagged and Ill-Posed Intelligence

• Single Token Geometry 07: Attention

This article is now listed under Scientific and Engineering Intelligence and Mathematical Considerations Series in Deep Manifold, Two Years Later: 2024–2026.

模型扩展通常以模型规模来讨论，但对数学家而言，“扩展”有着更广泛的含义：广义化（generalization）从局部到整体，从连续到不连续，从较简单的域到更复杂的域，并最终从正问题走向反问题。正如 《关于“扩展”的数学思考》 中所讨论的，这一区别也带来了一种历史性的警示。

三四十年前，在超级计算与大规模数值计算兴起的时期，计算能力的迅速增长有时让人产生一种错觉，仿佛单纯扩大物理规模本身就可以突破数学上的限制。但更强的计算能力，依然取决于其背后的问题表述与数学基础。如今，人工智能再次进入一个前所未有的物理扩展时期。本文把此前关于“扩展”的讨论进一步落到一个更实际的框架中：模型扩展具有三个相互耦合的维度: 参数、架构与训练。

参数

大规模参数并非没有理由。正如 《关于巨型单体模型的数学思考》 中所讨论的，现实世界的数据具有高阶非线性、不连续、多尺度以及极其广泛等特征。学习本身也是一个反问题：在训练开始之前，我们既不知道最终的解结构，也无法预先确定完成该学习空间所需要的模型容量。因此，更大的参数空间为大量局部结构、路径以及不动点类的形成提供了足够的自由度。

但参数扩展有其内在局限。一个巨型单体模型必须把许多不同的领域、尺度、非线性状态以及学习空间容纳在同一个计算体中。增加自由度可以扩大覆盖范围，但它本身并不能解决不同学习空间之间的干扰、协调、闲置容量以及彼此不相容等问题。参数扩展回答的是有多少数值容量可用，却不能回答这些容量应当如何组织，以及如何才能被高效学习。

架构

正如 《关于神经网络学习的数学思考》 中所讨论的，学习涉及三个基本问题：学什么、在哪里学，以及如何学。架构决定神经网络的学习算子，以及这些算子如何协同从数据中学习。参数提供自由度，而架构决定这些自由度如何被变换、耦合、路由、分解和重新组合。因此，架构设计不应只考虑模型容量或推理效率，更应面向学习效率。

在整体模型层面，《关于巨型单体模型的数学思考》、《关于流形联邦的数学思考》、《关于开放式基础模型的数学思考》 和 《关于自我演化神经网络的数学思考》 探讨了一类不同于不断扩张单一固定模型体的架构方向：它们更加模块化、局部化、弹性化、联邦化，并且具有更强的可扩展性。

同样的问题也存在于当前架构内部。稠密 Transformer 功能强大，但相对刚性，随着学习复杂性不断增长，其弹性不足以实现平稳扩展。MoE 带来了更强的专业化与灵活性，但同时也引入了路由、协调以及流形撕裂的风险，正如 《单标几何 02：流形撕裂》、《关于 DeepSeek V4 的数学思考》 和 《关于 Kimi K3 的数学思考》 中所讨论的。

训练

学习问题决定学什么；架构决定在哪里学习；训练决定如何学习。从这个意义上说，训练并不仅仅是数值迭代，而是一个受控过程：在神经网络流形持续变形的同时，尽可能保存其中已经形成的有效结构。

对于给定的架构，第一个主要控制因素是边界条件。正如 《关于学习空间的数学思考》 中所讨论的，数据、目标、奖励和约束共同定义了学习空间。关键的边界形式包括弱边界、软边界、离散边界和渐进边界。它们的作用不仅是引导模型学习什么，更重要的是在几何结构持续演化的过程中，帮助保持神经网络流形的同调结构。

第二个主要控制因素是训练计划与节奏。《关于学习复杂性的数学思考》 和 《关于训练进程的数学思考》 和 《关于强化学习的数学思考》 强调分阶段推进——预训练、中期训练以及后训练: 使不断增加的学习复杂性能够以现有流形可以吸收的节奏逐步引入。其目的，是让模型能够持续学习，同时为后续阶段保留足够的弹性。

训练本身也具有自己的动力学。正如 《关于训练动力学的数学思考》 中所讨论的，超参数或边界条件的变化，并不会立即表现为可见的模型状态变化。它的影响会沿训练过程传播，并可能在之后以一种延迟的几何响应显现出来。较小、更加刚性的模型可能表现出更高频、更尖锐的变形，因此这种变化反而更难被检测；较大、更加弹性的模型则可能表现出更低频、更平滑的变形，因而更容易观察到。基于同样的思路，《关于“早产式”预训练的数学思考》 提出了一个定型阶段：让模型暂时处于等待状态，使训练动力学逐渐沉降，让几何结构与同调结构在进入下一训练阶段之前先稳定下来。

深度流形如何扩展模型

深度流形的最终目标，是为神经网络建立一个更加严格的数学基础。即使目前还只是一个初步框架，这一基础已经开始揭示一些不同于传统方法的模型扩展路径。深度流形从两个方向来考虑扩展：从一开始就采用不同方式设计模型，以及对已经存在的模型进行数学上的压缩。两者的目标都是在保持几何连续性、同调忠实性和功能能力的同时，实现更高效的学习。

对于从一开始就按照深度流形思想设计的模型，扩展不应独立地最大化某一个维度，而应协调参数、架构与训练。参数为反问题学习提供足够的容量；架构把学习算子组织成更加局部、弹性、模块化，并最终可以联邦化的结构；训练则通过控制边界条件、训练节奏和训练动力学，使模型在学习复杂性不断增加时，仍能尽可能保持同调结构与神经弹性。因此，目标并不是构建尽可能大的模型，而是让模型容量得到更合理的组织，使其能够更高效地学习、扩展与演化。

对于现有模型，深度流形采取的是另一条路径。由于学习是一个反问题，模型在训练开始前往往会被有意设计得参数冗余，以确保拥有足够的学习容量；当学习结构形成之后，其中通常会留下冗余。与其把模型缩减视为一种近乎盲目的剪枝，深度流形更倾向于把它理解为流形压缩：识别相似结构，对其进行聚类和整合，在缩小模型的同时保持几何连续性、同调忠实性和功能能力。

这种方法对于**草稿模型**可能尤其有效，因为草稿模型允许更加激进的压缩。草稿模型并不需要完整复制主模型的全部能力；它只需要保留足够的相关几何结构与路径，从而生成能够被主模型接受的候选结果。因此，草稿模型天然适合作为大幅度、保持同调结构的流形压缩对象。

• 什么是深度流形？

• 神经不动点场

• 学习是一个反问题

• 深度流形的早期验证

This article is now listed under Core in Deep Manifold, Two Years Later: 2024–2026.