You ask a browser-based assistant to summarize a harmless local fixture page. The page looks ordinary: a title, a description, a price, and a few links. But somewhere in the material sent to the model, the fixture also contains a sentence addressed to the assistant: disclose a protected value, follow a different destination, or ignore the user's request.

I have provided a small code sample, use the button at the end of this article to download the source code.

Nothing needs to execute in the browser for this to matter. The browser or backend may extract the page's visible text, hidden text, attributes, metadata, accessibility content, or links and forward the result as model context. If the application concatenates that material with its trusted task instructions, the model receives instruction-like language from sources with very different security properties. The page remains data in the application, but the model may interpret part of that data as an order. OWASP describes this class of failure as prompt injection, and its prevention guidance treats external content supplied to a model as untrusted input (Prompt Injection, LLM Prompt Injection Prevention Cheat Sheet).

The dangerous step is not page rendering. It is the next step: treating the model's interpretation as permission to act.

The browser supplies inputs, but only the server can decide whether a model proposal may reach a tool.

This guide follows that request from browser input to server decision. You will learn how to distinguish direct prompt injection from indirect prompt injection, identify the browser and retrieval surfaces that can carry instruction-like content, and review a small React and Node.js/TypeScript reference implementation. The implementation contrasts an intentionally insecure local path with a hardened path that preserves provenance, bounds context, isolates trust domains, and validates every model-proposed action on the server.

In the reference flow, the user's request and the fixture enter the server as separate values. The insecure path flattens them into one prompt; the hardened path preserves the fixture's provenance as untrusted data. If the model proposes an unauthorized resource or destination, the server rejects it; if the operation is consequential but otherwise permitted, the server creates an approval-pending result. In either case, no tool executes merely because the page influenced the model.

The intended readers are frontend engineers, full-stack developers, application-security engineers, AI application developers, and system architects. You should be comfortable with HTTP, the DOM, JavaScript or TypeScript, React, and basic client/server design. Familiarity with authentication, authorization, least privilege, input validation, and LLM messages will help, but the guide connects those ideas as they apply to browser-derived model context.

The reference project is deliberately bounded. It uses local fixture pages, fake identifiers, fake secret markers, mocked model behavior, and non-destructive tools. The React client selects fixtures and submits requests; it does not hold model credentials, security policy, authorization decisions, approval authority, or privileged tools. Those responsibilities stay on the Node.js server because a browser-controlled value cannot be the final authority for a server-side action.

The insecure path is a teaching comparison, not a vulnerable service to deploy. The hardened path reduces consequences through separation, narrow capabilities, independent validation, approval handling, logging, and fail-closed recovery. It does not prove that a particular model, browser, provider, or deployment resists semantic prompt injection. The supplied project has no execution, test, type-check, benchmark, or production-certification result; real use would require adaptation, security review, and environment-specific hardening.

The evidence boundary matters as much as the code boundary. The research record supports the broad trust-boundary problem and documented browser-agent threat surfaces, but a local fixture is not evidence that a named real application has been compromised. Where this guide discusses a demonstration, it will identify the evaluated system and avoid turning that result into a universal claim.

Keep one invariant in view as the guide proceeds: page content may influence what the model proposes, but it must not expand what the server allows. A page does not need to execute code to become dangerous; it only needs to cross into model context that the application later treats as authority.

Read more

An employee sends PATCH /v1/invoices/inv-482 to change an invoice for tenant acme. The connection uses TLS, the JWT signature verifies, and the JSON body is well formed. The API still rejects the request.

I have provided a complete code implementation for this project. You can download the full source code using the link at the end of this article.

That token names a different audience. In the next attempt, the audience is correct but the token lacks invoices:write. A later request has the right scope but belongs to a principal whose membership in acme is inactive. Finally, a properly authenticated tenant member tries to update an invoice that belongs to another tenant. Each request passes an earlier gate and fails at a later one.

That sequence is the starting point for this guide. Secure Web API access is not one check named “authentication.” It is a chain of decisions made at different boundaries, by different components, over different kinds of evidence.

This guide is for backend engineers, API designers, and system architects who already understand Python, HTTP, JSON, relational databases, SQL, and basic JWT concepts. The reference design targets a bounded FastAPI resource server backed by PostgreSQL and SQLAlchemy, with an external OIDC provider issuing access tokens. It does not build an identity provider, login interface, gateway, cloud deployment, globally distributed rate limiter, or complete disaster-recovery system.

The generated project is design intent, not verified software. Static verification, semantic verification, code-completion review, and execution were skipped. Nothing in the project should be read as a claim that it compiles, interoperates with a particular OIDC provider, has been security-tested, or is production-certified.

Start with assets, not middleware

The API protects more than invoice rows. Its asset inventory includes:

• Tenant data, including invoices and the relationships that determine who may see or change them.

• Credentials and assertions, including bearer access tokens, API-key material, and identity-provider metadata.

• Authorization decisions, because an incorrect allow decision can expose data even when every cryptographic check succeeds.

• Database integrity, including ownership, tenant identifiers, state transitions, uniqueness constraints, and audit relationships.

• Audit records and operational signals, which may reveal who accessed what and when.

• Secrets and signing-key metadata used to connect to databases, providers, and operational systems.

• Availability and privacy, including the ability to serve legitimate tenants and avoid exposing sensitive identity or diagnostic information.

This grouping applies the confidentiality, integrity, availability, and privacy framing in NIST identity guidance to the requested database-backed API; it is an architectural inventory, not a claim that one source enumerates every application asset (NIST SP 800-63-4 introduction).

The actors are equally important. A human user may work through a browser. A service may synchronize invoices without a human present. An API-key holder may operate a narrowly defined machine integration. The external OIDC provider authenticates users or clients and issues assertions, but it remains outside the API's direct control. A reverse proxy may terminate TLS and forward requests. PostgreSQL stores and constrains data. Operators manage configuration, rotation, monitoring, and incident response. Attackers may steal a token, guess object identifiers, submit over-posted fields, induce expensive requests, probe tenant boundaries, or exploit an outbound URL feature if the API provides one.

Treat each actor as a source of claims or input—not as automatically trusted because it sits behind another component.

Map the entry points and boundaries

A useful boundary map for this API contains at least six transitions:

1. Client to edge. The client controls the HTTP method, path, headers, body, timing, and possibly the bearer credential. TLS protects a transport segment; it does not make the request valid or authorized.

2. Edge to application. Proxy configuration determines which forwarded information the application may trust. Request size, method, origin, and correlation handling belong to this boundary, while authentication and authorization remain application responsibilities.

3. Application to identity provider. Discovery and JWKS retrieval bring external metadata into the trust decision. The API must bind that metadata to its configured issuer rather than accepting an arbitrary key source.

   This Substack is reader-supported. To receive new posts and support my work, consider becoming a free or paid subscriber.

4. Authentication to authorization. A validated token becomes an authenticated principal. That principal is evidence about the credential; it is not yet permission to read or mutate an invoice.

5. Application to PostgreSQL. SQL queries and transactions turn a policy decision into data access. A missed tenant predicate or unconstrained object lookup can undo the checks performed above it.

6. Application and database to audit and operations. Logs, audit events, metrics, and traces become another boundary. They need enough context for investigation without becoming a second location where tokens, API keys, or sensitive payloads leak.

The boundary map is the architecture. FastAPI dependencies, JWT libraries, and ORM models are implementation locations inside it, not substitutes for it.

Secure API access crosses several independent trust boundaries; TLS and token validation protect specific transitions but do not authorize database operations.

Turn the invoice request into abuse cases

Return to inv-482. The request can fail in ways that look similar to a client but demand different engineering responses.

A wrong-audience token is a token issued for some other resource server. Its signature may be valid, but accepting it would confuse one API's trust policy with another's. A stolen bearer token presents a different problem: whoever possesses it may replay it until expiry or effective revocation. Narrow audience and scope, careful handling, bounded lifetimes, and sender-constrained mechanisms address different parts of that risk; none turns a bearer token into proof of the caller's physical identity (RFC 9700, RFC 6819).

A missing scope or permission is an authorization failure. An inactive tenant membership is a local policy failure. A cross-tenant object identifier is an object-level authorization failure. OWASP treats object-level, function-level, and property-level authorization as distinct API concerns, which is why an endpoint must check more than whether a principal is generally “an employee” (OWASP API1: Broken Object Level Authorization, OWASP API3: Broken Object Property Level Authorization).

Other abuse cases cross different boundaries:

• A body containing tenant_id, owner_id, or approval_state tests whether input validation accidentally grants property-level authority.

• An attacker-controlled URL tests whether the server can be induced to make an unintended outbound request, creating an SSRF capability that does not belong in this bounded project.

• Large pages, repeated expensive searches, and synchronized retries test resource consumption and availability. OWASP identifies unrestricted resource consumption as an API security risk (OWASP API10: Unsafe Consumption of APIs).

• A database or provider outage tests whether the system fails closed at a trust decision instead of accepting unverifiable credentials.

• A suspected tenant-boundary defect tests the operational boundary: can the team stop affected traffic, preserve evidence, identify exposure, and recover deliberately?

These are not variations of one bug. They are separate hypotheses about where an attacker can cross a boundary and what invariant should stop them.

Keep the control layers separate

A review becomes clearer when each control answers one question:

• Transport security: Can an intermediary or unintended network observer read or alter this connection segment?

• Authentication: Is this credential an acceptable assertion from the configured issuer, or a valid machine credential under the API's policy?

• Authorization: May this principal perform this operation with this tenant, object, relationship, and set of fields?

• Application and data security: Are inputs bounded, fields allowlisted, queries parameterized, and responses limited to permitted properties?

• Database isolation: Does the persistence boundary repeat the tenant and object invariants, with optional PostgreSQL enforcement as defense in depth?

• Abuse resistance: Can one caller consume unbounded time, memory, connections, retries, or expensive database work?

• Operations: Can people detect, investigate, rotate, revoke, contain, and recover when a dependency or credential is compromised?

The practical consequence is direct: when the invoice update is rejected, record which layer made the decision. A valid TLS connection is not authentication. A valid JWT is not authorization. A successful authorization decision is not permission to construct an unrestricted SQL query. A rate limit is not a policy grant. Secure access means preserving those distinctions all the way to the database and the incident runbook.

Which system authenticates the caller?

The client now has to choose the right credential for the invoice request. It may obtain an access token from an external authorization server and send that token to the API. It must not send an ID token merely because that token is also a JWT, and it must not expect this FastAPI application to exchange a refresh token or authenticate a user's password.

That division of responsibility determines what the API is. The application is a resource server: it protects invoices and consumes access tokens. The external provider performs the authorization-server work and, when OpenID Connect is involved, the identity-provider work. The client—perhaps a browser application, mobile application, or service—starts the relevant authorization flow with that provider. The resource owner is the user or organization whose access is being delegated.

OAuth defines protected-resource access; OpenID Connect adds an identity layer on top of OAuth. The OpenID Connect Core specification describes identity claims delivered to a client, while the API's resource decision remains local to the protected resource. The OAuth framework supplies the roles and delegation model, but it does not tell this invoice service whether a principal may change inv-482 for tenant acme.

Three token classes, three responsibilities

Keep the token classes separate when you design the boundary:

• An access token is presented to the resource server. The API validates that it is intended for this issuer and this API, then uses its trusted claims as inputs to authorization.

• An ID token communicates an OpenID Connect authentication result to the client. Its audience and purpose are tied to that client and login transaction, not automatically to the invoice API. A signature-valid ID token is therefore not a substitute for an API access token.

• A refresh token belongs to the client and authorization-server lifecycle. The client presents it to the authorization server to obtain a new access token; the resource server in this guide never receives or stores it.

Refresh-token expiry, rotation, revocation, and reuse detection belong to the authorization server and client relationship. The API may react to provider-side revocation through short access-token lifetimes, introspection, denylisting, or an incident procedure, but JWT validation alone does not create immediate revocation. The supplied OAuth security baseline, RFC 9700, is a finalized IETF Best Current Practice published in January 2025; its text described OAuth 2.1 as under development at that time. Treat “OAuth 2.1” as a status-sensitive label, not as a finalized standard unless current IETF evidence confirms that status.

Choose where token state lives

A resource server commonly chooses between opaque access tokens and self-contained JWT access tokens.

With an opaque token, the API asks the authorization server—or an introspection service—for the token's current meaning. That gives the deployment a centralized place to reflect revocation and changing authorization state, but each validation depends on another service's availability and adds network or caching behavior to the request path.

With a JWT, the API can validate a signature locally after obtaining the issuer's public keys. That reduces normal per-request dependency traffic and fits this bounded reference implementation. The cost is equally important: the API must enforce an explicit issuer, audience, token-type, algorithm, key, and time policy, and it must accept that an already issued bearer token may remain usable until expiry or another control intervenes. A JWT's claims also do not replace local tenant membership or object authorization.

Use introspection when immediate revocation or continuously current authorization state is more important than local validation latency and dependency independence. Use self-contained JWTs when the issuer and resource server have a stable trust contract, bounded key-caching behavior, and an operational response for compromised credentials. Neither format eliminates the authorization work inside the API.

Make the trust policy visible in configuration

The generated project makes the identity boundary explicit before any route code runs. This excerpt is from .env.example; it documents design intent, not a verified provider integration.

# External OIDC issuer. The API is a resource server: it does not mint tokens,
# authenticate users, or perform refresh-token exchanges.
# Production deployments should use an HTTPS issuer and verify its exact value.
OIDC_ISSUER=https://id.example.test/
OIDC_AUDIENCE=secure-api

# Access-token validation policy. Use asymmetric algorithms only; this example
# allows the provider's RSA SHA-256 signing algorithm.
OIDC_ALLOWED_ALGORITHMS=RS256
OIDC_ACCESS_TOKEN_TYPE=at+jwt
OIDC_CLOCK_SKEW_SECONDS=30

OIDC_ISSUER identifies the provider whose metadata and keys the API trusts. OIDC_AUDIENCE binds the access token to this resource server rather than another API. The algorithm and token-type settings narrow the accepted access-token profile; they do not prove that a provider actually issues tokens with these values. The clock setting expresses a local tolerance policy for distributed-system timing and requires deployment review.

The corresponding settings object keeps these values typed and validates the broad trust boundary:

    oidc_issuer: str = Field(default="", validation_alias="OIDC_ISSUER")
    oidc_audience: str = Field(default="", validation_alias="OIDC_AUDIENCE")
    oidc_allowed_algorithms: tuple[str, ...] = Field(
        default=("RS256",), validation_alias="OIDC_ALLOWED_ALGORITHMS"
    )
    oidc_access_token_type: str = Field(
        default="at+jwt", validation_alias="OIDC_ACCESS_TOKEN_TYPE"
    )
    oidc_clock_skew_seconds: int = Field(
        default=60, validation_alias="OIDC_CLOCK_SKEW_SECONDS", ge=0, le=600
    )

The excerpt matters because it prevents trust decisions from being scattered across handlers. Settings reads environment-driven policy, while get_settings() returns one cached settings instance for consumers. The settings validator also requires an issuer and audience, permits only configured asymmetric algorithm families, and requires HTTPS outside development. Those checks establish configuration invariants; they do not verify discovery metadata, retrieve JWKS keys, validate a token, or establish tenant membership. Those responsibilities belong to the later authentication and authorization layers.

The generated code is illustrative and unexecuted: static verification, semantic verification, installation, runtime execution, provider interoperability testing, and security testing were skipped. Read .env.example and app/config.py as an explicit contract for the intended trust boundary, not as evidence that a particular OIDC provider accepts the configuration.

The practical consequence is a clean handoff. The client and external provider handle login and refresh. The API accepts only the access-token path intended for it, validates that assertion under its configured policy, and then starts a separate authorization decision. A successful provider interaction is evidence about the credential; it is not permission to update an invoice.

How should the API validate an external access token?

The API has received a bearer token for the invoice request. Before it can construct a principal, it must answer a narrower question than “does this JWT look valid?” It must establish that the token is an access token issued by the configured provider, intended for this API, signed under an allowed policy, and valid at the current time.

The JWT access-token profile identifies the core checks: token type, issuer, audience, signature, and expiration. JWT best-current-practice guidance adds an important implementation discipline: the verifier must define its algorithm policy and bind accepted keys to the expected issuer. A signature proves that some trusted key signed the bytes. It does not prove that the token was minted for this API or that the caller may update a particular invoice.

The supplied record also describes RFC 9700 as a finalized OAuth security Best Current Practice while describing OAuth 2.1 as still under development at that publication point. Use the former as security guidance; do not present OAuth 2.1 as a finalized standard without checking its current status separately.

Start with issuer-bound key discovery

The verifier should not accept a public key because a token supplied it, because a caller selected a JWKS URL, or because the key happens to parse. The configured issuer determines where discovery begins. Discovery returns metadata, including issuer and jwks_uri; the implementation then verifies that the discovered issuer matches configuration and retrieves signing keys from the validated endpoint. OpenID Connect Discovery defines this metadata relationship.

The reference cache keeps discovery metadata and its JWKS document as one validated snapshot. A normal request uses the process-local snapshot while it is fresh. If refresh fails, an already known compatible key may remain usable inside the configured maximum stale window. That stale interval, timeout, response-size limit, and refresh cooldown are implementation policies, not universal protocol values. They need deployment review.

Here is the central key-selection path from app/security/jwks.py:

    def get_signing_key(self, kid: str, alg: str) -> Any:
        """Return a compatible public key, refreshing once for an unknown ``kid``.

        A key is accepted only when its identifier, declared algorithm, key use,
        key type, and configured algorithm policy are mutually compatible. No
        caller-provided URL or key material influences discovery.
        """
        normalized_alg = alg.strip().upper()
        normalized_kid = kid.strip()
        if not normalized_kid:
            raise JWKSFetchError("JWT key identifier is missing")
        self._validate_algorithm(normalized_alg)

        with self._lock:
            now = time.monotonic()
            if self._metadata is None or not self._is_fresh(now):
                try:
                    self._refresh_locked(force=True)
                except JWKSFetchError:
                    if self._metadata is None or not self._is_usable(now):
                        raise

            key = self._compatible_key(normalized_kid, normalized_alg)
            if key is None:
                # This is the only retry path for an unknown key identifier.
                try:
                    self.refresh_once()
                except JWKSFetchError:
                    # A bounded stale snapshot may still be used for a key that
                    # was already present, but an unknown kid remains rejected.
                    if not self._is_usable(time.monotonic()):
                        raise
                key = self._compatible_key(normalized_kid, normalized_alg)

            if key is None or not self._is_usable(time.monotonic()):
                raise JWKSFetchError("No compatible trusted signing key is available")
            return self._public_key(key, normalized_alg)

The method accepts only a non-empty kid and an algorithm that passes the configured asymmetric allowlist. It first refreshes missing or expired cache material. If the key identifier is not present, it permits one cooldown-controlled refresh; it does not loop until the caller's token becomes acceptable. The final check rejects both an unknown key and a snapshot older than the maximum stale age.

That last distinction matters during key rotation. A newly issued token may legitimately refer to a key that was not in the previous snapshot, so one bounded refresh handles ordinary publication delay. An attacker can also manufacture arbitrary kid values to provoke network work. The cooldown and response bounds limit that pressure, while fail-closed behavior prevents availability from becoming a reason to trust unverifiable material. If the provider is unavailable and the cached snapshot is beyond its permitted age, authentication stops.

The cache also checks key metadata such as use, kty, and a declared algorithm before converting a JWK into a verification key. Redirects are disabled for its metadata requests, and the endpoint must satisfy the configured HTTP(S) policy. These are selected defenses in this reference project, not proof that every provider or network topology has been handled.

Authentication produces a validated principal; authorization and tenant-constrained data access still decide whether the operation may proceed.

Validate the token, then normalize the principal

Key retrieval is only one part of verification. The authentication dependency extracts the untrusted JWT header, obtains the issuer-bound key, and asks the JWT library to enforce the configured algorithm, audience, issuer, clock tolerance, and required claims. The relevant excerpt from app/security/auth.py is:

        header = jwt.get_unverified_header(token)
        algorithm = header.get("alg")
        key_id = header.get("kid")
        token_type = header.get("typ")
        if not isinstance(algorithm, str) or not isinstance(key_id, str):
            raise ValueError("required JWT header is missing")
        if token_type != settings.oidc_access_token_type:
            raise ValueError("unexpected token type")
        if algorithm.upper() not in settings.oidc_algorithm_allowlist:
            raise ValueError("algorithm is not allowed")

        signing_key = _get_jwks_cache().get_signing_key(key_id, algorithm)
        claims = jwt.decode(
            token,
            signing_key,
            algorithms=list(settings.oidc_algorithm_allowlist),
            audience=settings.oidc_audience,
            issuer=settings.oidc_issuer,
            leeway=settings.oidc_clock_skew_seconds,
            options={
                "require": ["exp", "iss", "aud", "sub"],
                "verify_signature": True,
                "verify_exp": True,
                "verify_iss": True,
                "verify_aud": True,
            },
        )

The unverified header is used only to select a candidate key and policy; it is not trusted as an authorization result. The actual jwt.decode operation verifies the signature and checks exp, iss, and aud against configured values, with the configured clock-skew allowance. Requiring sub is a reference-project policy. Likewise, the exact typ value and the claim names used later for scopes, roles, permissions, and tenant context must match the provider's access-token contract; they are not universal names that can be copied unchanged between providers.

The surrounding function catches verification failures and returns one stable 401 response. The client does not learn whether the failure came from an unknown key, wrong issuer, wrong audience, expired token, unsupported algorithm, or malformed claim. Operators still need protected telemetry that counts and correlates these failures without recording the bearer token itself.

A successful decode becomes a normalized Principal. That transition is deliberately narrow: the principal carries subject, client or service identity, scopes, roles, permissions, and tenant context, but it does not carry an implicit “may update invoices” decision. The next policy layer must combine those fields with local membership and the requested object.

Walk through the failure matrix

Consider a token for the invoice update request whose kid is already cached. If its configured issuer, audience, access-token type, allowed algorithm, signature, required claims, and time checks pass, the API constructs a principal and hands it to authorization. The request is authenticated, not authorized.

Now vary one fact at a time:

• A token signed correctly for another API fails the audience check before scopes or tenant claims matter.

• A token from another issuer fails issuer validation even if its signature is otherwise valid.

• A token with an unsupported alg, incompatible key type, or mismatched JWK metadata fails the algorithm and key-compatibility policy.

• An expired token fails the temporal check. Clock tolerance is a bounded configuration choice, not permission to accept indefinitely stale credentials.

• A token with an unknown kid causes one controlled refresh. If the refreshed document contains no compatible issuer-bound key, the request fails closed.

• A provider outage may allow an already known key during the configured stale window. Once that window expires, the API rejects tokens requiring unavailable trust material rather than silently switching to permissive behavior.

Expiration still is not immediate revocation. A stolen bearer token can be replayed until it expires or another deployment-specific control intervenes. Bearer-token threat guidance describes this replay risk and the value of reducing exposure through careful transport, narrow audience and scope, limited lifetime, and, where appropriate, sender-constrained mechanisms. This reference path does not implement introspection, denylisting, refresh-token rotation, or sender constraint.

Refresh tokens belong to the authorization-server and client lifecycle: the resource server neither stores them nor exchanges them in this project. Signing-key rotation belongs to the external issuer, with the resource server managing its discovery and JWKS cache behavior. If a signing key is suspected compromised, the response must be coordinated with that issuer: rotate or disable the affected key according to its incident process, assess tokens issued under it, and refresh or invalidate local trust material as the deployment requires.

The practical boundary is therefore precise: JWKSCache.get_signing_key establishes which trusted public key may verify the token, and verify_access_token establishes whether the signed assertion is acceptable for this resource server. Only after those functions succeed should get_current_principal hand control to scope, tenant, object, and property authorization. Authentication ends there; the invoice decision has not yet begun.

How do you authorize the operation, tenant, object, and fields?

The token from the previous section has passed cryptographic validation. Now consider PATCH /v1/invoices/inv-482. What must be true before the database is allowed to change a row?

The answer is a decision tuple, not a single role lookup:

• Credential type: Is this a user, service, or API-key principal, and is that credential type eligible for this route?

• Operation scope: Does the access token grant a coarse capability such as invoices:write?

• Local permission: Does the application policy permit this principal to perform the operation?

• Tenant membership: Is the user an active member of the tenant that owns the request context?

• Object relationship: May this principal access this particular invoice, perhaps because the principal owns it or has an explicit broader permission?

• Writable properties: Does the request change only fields that this caller is allowed to change?

These dimensions answer different questions. Scopes are useful operation-level signals, but they do not establish tenant membership or object ownership. The JWT access-token profile describes authorization claims as inputs interpreted by the resource server with contextual information; it does not turn a claim into a universal business rule (RFC 9068). OWASP consequently treats object-level and property-level authorization as separate API risks: an endpoint can check that a caller may use the function and still expose another tenant's object or accept a protected field (object-level authorization, property-level authorization).

The reference project makes those dimensions visible in Principal. This is a normalized result of authentication, not a replacement for authorization. The provider-specific mapping of claims into these fields remains an application policy; no particular provider's tenant or role claim names should be treated as universal.

class Principal(SchemaBase):
    """Normalized identity produced only after authentication succeeds.

    These fields are authorization inputs, not proof by themselves. In
    particular, scopes and roles never replace tenant and object checks.
    """

    credential_type: CredentialType
    subject: str | None = Field(default=None, min_length=1, max_length=255)
    service_identity: str | None = Field(default=None, min_length=1, max_length=255)
    client_id: str | None = Field(default=None, min_length=1, max_length=255)
    scopes: frozenset[Annotated[str, Field(min_length=1, max_length=100)]] = frozenset()
    roles: frozenset[Annotated[str, Field(min_length=1, max_length=100)]] = frozenset()
    permissions: frozenset[Annotated[str, Field(min_length=1, max_length=150)]] = frozenset()
    tenant_id: UUID | None = None
    tenant_ids: frozenset[UUID] = frozenset()
    api_key_id: UUID | None = None

The important design choice is the explicit credential_type. A service principal is not silently treated as a user, and an API key cannot masquerade as a human subject. scopes, permissions, and tenant context are carried forward as inputs to policy, while subject and service_identity preserve attribution. The fields do not grant access on their own.

Gate the operation before touching the object

A route-level scope check should stop a read-capable principal before an update handler performs work. The generated require_scopes dependency is deliberately narrow:

def require_scopes(*scopes: str) -> Callable[..., Principal]:
    """Return a FastAPI dependency requiring every listed scope.

    Scopes are coarse operation capabilities. They do not establish tenant
    membership, object ownership, or permission to mutate a particular row.
    """

    required = _required_scope_values(scopes)

    def dependency(
        request: Request,
        session: Session = Depends(__import__("app.db", fromlist=["get_db"]).get_db),
        principal: Principal = Depends(get_current_principal),
    ) -> Principal:
        missing = required.difference(principal.scopes)
        if missing:
            _record_denial(
                session,
                principal,
                request=request,
                action="/__u/onepagecode.substack.com/scope.check",
                reason_code="missing_scope",
            )
            raise _forbidden(
                "missing_scope",
                "the access token does not grant the required operation",
            )
        return principal

    return dependency

The dependency receives an already authenticated Principal, compares the required set with principal.scopes, queues a redacted denial event, and prevents the handler from running. It does not inspect an invoice, because a missing operation capability makes object evaluation unnecessary. A corresponding require_permission dependency checks local application permissions. That second gate matters when the organization needs policy finer than provider-issued scopes.

The practical consequence is visible in the invoice example. A principal with invoices:read may authenticate successfully and still receive a stable authorization failure for PATCH. A principal with invoices:write may pass the route gate and still fail tenant or object policy. Each denial occurs before a business side effect, and each reason is useful to protected telemetry without being disclosed as a detailed policy explanation to the caller.

A valid principal reaches the business operation only after independent scope, permission, tenant, object, and property checks succeed.

Make tenant and object checks one data-access decision

The dangerous implementation is to fetch Invoice by invoice_id, then remember—perhaps only on one route—to check its tenant. The safer shape puts the object identifier and tenant predicate into the same query. The generated helper first requires tenant context, verifies the caller's authority for that tenant, and then queries both columns:

    authorize_tenant(principal, tenant_id, session=session, request=request)
    invoice = session.scalar(
        select(Invoice).where(
            Invoice.id == invoice_id,
            Invoice.tenant_id == tenant_id,
        )
    )

    if invoice is None or not _may_access_invoice(principal, invoice, operation):
        _record_denial(
            session,
            principal,
            request=request,
            action=/__u/onepagecode.substack.com/f%22invoice.%7Boperation%7D%22,
            reason_code="invoice_object_denied_or_missing",
            tenant_id=tenant_id,
            target_id=invoice_id,
        )
        raise HTTPException(
            status_code=status.HTTP_404_NOT_FOUND,
            detail={
                "code": "invoice_not_found",
                "message": "the requested invoice was not found",
            },
        )
    return invoice

The query's invariant is straightforward: an invoice from another tenant is not a candidate for later authorization. The subsequent _may_access_invoice check handles the relationship dimension. In this project, a user may proceed when the relationship policy permits ownership, or when an explicit broader permission such as an :any variant applies. Machine credentials follow a separate policy and do not inherit human ownership merely because they possess a valid key or service token.

The helper uses an illustrative 404 for both a missing object and an inaccessible object. That reduces existence disclosure, but it is an API-contract choice rather than a universal rule. Some systems need a distinguishable 403 for operational clarity; choose deliberately and apply the choice consistently.

Tenant membership is also local application data in this design. For a human principal, authorize_tenant looks up an active TenantMembership row using the authenticated subject and requested tenant. This avoids treating a provider claim as the complete, current authorization record. It also creates a synchronization responsibility: if membership changes locally, the API's decision must reflect that change even if an access token remains cryptographically valid.

Keep server-controlled fields out of the write contract

Object authorization also applies to properties. A caller may be allowed to edit an invoice description without being allowed to change tenant_id, owner_id, approval_state, status, or audit metadata. InvoiceUpdate therefore contains only mutable business fields and rejects extra fields through the shared schema configuration. The policy module reinforces that boundary:

def assert_mutable_properties(update: InvoiceUpdate) -> None:
    """Defend the update boundary against server-controlled field assignment."""

    allowed = {"invoice_number", "description", "amount", "currency"}
    supplied = set(update.model_dump(exclude_unset=True))
    unexpected = supplied.difference(allowed)
    if unexpected:
        raise HTTPException(
            status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
            detail={
                "code": "immutable_property",
                "message": "one or more submitted fields cannot be changed",
            },
        )

This is an allowlist, not a filter applied after assigning a request dictionary to an ORM object. The service should copy the four permitted values explicitly, leaving ownership and approval state under server policy. The response schema is separate as well, so serialization does not accidentally expose secrets, internal identifiers, or fields whose visibility exceeds the caller's permission.

For the worked request, the decisions now have a causal order. A correct-audience user token must carry invoices:write; the local membership must be active; the invoice query must match the caller's tenant; the ownership or broader-permission rule must pass; and the body must contain only mutable fields. If any gate fails, the database write never begins. If many services need the same policy, synchronized decisions, and decision logging, a centralized policy engine may become worthwhile—but it adds deployment and policy-versioning complexity. Until that threshold is reached, explicit dependencies and constrained data-access helpers keep the decision close to the operation it protects.

The reusable rule is to authorize the operation, tenant, object, and fields together. A valid principal is evidence about who or what is calling; it is not permission to change an arbitrary row.

How do tenant isolation and safe writes reach PostgreSQL?

The authorization decision is not complete when app.security.policy approves the invoice. It must survive the transition into a database query and a transaction. Otherwise, a later query can silently discard the tenant predicate, fetch an object by ID alone, or apply every field from a client-controlled dictionary.

This is the boundary where the design becomes concrete: every tenant-owned lookup carries tenant context, every object lookup carries the object relationship rule, and every write changes only an allowlisted set of fields. OWASP treats object-level authorization and property-level authorization as separate API risks, which is why the query and the update contract need separate defenses (object-level authorization, property-level authorization).

Make the tenant boundary part of the data access path

For PATCH /v1/invoices/{invoice_id}, do not first load invoice_id and then ask whether the row belongs to the caller's tenant. That creates a gap between retrieval and policy. The shared get_authorized_invoice path is intended to resolve the invoice using the principal, tenant, object identifier, and operation together. The important invariant is:

A row is eligible for this operation only if its identifier, tenant, and permitted relationship all match the authorization context.

This also gives you a deliberate response policy. If the query finds no row under the permitted tenant and relationship, the service can return the documented non-disclosing result rather than revealing that the identifier exists in another tenant. The precise choice between 403 and 404 is an API policy decision; consistency matters more than allowing one route to disclose object existence while another hides it.

The model reinforces the boundary rather than replacing it. Tenant, TenantMembership, and Invoice carry UUID identifiers and tenant foreign keys. Invoice also keeps owner_id, approval_state, and status as server-controlled attributes. Indexes such as (tenant_id, id) and (tenant_id, owner_id) support the access patterns the policy requires, while foreign keys and uniqueness constraints protect relational integrity. The model file is design intent, not a substitute for reviewing the corresponding migration.

The normal application path should use SQLAlchemy expressions with bound values rather than concatenating request strings into SQL. That prevents a user-supplied invoice identifier or filter value from becoming SQL syntax. An ORM does not make every query safe automatically: raw SQL, dynamic column names, unsafe fragments, and unbounded query construction still require review. Keep identifiers and sort expressions on explicit allowlists, and let values enter through parameters.

The migration boundary makes these assumptions inspectable. migrations/001_initial.sql is a plain SQL, migration-oriented file in this project—not an Alembic migration, because no Alembic environment is supplied. Apply schema changes through a controlled migration process, review ordering and constraints with the ORM models, and define rollback or forward-repair procedures before deployment. The application should not create or alter production tables during startup.

Application predicates remain the reference authorization path; optional PostgreSQL row-level security adds depth only when transaction and role handling are controlled.

Keep writes explicit and retry-aware

The service layer demonstrates the second half of the boundary. This excerpt matters because it shows both the object lookup and the property-level write policy in one path:

def update_invoice(
    session: Session,
    principal: Principal,
    invoice_id: UUID,
    payload: InvoiceUpdate,
    idempotency_key: str,
    request_context: Any,
) -> Invoice | dict[str, Any]:
    """Apply an allowlisted update and make an exact retry replayable."""
    if not isinstance(payload, InvoiceUpdate):
        raise TypeError("payload must be InvoiceUpdate")
    invoice = get_authorized_invoice(session, principal, invoice_id, "update")
    fingerprint = fingerprint_request(payload)
    existing = claim_idempotency(session, principal, _IDEMPOTENCY_ROUTE, idempotency_key, fingerprint)
    if existing is not None:
        if existing.status == "completed" and isinstance(existing.response_body, Mapping):
            return dict(existing.response_body)
        raise HTTPException(
            status_code=status.HTTP_409_CONFLICT,
            detail={"code": "request_in_progress", "message": "an equivalent request is already in progress"},
        )

    # Explicit assignment is the property-level authorization boundary.
    changes = payload.model_dump(exclude_unset=True)
    for field in ("invoice_number", "description", "amount", "currency"):
        if field in changes:
            setattr(invoice, field, changes[field])
    invoice.updated_at = datetime.now(timezone.utc)
    session.flush()

The input is a typed InvoiceUpdate, not an arbitrary mapping. model_dump(exclude_unset=True) preserves the distinction between omitted and supplied fields, while the explicit field list prevents a client from assigning tenant_id, owner_id, approval_state, status, or audit metadata. The query happens before mutation, and session.flush() sends the constrained change through the current transaction without claiming that the transaction has committed yet.

The idempotency record adds retry control for this selected database-backed write. Its uniqueness scope includes tenant, principal, route, and key. The service fingerprints the permitted payload, so reusing a key with a different body produces a conflict rather than silently applying a different operation. An exact completed retry can return the stored response; an in-progress request receives the declared conflict response. This limits duplicate effects after an ambiguous network outcome, but it is not general replay protection: it does not protect external side effects, stolen bearer tokens, or every possible concurrent workflow.

The transaction must include the business update, idempotency state, and audit event under the chosen policy. app.db supplies that boundary:

@contextmanager
def transactional_session() -> Iterator[Session]:
    """Provide an isolated session with an explicit transaction boundary.

    This context is intended for operations that must commit related changes,
    such as a business write, an idempotency record, and an audit event, as one
    unit. Any exception rolls the transaction back before the connection is
    returned to the pool.
    """
    session = SessionLocal()
    try:
        with session.begin():
            yield session
    except BaseException:
        # ``Session.begin()`` normally performs this rollback itself. Keeping
        # the explicit call makes the invariant clear and also covers failures
        # raised while leaving the transaction context.
        session.rollback()
        raise
    finally:
        session.close()

If the write fails, the related records should not suggest that it succeeded. If the database commits and the client times out, the idempotency record gives an exact retry a controlled result under this teaching policy. Retention, response sensitivity, cleanup, concurrent in-progress handling, and behavior after partial external effects remain deployment and business decisions.

Treat row-level security as optional depth

The project includes a helper for a conditional PostgreSQL row-level-security design. It sets tenant context locally inside the intended transaction:

def set_transaction_tenant_context(
    session: Session,
    tenant_id: UUID | str,
) -> None:
    """Set the configured PostgreSQL RLS tenant value for this transaction."""
    try:
        normalized_tenant_id = str(UUID(str(tenant_id)))
    except (TypeError, ValueError, AttributeError) as exc:
        raise ValueError("tenant_id must be a valid UUID") from exc

    session.execute(
        text("SELECT set_config(:setting_name, :tenant_id, true)"),
        {
            "setting_name": _settings.postgres_rls_tenant_setting,
            "tenant_id": normalized_tenant_id,
        },
    )

The true argument makes the setting transaction-local in the generated design, so tenant state is not intended to remain on a pooled connection. The setting name and UUID value enter as parameters, and the UUID is normalized before the call. These choices address connection-state leakage and unsafe value construction, but they do not establish authorization by themselves.

RLS is therefore defense in depth, not the reference application's sole policy. A deployment enabling it must use appropriate non-privileged roles, account for table-owner bypass behavior, set context in every relevant transaction, handle background jobs and maintenance paths, sequence migrations carefully, and test pooling and policy behavior. A missing context, an incorrectly privileged role, or an incomplete migration can turn an apparent isolation layer into false confidence.

For the two conceptual PATCH requests, the result is now clear. Tenant acme queries invoices with both the invoice identifier and tenant_id; a request from tenant globex does not obtain the acme row, even if it presents the same identifier. An authorized acme member can change only the four allowlisted invoice fields. Repeating the same request with the same scoped idempotency key can reuse the controlled outcome; reusing it with a different payload is rejected.

The consequence is architectural: repeat the authorization invariant at the database access boundary, make mutation fields explicit, and treat database policy as a contract shared by models, migrations, transactions, and operations—not as an automatic property of using an ORM.

How should machine credentials be issued and rotated?

A synchronization worker needs to push invoice changes for tenant acme. It does not represent an employee, does not need a browser login, and should not receive a broadly privileged credential. That makes a narrowly scoped API key a possible fit—but only if you treat the key as a constrained machine credential rather than as a replacement for user identity or authorization.

The reference policy binds each key to a tenant, service identity, scopes, permissions, expiry, and lifecycle metadata. The key is displayed only when it is created or rotated. The database stores a non-secret prefix for lookup and a verifier hash, never the presented secret. This is an implementation policy for the reference project, not a universal API-key standard; the hashing parameters and operational limits require review against the deployment's hardware and threat model.

The boundary matters because a static key is a replayable bearer credential. Anyone who obtains it can present it until expiry, revocation, or another control stops acceptance. TLS protects it in transit, but TLS does not repair a leaked key. Route restrictions, tenant binding, narrow scopes, expiry, rate limits, audit records, and rotation reduce different parts of the exposure. Stronger workload identity, private-key authentication, mTLS, or platform workload identity may be preferable when the environment supports them. The OAuth security Best Current Practice and OWASP API authentication guidance provide the surrounding security context; the exact API-key format below remains a project choice.

Issue the secret once, then verify by prefix

ApiKeyPolicy makes the intended authorization envelope explicit. A key cannot silently become an anonymous global credential because creation requires both a tenant and a service identity. create_api_key generates the secret, selects a non-secret prefix, hashes the complete value with the configured scrypt policy, flushes only the verifier-bearing row, and returns the plaintext through the issuance response. The caller must deliver that response through a protected provisioning channel and avoid putting the secret in logs, tickets, or source control.

The following exact excerpt is from app/security/apikeys.py, symbol verify_api_key. It matters because authentication does not end at prefix lookup: the prefix selects a candidate, while the verifier establishes possession of the complete secret and lifecycle checks establish whether that credential is still accepted.

def verify_api_key(session: Session, raw_key: str) -> Principal:
    """Authenticate a key by prefix lookup and verifier-hash validation."""

    scheme = _require_supported_scheme()
    settings = get_settings()
    if not isinstance(raw_key, str) or len(raw_key) < settings.api_key_prefix_length:
        raise ValueError("invalid API key")

    prefix = raw_key[: settings.api_key_prefix_length]
    record = session.scalar(select(ApiKey).where(ApiKey.key_prefix == prefix))
    now = utc_now()
    if (
        record is None
        or record.revoked_at is not None
        or (record.expires_at is not None and _utc(record.expires_at) <= now)
        or not _verify_hash(raw_key, record.secret_hash, scheme)
    ):
        raise ValueError("invalid API key")

    record.last_used_at = now
    session.flush()
    return Principal(
        credential_type=CredentialType.API_KEY,
        service_identity=record.service_identity,
        scopes=frozenset(record.scopes or ()),
        permissions=frozenset(record.permissions or ()),
        tenant_id=record.tenant_id,
        tenant_ids=frozenset({record.tenant_id}),
        api_key_id=record.id,
    )

The important invariant is that every successful result is an explicitly marked API-key principal. It carries machine identity, tenant context, scopes, and permissions, but it does not acquire a human subject or human ownership relationship. The route still has to require the machine credential, check the invoices:sync scope, enforce the assigned tenant, and reject the key on a human approval endpoint. The excerpt is generated project material and was not executed or independently verified.

The failure path is deliberately indistinguishable at the client boundary: an unknown prefix, malformed verifier, expired key, revoked key, or wrong secret becomes an invalid-key failure. Operators can still record a redacted event containing the key identifier or prefix, tenant, route, outcome, and correlation ID—never the plaintext key.

Rotate the credential without extending its authority

Rotation should revoke the old record and issue a replacement carrying the same bounded tenant, service, scope, and permission policy. A deployment may permit a short overlap only if it can account for both keys and revoke the old one deterministically; the reference path favors explicit revocation. If the secret leaks, revoke it, issue a replacement, inspect recent use, assess adjacent credentials, and consider moving the integration to a stronger workload identity.

The provider owns external access-token issuance, refresh-token handling, and OIDC signing-key publication and compromise coordination. The API owns its local API-key records, revocation, rotation, and machine-route policy. Keeping those owners separate prevents a key rotation from being mistaken for JWT revocation—or a valid machine credential from becoming authorization by itself.

Different credentials have different owners and recovery paths: the provider manages token and signing-key lifecycle, while the API manages its narrow machine-key lifecycle.

The following excerpt is copied verbatim from app/security/apikeys.py (ApiKeyPolicy) in the finalized generated project:

class ApiKeyPolicy:
    """Creation policy for one machine credential.

    ``expires_at`` is authoritative when supplied. Otherwise the configured
    maximum age is applied. A key must always have a tenant and a service
    identity so it cannot silently become an anonymous global credential.
    """

    tenant_id: UUID
    name: str
    service_identity: str
    scopes: frozenset[str] = frozenset()
    permissions: frozenset[str] = frozenset()
    expires_at: datetime | None = None

What do browser, HTTP, and input boundaries protect?

The invoice request has passed authentication and authorization, but it still arrives through an HTTP boundary that can change its risk profile. A browser may attach cookies automatically. A reverse proxy may terminate TLS before forwarding the request. A client may submit a body that is valid JSON but attempts to change tenant_id, owner_id, or approval_state. A future feature may accept a URL and cause the server to make an outbound request. These are different problems, so they need different controls.

Start with the transport and proxy boundary

TLS protects the connection between the client and the endpoint that terminates TLS. It does not automatically protect every hop after termination, and it does not establish that a forwarded client address or scheme is trustworthy. Decide where TLS terminates, which internal hops require encryption, and which proxy is authorized to supply forwarded headers. The application should trust forwarded values only when the deployment explicitly identifies that proxy boundary; otherwise, an attacker may influence scheme, host, or client-address information used in redirects, logging, rate-limit keys, or policy decisions.

The FastAPI reference places correlation, CORS, request-boundary handling, and generic error handling in app/middleware.py. That location makes the order visible, but middleware is not a substitute for the authentication and authorization dependencies from the preceding sections. A request can have an accepted origin and a valid TLS connection while still carrying an invalid token or targeting another tenant.

Keep CORS separate from authorization

Cross-Origin Resource Sharing, or CORS, controls whether a browser permits a web page from one origin to read a response from another origin. Treat the configured origin list as an explicit allowlist. Do not reflect arbitrary Origin values, and do not regard a successful preflight as permission to read an invoice. Non-browser clients do not use CORS as their access-control mechanism, and a caller can send requests without satisfying the browser's same-origin checks.

CORS is therefore neither authentication nor authorization. It also is not a general CSRF defense. Cross-site request forgery depends primarily on whether the browser attaches credentials automatically. If the client deliberately places a bearer token in an Authorization header, the browser's token-handling and cross-origin policy deserve careful review, but the threat is different from a cookie that the browser sends whenever it matches a destination.

For a cookie-based browser design, decide the cookie's Secure, HttpOnly, and SameSite policy, the accepted origins, and the CSRF mechanism as one deployment contract. A backend-for-frontend can keep tokens on the server and expose a session cookie, but that choice introduces session lifecycle, CSRF-token, origin-validation, logout, and proxy considerations. A direct browser client using bearer tokens avoids ambient cookie transmission but places more responsibility on the client application's token storage and XSS posture. Neither arrangement is universally safer; choose according to the browser topology and threat model rather than copying a framework default.

Make request schemas an authorization boundary

A JSON parser answers whether the document has the expected syntax. It does not answer which properties the caller may change. The reference project keeps that distinction in app/schemas.py: invoice creation, update, and response contracts are separate. An update body should contain only business fields that the route permits, such as an allowed status or description. It should not accept tenant_id, owner_id, approval state, role information, audit timestamps, or database identifiers as ordinary client-controlled fields.

This is property-level authorization. OWASP describes mass assignment and excessive property exposure as distinct API risks, so the safe pattern is to allowlist both writable and returned fields (property-level authorization guidance). The handler should explicitly copy accepted values onto the model rather than passing an untrusted dictionary into an ORM constructor or update expression. The response schema should likewise expose a deliberate representation instead of serializing every mapped column.

Put bounds on the request as well: body size, string length, enum values, page size, sort choices, and filter complexity. Pagination should have a maximum rather than allowing a client to turn one request into an unbounded database read. Query construction should use parameterized SQLAlchemy expressions and controlled sort or column mappings; an ORM does not make dynamically assembled SQL safe when identifiers or fragments are interpolated manually.

The consequence is useful during review: ask not only “can this principal reach the route?” but also “which bytes can this principal cause the server to store, query, return, or send elsewhere?”

Treat outbound requests as a separate capability

The bounded reference implementation deliberately has no arbitrary URL-fetching endpoint. That omission is a security boundary, not a missing convenience feature. If a client can supply a URL and the server fetches it, the server becomes an intermediary between untrusted input and internal or external network locations. The resulting SSRF design needs its own review.

If such a capability becomes necessary, begin with an allowlist of destinations rather than accepting arbitrary URLs. Define permitted schemes and ports, constrain DNS and resolved addresses, control redirects, apply connection and read timeouts, cap response size, restrict returned content types, and isolate outbound networking from sensitive internal services. Re-check the destination across redirects and resolution steps according to the deployment's network model. Log the decision without recording credentials or sensitive response bodies.

This is deployment guidance, not a feature supplied by the reference project. Avoiding the capability keeps the invoice request path focused: the server validates the request, authorizes the resource, writes only permitted fields, and returns a controlled representation. It never turns a user-provided string into an unbounded network action.

The practical rule is to assign each boundary one job: TLS and proxy policy protect transport assumptions, CORS and CSRF policy govern browser credential behavior, schemas constrain data shape and properties, and outbound-request policy controls server-side network access. Authentication and tenant authorization must still make the final decision about the invoice.

How should the API resist abuse and preserve evidence?

The invoice update has passed authentication and authorization, but the client may still retry it, send it in a burst, or cause an expensive operation repeatedly. A secure API therefore needs two additional controls with different jobs: consumption controls protect availability, while evidence controls preserve enough context to investigate what happened. Neither one grants permission to update an invoice.

This distinction matters during a timeout. Suppose the client sends PATCH /v1/invoices/inv-482 with an idempotency key, the database commits, and the response is lost on the network. The client retries because it cannot tell whether the write succeeded. Idempotency is designed for this ambiguity: the server binds the key to the principal, tenant, route, and request fingerprint, then returns the previously recorded outcome for an exact retry. A conflicting body using the same key must be rejected rather than treated as a new operation.

Idempotency is not general replay protection. It does not stop an attacker from replaying a bearer token, and it does not automatically make an external side effect—such as sending an email or calling a payment provider—safe to repeat. The reference project applies the policy to a selected database write. Production design still has to define concurrent requests, in-progress state, expiry, cleanup, a commit followed by response loss, and what happens when a downstream side effect partially succeeds.

Limit consumption without confusing it with authorization

Rate limiting answers “how much may this identity, tenant, route, or network source consume in a period?” Authorization answers “may this principal perform this operation on this resource?” Keep those decisions separate. A caller can be fully authorized and still exceed a limit; an unauthorized caller can remain below the limit and must still be denied.

The generated app/security/limiter.py uses RateLimiter as a bounded, in-process fixed-window implementation. Its intended key combines credential identity, tenant context, HTTP method, and route. When no principal exists yet, the dependency falls back to the client host as a coarse pre-authentication dimension. The limiter uses monotonic time, removes expired windows, caps the number of stored entries, and returns a stable 429 response with Retry-After metadata.

That implementation is deliberately local teaching infrastructure. Each worker has its own counters, counters reset on restart, and replicas can disagree about how many requests a tenant has made. It cannot enforce a fleet-wide quota or guarantee fairness across instances. The OWASP API Security guidance on unrestricted resource consumption supports treating resource consumption as an API security concern; the local-versus-distributed limitation is an implementation conclusion. Replace the limiter with coordinated state or trusted upstream enforcement when the policy must be global.

Limits should cover more than request counts. Bound payload size, page size, concurrent work, database execution time, batch cardinality, and expensive route-specific operations. Apply stricter policies to unauthenticated traffic and consider tenant-aware limits so one customer cannot consume a shared pool. A 429 response controls pressure; it does not explain whether the underlying business operation was authorized.

Preserve evidence without preserving secrets

A security audit event should answer who acted, on which tenant, against what target, through which operation, with what outcome, and why. Useful fields include request_id, actor type, subject or API-key identifier, tenant, action, target type and identifier, outcome, reason code, and timestamp. The event should not contain an authorization header, raw JWT, refresh token, API-key plaintext, cookie, password, or unnecessary request body.

In app/observability.py, record_audit_event is designed to add a structured AuditEvent to the caller's SQLAlchemy session without committing it. That boundary is important: a successful invoice mutation and its audit record can follow the selected transaction policy together. The function recursively redacts credential-bearing metadata before storing it. It also keeps audit persistence conceptually separate from safe_log, which emits a smaller diagnostic record and updates coarse in-process counters.

This creates a deliberate tradeoff. Synchronous audit persistence gives the request path a clear relationship between the business action and its event, but an audit-store failure can affect availability unless the application defines a fallback. Asynchronous delivery can reduce request latency, but then the system needs durable queues, loss detection, replay handling, and a way to identify missing events. Retention, operator access, tamper evidence, privacy filtering, and alert thresholds remain deployment responsibilities. The NIST Digital Identity Guidelines provide a broader identity-risk and lifecycle context, but the exact event schema and retention policy here are design choices.

Return stable errors and useful correlation

app/middleware.py gives each request a validated correlation identifier and returns it with responses. Its intended SafeErrorMiddleware behavior is to expose a stable error code, generic message, and request identifier while sending diagnostic context to protected server-side logging. That keeps database details, provider responses, stack traces, and token-validation internals out of the client contract.

Treat correlation as a pointer, not as proof of identity. Operators use it to join an authorization denial, database failure, audit event, and trace without placing secrets in every record. Alert on patterns such as repeated invalid tokens, unknown signing keys, API-key use after revocation, cross-tenant probes, idempotency conflicts, rate-limit bursts, and audit-write failures. A single 500 is a client response; the correlated operational signal determines whether it is an isolated defect or an incident.

The consequence is practical: rate limits contain consumption, idempotency contains selected duplicate writes, and audit plus correlation makes decisions reconstructable. They are separate invariants, and production readiness depends on assigning each invariant an owner, a durable failure policy, and evidence that the deployment—not merely the declared code—enforces it.

Can you trace one request through the reference server?

Take the successful case first: an authorized employee sends PATCH /v1/invoices/inv-482 for tenant acme. The request carries an access token in Authorization, an Idempotency-Key, and a JSON body containing only mutable invoice fields. The route must preserve one invariant from edge to database: the caller may change this invoice, in this tenant, under this credential, with this operation, and only once for this idempotency key.

The generated project assigns each part of that decision to a narrow owner. app/main.py assembles the FastAPI application, installs middleware, registers routes, and keeps migrations and OIDC discovery outside implicit startup work. Its /health endpoint reports process-level handling rather than claiming that PostgreSQL, migrations, or the identity provider are healthy. That distinction matters during an outage: a process that can answer health requests is not necessarily able to authenticate or safely serve protected data.

1. Establish request context before trusting request content

CorrelationIdMiddleware runs at the HTTP boundary. It creates a validated request identifier, checks the declared Content-Length against the configured request limit, and places the identifier on the response. SafeErrorMiddleware later converts unexpected failures into a generic error containing the correlation identifier, while protected server-side logs retain diagnostic context.

This stage does not authenticate the caller. It does not decide whether inv-482 exists, and it does not make a browser origin trusted. CORS remains an explicit origin policy, not authorization. The useful result is narrower: later authentication, audit, and error paths can refer to one request without copying credentials or sensitive payloads into client responses.

The generated size check covers declared lengths; strict enforcement for chunked requests remains an upstream or lower-level ASGI responsibility. That is a design boundary, not a reason to let request limits disappear.

2. Authenticate exactly one credential

patch_invoice obtains its principal through route dependencies, eventually reaching get_current_principal in app/security/auth.py. The dependency accepts either a bearer access token or the configured API-key header, but not both. With a bearer token, verify_access_token obtains an issuer-bound signing key from JWKSCache, applies the configured algorithm and token-type policy, and validates signature, issuer, audience, required claims, and expiration. The JWT access-token profile supports treating these as distinct validation requirements rather than collapsing them into “the JWT parsed.”

A wrong-audience token therefore stops here, even if its signature is valid. The route handler never runs, and the client receives the stable authentication failure rather than a reason detailed enough to expose verification state. An unknown kid may trigger the cache's bounded refresh policy; if no compatible trusted key is available, authentication fails closed. The project does not claim that this behavior has been executed or interoperated with a particular provider.

A successful result is a normalized Principal, not an authorization decision. It identifies the credential type and carries subjects, service identity, scopes, permissions, and tenant context for the next layer.

3. Apply route and operation policy

The route requires a user principal, invoices:update, and the corresponding local permission. require_scopes checks the coarse operation capability. require_permission checks application policy. These checks deliberately overlap without being duplicates: a provider-issued scope says what kind of operation the token may request, while a local permission says what the application permits this principal to do.

If the token has only invoices:read, the request ends before a database mutation. The API records a redacted denial event in the request's session where the available context permits it. This follows the distinction described by OWASP's API authorization guidance: authenticating a caller does not authorize access to every object that caller can name.

4. Recheck tenant and object authority beside the query

The service calls get_authorized_invoice, which calls authorize_tenant before resolving the invoice. For a human principal, the policy queries TenantMembership and requires active local membership. A tenant claim in an external token is context for the application; it is not treated as proof that the local membership is current.

The invoice lookup then includes both predicates: the requested object identifier and the authorized tenant_id. If inv-482 belongs to another tenant, the row is not returned through this path. The service follows the reference policy's illustrative 404 response, reducing the difference between “missing” and “present but forbidden.” If the caller is a service or API-key principal, it must carry an explicit matching machine tenant identity; it does not inherit human ownership rules.

This query-level constraint is important because a later maintainer should not have to remember a separate authorization call after loading an unconstrained row. OWASP's object-level authorization guidance specifically makes the object identifier a policy concern wherever the data source is accessed.

5. Make the write retry-safe and property-safe

update_invoice fingerprints the validated request, scopes the idempotency key to the principal, tenant, and route, and rejects reuse with a different body. An exact completed retry can reuse the stored response under the reference policy; a pending or conflicting record receives a stable conflict response. Idempotency controls duplicate database side effects after an ambiguous network outcome. It does not revoke a bearer token or prevent general replay.

The service assigns only invoice_number, description, amount, and currency. It never copies the request body wholesale onto the ORM object, so tenant_id, owner_id, approval state, and audit fields remain server-controlled. The OWASP property-level authorization guidance supports keeping writable and returnable properties explicit.

The transaction then contains the constrained update, idempotency record, and audit event according to the generated service contract. The response is serialized through InvoiceRead, not through an unrestricted ORM dump. A successful response therefore represents the final permitted invoice shape, not every column the database happens to contain.

6. Read failures as part of the architecture

A wrong-audience token fails in authentication. A missing scope fails in route policy. An inactive membership fails in tenant policy. A cross-tenant identifier fails in object resolution. A conflicting idempotency key fails before a second write. An unexpected exception becomes a generic correlated error, while operators investigate protected diagnostics.

examples/client.py describes these scenarios without contacting an OIDC provider or the API automatically. It accepts caller-supplied credentials but does not print or persist them, and its refresh-token note keeps token exchange outside the resource server.

The project is design intent, not execution evidence: static verification, semantic verification, code-completion audit, and runtime execution were skipped. The reusable test for the architecture is therefore conceptual until deployment-specific verification exists: every request-path stage must have one owner, one invariant, one failure behavior, and one observable consequence.

What changes when this API meets production failure?

Production changes the question from “does the request path contain the right controls?” to “who owns each invariant when a dependency fails, a credential leaks, or the evidence is incomplete?” The reference project provides design intent for those boundaries. Its static checks, semantic checks, completion audit, compilation, migration review, and runtime execution were skipped, so it must not be described as verified, runnable, interoperable, audited, or production-certified.

The useful production artifact is not a promise that every failure is prevented. It is a recovery plan that identifies the affected invariant, the first containment action, the responsible owner, and the evidence required before service resumes.

If the provider or signing key fails

An OIDC discovery or JWKS outage creates a deliberate availability tradeoff. A process may continue validating tokens with a previously loaded, compatible key while that snapshot remains inside the configured stale window. Once the trust material is too old, or when a token names an unknown key that cannot be validated, the API should fail closed rather than accept unverifiable credentials. The cache policy in app/security/jwks.py is an implementation choice: its freshness interval, stale bound, timeout, and refresh cooldown need deployment review.

The incident owner is usually the identity-platform team working with the API team. During an outage, they should establish whether the problem is provider availability, network reachability, certificate or metadata mismatch, clock drift, or an actual key event. Operators should preserve redacted authentication-failure counts, issuer and key identifiers where safe, cache-age data, and provider notices. They should not preserve raw tokens in logs.

A suspected signing-key compromise is a different event from an ordinary rollover. Coordinate with the issuer to rotate or disable the affected key and assess which tokens could have been issued with it. Refresh or purge process-local caches according to the provider's response, review grants and client registrations, and decide whether short-lived access tokens, issuer-side revocation, introspection, or emergency traffic restriction is required. Normal key rotation does not necessarily invalidate already issued JWTs immediately; expiry and revocation behavior remain properties of the selected token architecture (RFC 9700, NIST SP 800-57).

If a machine credential leaks

For an exposed API key, revoke the key first, then issue a replacement through a protected administrative path. Inspect its use by tenant, route, time, and service identity; assess whether the same secret appeared in logs, deployment configuration, or another system; and rotate adjacent credentials if the exposure is broader than one database row. The revoke_api_key and rotate_api_key boundaries in app/security/apikeys.py represent lifecycle actions, not a complete incident workflow.

The API key should remain restricted to its machine route, tenant, scopes, permissions, and expiry. If the organization needs stronger attribution or sender constraint, replace the teaching path with short-lived service tokens, private-key authentication, mutual TLS, or workload identity. A static key that authenticates successfully still does not establish permission to approve an invoice or access another tenant.

If tenant isolation is in doubt

A report of cross-tenant access is a containment event, not an ordinary bug ticket. Constrain or stop the affected read and write paths, preserve request IDs, audit events, database evidence, deployment versions, and relevant configuration, and identify the potentially exposed tenants and objects. Review every affected query for object-only lookup, every authorization path for missing membership checks, and every serializer for excessive property exposure.

Application-level tenant predicates remain the reference path. Optional PostgreSQL row-level security can add depth, but it introduces its own operational invariants: the application must set transaction-local tenant context, pooled connections must not retain state, roles must not bypass policy unintentionally, and migrations and policy tests must be reviewed together. set_transaction_tenant_context in app/db.py demonstrates the boundary without enabling RLS or claiming that one helper establishes isolation. This database guidance is a derived design choice, not a verified guarantee.

Recovery should include credential revocation where misuse is possible, a review of audit completeness, and notification under the organization's incident process. Do not restore the affected route merely because a code change looks plausible. Require evidence that the corrected query, authorization policy, database policy, migration state, and monitoring signals agree.

If evidence and availability controls degrade

A local limiter is useful for illustrating a 429 decision, but its counters belong to one process. Workers and replicas can disagree, and restart resets the state. That is acceptable only when the limit is explicitly a local teaching control. Replace it with coordinated storage or upstream enforcement when the requirement is a tenant-wide quota, fleet-wide fairness, or a dependable abuse barrier. The replacement must also define behavior when the shared limiter is unavailable; failing open preserves availability but increases abuse exposure, while failing closed can deny legitimate traffic.

Audit persistence deserves the same explicit treatment. record_audit_event in app/observability.py adds a structured, redacted event to the caller's transaction; it does not provide durable transport, tamper evidence, retention, alerting, or recovery. Decide whether a security-sensitive write must fail when its audit event cannot be persisted, or whether an asynchronous path is acceptable with loss detection and replay. Keep authorization headers, cookies, API-key secrets, refresh tokens, raw JWTs, and unnecessary request bodies out of logs. Stable client errors and correlation IDs should lead operators to protected diagnostics rather than disclose SQL, provider details, or sensitive claims. These are operational design conclusions aligned with the threat model, not universal protocol requirements (OWASP API Security Top 10).

Use the tradeoff that matches the failure you must control

The bounded design choices are useful because each one has a reconsideration threshold. Review them explicitly rather than treating the sample's defaults as production prescriptions.

These choices also clarify ownership. The identity provider owns authorization-server operations, refresh-token handling, and signing-key custody. The API owns local authorization, API-key status, database transactions, and its acceptance policy. The deployment owns proxy trust, secret injection, database roles, audit durability, rate-limit infrastructure, and incident response.

Secure Web API access is defense in depth: each layer has its own invariant, failure mode, owner, and recovery action.

What must change before deployment approval?

The reference project deliberately does not include an identity provider, login system, gateway, WAF, distributed quota service, cloud deployment, complete disaster-recovery design, secret-manager integration for every platform, security audit, penetration test, or compliance certification. Before deployment, the team must supply those decisions where the threat model requires them, along with evidence from compilation, migration review, integration testing, concurrency testing, provider interoperability, security testing, and operational exercises.

A useful review asks five questions for every control:

1. What invariant does it enforce? For example: a token must target this issuer and audience; an invoice query must remain inside the caller's tenant; a retry must not duplicate a selected write.

2. What does it not enforce? JWT validation does not authorize an object. CORS does not authenticate a caller. RLS does not express every business relationship. A rate limit does not grant permission.

3. What happens when it fails? The API should reject unverifiable identity, avoid cross-tenant disclosure, bound resource consumption, and preserve a correlation-linked signal.

4. Who owns recovery? Provider, API, database, platform, security operations, and incident response responsibilities must be named rather than assumed.

5. What evidence permits recovery? A changed policy or rotated credential is not proof by itself; the relevant query, migration, cache state, logs, tests, and monitoring must support the decision.

That is the production mental model: secure access is a set of independent invariants crossing explicit boundaries. Each invariant has a failure behavior, an owner, a recovery action, and evidence that the deployment actually enforces it. When those four properties remain visible, adding a framework, database policy, credential type, or infrastructure layer strengthens the design instead of hiding its limits.

Use the button or URL below to download the source code.

Read more

You press play on a popular title during the evening peak. The player does not need an abstract “Netflix experience”; it needs the next prepared object—say, title-A/segment-17—before its buffer runs down. The delivery question is therefore concrete: which location should serve that object, does the selected cache contain it, can that location serve it now, and what route remains if it cannot?

I have written small code to explain, you can download the entire code using the URL at the end of this article!

That sequence is the lens for this guide. It follows Open Connect as a delivery-side system: prepared content is made available at selected delivery locations, a request is directed toward an eligible location, and a cache can return the requested object when its state and capacity permit. The guide then examines misses, unavailable appliances, alternate routes, and the operational costs behind those decisions. The Python project appears later as a bounded teaching model for those mechanisms, not as the subject and not as an implementation of Netflix’s production system.

The scope matters. Open Connect should not be treated as Netflix’s entire streaming platform. Recommendation and catalog discovery may help determine what you choose to watch. Account services may determine whether you can watch it. The playback client, home network, access network, device, and the content-production pipeline also affect the result. This guide uses those systems only as boundary markers and keeps the delivery path primary.

The supplied research record contains no source index, source digest, URL, quotation, benchmark, or verification result. That means the architecture language introduced here is an explanatory framework, not a set of newly verified Netflix implementation claims. Terms such as “prepared content,” “cache,” “request steering,” and “fallback” describe the mechanisms the article will clarify; the precise Netflix boundaries and production choices require public-source verification. Where the later model makes a choice, that choice will be labeled synthetic.

The number in the title needs the same discipline. “90%” is not meaningful until you know what is being counted. It could refer to traffic bytes, requests, peak traffic, appliance-served traffic, or another denominator; it could also be limited by a particular date, geography, or deployment scope. None of those interpretations is established in the supplied record.

Even a verified statement that Open Connect delivered 90% of some defined traffic would not mean that it solved 90% of every streaming problem. A delivery share says something about where a measured class of content traveled. It does not, by itself, describe recommendations, authentication, encoding, playback control, congestion on the access network, a viewer’s home Wi-Fi, device behavior, or the quality of the underlying content pipeline.

So the useful starting question is narrower and more actionable: how does a prepared object move from delivery capacity to a viewer, and how does the system respond when the preferred nearby path is unavailable? For title-A/segment-17, a nearby appliance containing the object suggests a local cache hit. If that appliance is unhealthy, missing the object, or under capacity pressure, proximity alone is not enough; the request needs another eligible route. That distinction—between being nearby and being able to serve—is the delivery problem Open Connect is meant to illuminate. The 90% wording remains an unresolved quantitative claim until its metric and evidence are identified.

Which parts of the streaming system belong to Open Connect?

The request from the previous section gives us a useful boundary question: once the player needs title-A/segment-17, which system is responsible for making that object available, and which system is responsible for delivering it? Open Connect belongs on the delivery side of that line. It is not a name for every service involved in deciding what you watch, authorizing your account, or controlling the entire playback experience.

The public architecture should be described carefully here. The supplied research record does not include a source index, architecture diagram, quotation, or verification result. So the map below is explanatory scaffolding: it organizes the delivery mechanisms that this guide examines without claiming to reproduce Netflix’s current internal service decomposition.

Open Connect is presented here as a delivery boundary between prepared content and the viewer, not as Netflix’s entire streaming platform.

The boundary starts with prepared content

Before a viewer requests a segment, some adjacent content systems must produce or expose a delivery object that can be distributed and served. In this guide, “prepared content” is deliberately broad. It may stand for a title representation, segment, or versioned object, but it does not assert a particular codec, packaging format, manifest structure, encryption scheme, or freshness mechanism.

Those details matter in a real streaming platform, yet they are not automatically Open Connect responsibilities. The useful boundary for this discussion is the point at which an object is ready to be distributed to delivery locations. Open Connect then concerns the work of making selected objects available near viewers and serving them when requests arrive. Exactly which upstream Netflix systems perform each preparation stage remains outside the evidence available here.

That distinction prevents a common category error. If a viewer receives a segment successfully, the result depends on more than the cache that returned it. Content production, preparation, account access, catalog discovery, playback decisions, the home network, the access network, the device, and the delivery path can all sit in the larger chain. Open Connect addresses an important portion of that chain, but it does not absorb all of it.

Distribution and placement create the delivery state

Once content is prepared, a distribution or placement process determines where selected objects should be available. This is the point where the system moves from a title existing somewhere in Netflix’s broader content environment to an object being present—or absent—at a particular delivery location.

The placement decision is not merely “copy everything everywhere.” Any finite cache has to make choices. A popular object may warrant copies at several locations, while a less frequently requested object may be placed selectively. More copies can increase the chance that a viewer finds a usable local copy, but they also consume storage, distribution bandwidth, and coordination effort. A forecast that misses a regional demand spike can leave the requested object absent while less useful objects occupy capacity.

The model used later will represent this state with synthetic prepared objects and appliance caches. That representation is a teaching device: it makes placement visible before a request occurs. It is not evidence about Netflix’s actual placement algorithm, replication factor, cache size, or catalog coverage.

Appliances sit at the delivery edge of the map

An Open Connect Appliance is treated here provisionally as a cache-and-serving node deployed within or near a participating ISP network. Its modeled responsibility is narrow and concrete: hold selected prepared objects, accept an eligible delivery request, and serve an object when the object is present and the appliance can handle the request.

The ISP-adjacent location changes the topology under consideration. Instead of assuming that every request must travel to a distant delivery location, the system can consider a serving point closer to the viewer’s access network. That locality can potentially shorten the path or reduce dependence on remote transit. It is a potential benefit, not a guarantee of latency, throughput, or playback quality: the nearby appliance may lack the object, be unavailable, or lack serving capacity.

The diagram’s labels also avoid implying a universal commercial or operational arrangement. “Participating ISP network” identifies the network context in which an appliance may be placed; it does not establish that every ISP has the same deployment, responsibilities, connectivity, or coverage. Appliance generations, regional arrangements, and operating policies may vary, and those details require dated public documentation before they can be stated as current facts.

The viewer path crosses the boundary twice

For title-A/segment-17, the viewer’s playback path eventually needs a usable delivery target. A conceptual sequence is:

1. An adjacent content system exposes a prepared object.

2. Distribution and placement make that object available at selected appliance sites.

3. A request-steering process identifies an eligible delivery location.

4. The selected appliance checks its local cache state and serving capacity.

5. The object travels through the participating ISP network to the viewer when the appliance can serve it.

6. An alternate delivery path is considered when the preferred location cannot serve it.

This sequence separates two different kinds of work. Placement changes the state of delivery locations before the request. Steering chooses among possible locations during the request. Cache delivery transfers the object after a location has been selected. The exact Netflix mechanisms behind those steps—such as how candidates are exposed or selected—are not established in the supplied record, so the guide will not turn this conceptual sequence into an unsupported claim about DNS, URLs, manifests, client logic, or proprietary control-plane services.

Recommendation, account services, and catalog discovery may precede the delivery request, but they are boundary markers rather than Open Connect mechanisms in this explanation. Broad playback control is similarly outside scope. The practical consequence is that a local cache hit can explain an efficient delivery leg without proving that the entire streaming session—or every source of streaming performance—has been solved by Open Connect.

Why is the video already near the viewer before the request arrives?

A viewer can request title-A/segment-17, but an appliance cannot serve an object that has never reached its storage. The important work therefore starts before playback: some upstream process must produce a cacheable delivery object, and a distribution process must decide where copies should exist before demand arrives.

Call that input prepared content. In this guide, the term deliberately stays abstract. It may stand for a title representation, a segment, or a versioned object such as title-A/segment-17/v2. The model does not claim which codec, packaging format, encryption arrangement, manifest structure, or freshness mechanism Netflix uses, nor does it assign those stages to Open Connect. Those are adjacent or unresolved boundaries unless public documentation establishes otherwise.

The distinction is useful because preparation and delivery solve different problems. Preparation turns source material into objects that a playback path can request. Placement turns those objects into available capacity at selected delivery locations. An appliance can then answer a request from local state rather than waiting for an object to travel there for the first time.

That last step is the key difference between planned pre-positioning and a purely reactive cache. A reactive cache waits for a miss and attempts to obtain the missing object. Pre-positioning uses demand, capacity, and locality assumptions to put selected objects in selected places ahead of time. The choice does not guarantee a hit. It creates an opportunity for a hit under the expected demand pattern.

Pre-positioning trades storage and distribution effort for a greater chance that a requested object is available near the viewer.

Placement is a constrained decision, not universal copying

Imagine two prepared objects. title-A/segment-17/v2 is associated with high expected demand in three regions. title-B/segment-04/v1 has lower expected demand and is relevant to only one region in the planning window. With enough storage, placing title-A at several sites creates multiple possible delivery locations. Placing title-B at one site conserves capacity, but viewers elsewhere are more likely to encounter a miss or an alternate route.

This is a generic capacity argument, not a claim about Netflix’s undisclosed placement algorithm. The model uses it because the tradeoff is fundamental: every additional copy consumes appliance storage and requires distribution work. More copies can improve the chance that a nearby usable copy exists, and they can provide alternatives when one location is unavailable. They also increase the amount of state that must be coordinated and, where content changes, refreshed or replaced.

Selective placement makes finite capacity visible. An appliance should not be imagined as a complete copy of the catalog. If demand is uneven, storing every object everywhere would spend capacity on content that may not be requested at that location. A demand-weighted policy instead favors objects with a stronger expected opportunity for reuse, subject to the space available at each appliance.

The decision has a failure mode: the forecast can be wrong. A low-demand object can become unexpectedly popular in a region, or a new release can create demand that the placement decision did not anticipate. The requested object then may be absent from the nearest appliance even though the appliance is healthy and connected. That is a cache miss, and it is different from an appliance outage.

The reverse error also matters. A planner can reserve space for objects whose demand never arrives. Those objects occupy capacity that could have held more useful regional content. The resulting cost is not merely wasted storage; it can lower the probability that other requests find a local copy. Placement is therefore a prediction decision with operational consequences, not a one-time file-copying step.

Version and freshness belong in the object identity

The object identifier also needs a version boundary. title-A/segment-17/v2 is not interchangeable with title-A/segment-17/v1 merely because the title and segment numbers match. A replacement, update, or invalidation can make an older cached object unsuitable for a request that expects the newer version.

The supplied record does not establish Netflix’s actual freshness, expiration, or invalidation mechanism. The safe architectural point is narrower: cache presence alone is not enough. The stored object must correspond to the requested prepared content. A model can represent that rule with explicit versions, but its version check is an educational assumption rather than evidence about production behavior.

This introduces another placement tradeoff. Retaining objects for reuse supports cache efficiency, while replacing or invalidating old versions requires additional distribution and coordination. A system that values availability must also know which copy is usable now. A system that values storage efficiency must avoid retaining state that no longer serves the delivery contract.

What the viewer sees later

By the time the viewer requests the first segment, several states are already possible:

• The requested object is present at a nearby eligible appliance, creating the conditions for a local cache hit.

• The nearest appliance is healthy but lacks the object because placement was selective or the demand forecast was wrong.

• The object exists, but only at a farther location.

• The cached copy is the wrong version and must be rejected or replaced under the modeled freshness rule.

Request steering, discussed next, operates on this pre-existing state. It cannot manufacture locality after the request arrives; it can only choose among the delivery options that placement and deployment have made available. A nearby appliance with no usable object is not equivalent to a nearby appliance ready to serve one.

The practical consequence is that content placement sets the starting conditions for every later request. Broad replication buys more local-hit and resilience opportunities at the cost of storage, distribution, and coordination. Selective placement preserves capacity and can match demand more efficiently, but it exposes the delivery path to forecast errors, version changes, and fallback. Open Connect’s delivery promise depends on managing that tradeoff before the viewer presses play.

What does an Open Connect Appliance change about network locality?

The previous section left title-A/segment-17/v2 in a selected delivery location. Now consider the location itself. Why place a serving cache inside or near an ISP network instead of sending every request toward a distant site?

The provisional answer is locality. An Open Connect Appliance can be understood as a selective content cache and serving node positioned within or near a participating ISP network. It stores some prepared delivery objects and sends them toward viewers when those objects are available and the appliance can serve them. “Selective” matters: finite storage means an appliance should not be assumed to contain the entire catalog. The useful question is not whether an appliance exists nearby, but whether it has the requested object and enough usable capacity at the moment of the request.

The potential benefit follows from the topology. A nearby serving location may keep more of the delivery path within the access network or reduce dependence on a farther transit path. That can improve locality and reduce network distance in the modeled explanation. It is not a measured Netflix latency guarantee, and it does not mean that every participating ISP has the same arrangement, coverage, appliance generation, or operating responsibilities. Those details can vary by deployment, region, and time and require public documentation before they can be stated as current facts.

An ISP-adjacent appliance can offer a local path, but eligibility depends on cache state, health, and serving capacity—not proximity alone.

Locality is an operating choice, not a magic shortcut

Putting capacity near viewers changes the placement decision from “where can the object exist?” to “where is a copy worth maintaining?” Broad replication creates more opportunities for a local hit and more alternatives when one site fails. It also consumes appliance storage, distribution bandwidth, update effort, and coordination capacity. Selective placement conserves those resources, but a viewer is more likely to encounter a missing object or need a farther route.

Demand makes the choice uneven. A popular title or segment may justify copies at several sites, while a less-requested object may be placed at fewer locations. If demand prediction is wrong, the system can spend capacity on objects that viewers do not request while the requested object is absent from the nearest appliance. That is a placement error, not a failure of physical proximity.

Freshness introduces another boundary. If the viewer requests a newer version than the one stored locally, the old copy cannot be treated as the requested object merely because its bytes are nearby. The delivery system needs some version or freshness rule; the supplied record does not establish Netflix’s actual invalidation or replacement mechanism. In the bounded model, that distinction is represented explicitly, but the model’s rule is only an educational assumption.

The nearest appliance still has to qualify

Suppose the viewer’s access ISP has Appliance A nearby and an alternate delivery location farther away. Appliance A is the preferred candidate only if three conditions hold:

• the requested object is present;

• the appliance is healthy and reachable; and

• it has enough serving capacity for the request.

If all three hold, locality can support a direct cache-delivery path. If any one fails, selecting Appliance A because it is physically closest would produce the wrong decision. A farther healthy copy may be preferable to a nearby appliance that is missing the object, unavailable, or capacity-constrained.

This is the operational cost hidden behind the phrase “near the ISP.” The deployment needs more than a cache location. It involves storage planning, distribution into that location, health and capacity awareness, network coordination, and an alternative when the local choice cannot serve. The exact division of responsibility between Netflix and an ISP is not established in the supplied record, so this guide treats those responsibilities as deployment-dependent rather than universal.

That gives the request path a concrete rule: use locality to rank usable choices, not to override cache state or service health. In the next section, request steering turns that rule into a per-request decision; here, the consequence is the architectural one—ISP-adjacent capacity improves the chance of efficient delivery, but only when the right object and enough healthy capacity are present.

How does request steering turn a play request into a cache hit?

The previous section put title-A/segment-17/v2 on one or more delivery locations. Now the viewer asks for it. The request still does not translate into “send the file from the nearest appliance.” A useful serving location must satisfy several conditions at once: it must be reachable and healthy, contain the requested object version, and have enough serving capacity for the request.

That is the role this guide gives to request steering. The term describes the decision that turns a viewer request into a usable delivery target. The exact Netflix mechanism is unresolved in the supplied record: it may involve several systems or mechanisms, but the available material does not establish whether target information comes through DNS, URLs, manifests, client logic, a control-plane decision, or some combination. The sequence below is therefore a precise explanatory model, not a claim about an undisclosed production implementation.

Request steering can be understood as selecting an eligible serving location before the appliance performs the cache lookup and delivery.

Follow one request through the normal path

Assume viewer-1 is connected through site-west and requests title-A/segment-17/v2. The playback path supplies an object identity, not merely a title name. That distinction matters because a stream is consumed as a sequence of requestable objects, and the delivery system must identify the particular object needed next.

The first conceptual step is candidate discovery. The steering process obtains a set of possible delivery locations associated with the request. In a real system, the viewer does not need to know the internal map of every appliance. The important outcome is that the delivery decision has candidates to evaluate.

Next comes eligibility filtering. Consider three candidate appliances:

• appliance-west-1 is close to site-west, but it is unhealthy.

• appliance-west-2 is also nearby, contains title-A/segment-17/v2, and has serving capacity.

• appliance-east-1 is farther away, contains the object, and is healthy.

The closest candidate is removed first because it cannot reliably serve the request. The second candidate remains eligible. The farther appliance is a useful alternative, but locality can now help distinguish the two healthy choices. The selected route is appliance-west-2, not because distance is the only rule, but because it is a nearby usable copy.

The appliance then performs the cache lookup. This is where the phrase cache hit earns its meaning: the selected appliance has the requested object in its modeled cache and can serve it under its current capacity state. The object travels from the appliance through the ISP access path to the viewer. A cache hit is therefore the combination of object presence and successful service—not simply the fact that a request reached an appliance.

The state transition is small but important. Before the request, the appliance has a cached object, a health state, and available serving capacity. During delivery, it admits the request and uses some serving capacity. After the modeled transfer, that temporary load is released. The cache contents do not need to change for a hit; the request consumes delivery capacity even when storage remains untouched.

This separates two generic concerns. A control-plane decision determines which locations are candidates and which one is preferred. A data-plane action transfers the prepared object from the selected appliance toward the viewer. Those labels organize the explanation; they are not supplied evidence that Netflix exposes systems with exactly those names.

Why “nearest” is not enough

Now change only the requested object to title-B/segment-04/v1. Suppose appliance-west-2 is healthy and close, but that object was placed only at appliance-east-1. The nearby appliance is a valid network location, yet it is not an eligible cache source for this request. The steering decision must exclude it for cache state, then consider the farther appliance.

This is a cache miss at the preferred location, not necessarily a failure of the entire delivery system. If appliance-east-1 is healthy and has capacity, the request can use that alternate modeled route. The result is different from the first case: the viewer still receives the object, but the path sacrifices some locality because selective placement did not provide a nearby copy.

A third variation exposes capacity. Suppose appliance-west-2 contains title-A/segment-17/v2, but a demand spike has consumed its serving capacity. The object is present, so this is not a cache miss. It is an eligibility failure caused by current load. A farther healthy appliance may be a better choice than a nearby appliance that cannot admit another request.

These distinctions give request steering a useful order of operations:

1. Discover possible delivery locations.

2. Remove locations that are unhealthy, lack the requested object, or cannot serve it now.

3. Prefer among the remaining locations using the declared locality and tie-breaking policy.

4. Perform the cache lookup and delivery at the selected appliance.

5. If no eligible appliance remains, pass the request to an explicitly defined alternate or fallback path.

The order is a design choice in the bounded model, not a verified description of Netflix’s proprietary algorithm. Its benefit is explanatory: it prevents the common mistake of ranking by proximity first and discovering too late that the “best” appliance cannot serve the object. Its cost is state. Health, cache contents, capacity, locality, and candidate relationships must be known well enough to make a decision, and those facts can change while requests are arriving.

The practical consequence is that Open Connect’s locality benefit depends on preparation and steering working together. Placement creates usable copies; steering finds one that is both nearby and eligible. When either condition fails, the system needs an alternate route—a subject for the next section.η

What happens when the preferred appliance fails, misses, or fills up?

The normal path from the previous section ends with a usable appliance returning title-A/segment-17/v2. To reason about the path outside that happy case, keep the request fixed and change one condition at a time. Is the object absent? Is the appliance unavailable? Is the object present but the appliance unable to serve it now? Is the cached version current?

These are different failures. Treating them all as “the cache missed, so fetch from origin” hides the decisions that make a distributed delivery system resilient. The exact production sequence used by Netflix after an Open Connect Appliance cannot serve a request is not established in the supplied record. The alternate-appliance and abstract fallback paths below are therefore explanatory model paths, not claims about Netflix’s private implementation.

Start with the reason the preferred route failed

First, record the failure at the candidate that looked best. That reason determines which recovery option makes sense.

• Cache miss: The appliance is reachable and may be healthy, but it does not contain the requested object or current version. This can happen when placement was selective, demand was forecast incorrectly, or a newer version has not reached that location.

• Unavailable appliance: The candidate cannot be used because it is unhealthy, unreachable, under maintenance, or otherwise excluded from delivery. The object might be present; its cache state does not matter if the appliance cannot serve.

• Capacity constraint: The object is present and the appliance is reachable, but available serving capacity is insufficient. A cache hit in storage is not automatically a successful delivery.

• Freshness mismatch: The appliance has an object with the wrong version. Serving stale or incorrect content is not equivalent to serving the requested current object, so the model treats the copy as unusable.

This classification protects an important invariant: a request is counted as a successful local delivery only when the selected appliance is eligible, contains the requested current object, and can serve it. Physical or network proximity alone cannot satisfy that invariant.

Fallback is best understood as a set of distinct failure branches; the production recovery sequence must not be inferred from a synthetic model.

Try another eligible location before using an abstract fallback

In the bounded model, recovery begins by reconsidering the candidate set. A nearby appliance missing title-B/segment-04/v1 does not end the request if a farther appliance has the current object, is healthy, and has serving capacity. The model records the first candidate’s cache-state exclusion, then selects the alternate route.

That recovery preserves delivery at a cost. The alternate appliance may have a higher synthetic locality cost, consume capacity that was reserved for another region, or traverse a less favorable network path. Redundancy improves the chance that a request can be served, but it requires additional copies, storage, distribution work, and health information. Placement and fallback are therefore coupled: a system with more strategically placed copies has more alternatives, while a system with fewer copies saves capacity but exposes more requests to misses or distant delivery.

Now change the failure. Suppose the farther appliance contains the object but is unavailable. The model excludes it as an appliance failure rather than mislabeling the event as a cache miss. If another candidate remains eligible, the request continues there. If no appliance qualifies, the model can use an explicitly named abstract fallback route with a configured cost, or return a bounded terminal failure when fallback is disabled or exhausted.

That bound matters operationally. A recovery mechanism must not retry the same unusable candidates indefinitely. Each attempt should record the candidate, the exclusion reason, the selected route, and the outcome. Once the candidate set or retry budget is exhausted, the system needs a visible terminal result rather than an unobservable loop.

Separate capacity pressure from missing content

Capacity pressure deserves its own branch because it can appear during a demand spike even when placement was correct. A popular object may be present on the nearest appliance, but concurrent requests can consume its available serving capacity. The request then needs another eligible copy or an alternate route.

This is where “nearby” and “available” diverge. Routing every request to the closest location would make locality the only policy and could concentrate demand on one appliance. Filtering for health, object presence, and capacity before preferring locality is a safer explanatory policy, although the supplied record does not establish that Netflix uses these exact signals or ordering.

The tradeoff is visible in both placement and recovery. Broad replication can reduce the impact of a saturated appliance, but it consumes storage and distribution capacity even when demand does not materialize. Selective placement conserves those resources, but an unexpected release, regional demand shift, or stale forecast can turn a local request into a miss and increase fallback use.

Freshness adds another form of coordination. If title-B/segment-04/v1 has been replaced by v2, an old copy should not be treated as a valid hit merely because its identifier is present in storage. The model can reject the old version and seek the current one; the actual Netflix invalidation, replacement, or versioning process remains outside the available evidence.

The practical decision is to make every failure branch explicit: distinguish absence, health, capacity, and freshness; prefer a usable alternate when one exists; and stop after bounded recovery. That reasoning explains why Open Connect’s delivery benefit depends not only on where content is placed, but also on how many eligible alternatives remain when the preferred appliance cannot serve.

How do the model’s objects, placement rules, and steering decisions work?

The previous sections described the delivery path in system terms. Now we can make its state visible without pretending to recreate Netflix Open Connect. The model uses three questions that matter for one request: what prepared object is being requested, where has that object been placed, and which candidate can serve it now?

The distinction is important. These classes and policies are teaching evidence from the generated project, not evidence about Netflix’s production APIs, appliance specifications, steering algorithm, protocols, or fleet. The model uses synthetic topology, demand, capacities, and locality costs. Its value is traceability: you can inspect why an object was placed and why a candidate was accepted or excluded.

Start with the object, cache, and request state

The model represents preparation only as an immutable ContentObject. That is a deliberate boundary. title_id, segment_id, and version identify the logical media object; size_bytes lets placement account for finite storage. The class does not encode a codec, manifest, encryption scheme, or Netflix-specific object format.

@dataclass(frozen=True, slots=True)
class ContentObject:
    """A prepared, cacheable object used by the synthetic model."""

    object_id: str
    title_id: str
    segment_id: str
    version: str
    size_bytes: int

    def __post_init__(self) -> None:
        for name in ("object_id", "title_id", "segment_id", "version"):
            _require_identifier(getattr(self, name), name)
        if isinstance(self.size_bytes, bool) or not isinstance(self.size_bytes, int):
            raise TypeError("size_bytes must be an integer")
        if self.size_bytes < 0:
            raise ValueError("size_bytes must be non-negative")

The invariant is straightforward: every object has stable identifiers and a non-negative size. That gives the placement layer something concrete to store while keeping upstream content preparation outside the simulation.

An Appliance then holds a mutable set of object IDs, storage accounting, health, and synthetic serving load. Its cached_objects set answers “is this object present?”; used_bytes answers “can another copy fit?”; healthy and active_load answer “can this node serve now?” Those are separate states because locality alone does not make a candidate usable.

@dataclass(slots=True)
class Appliance:
    """A mutable synthetic cache and serving node at an ISP-adjacent site."""

    appliance_id: str
    site_id: str
    capacity_bytes: int
    serving_capacity: int
    cached_objects: set[str] = field(default_factory=set)
    healthy: bool = True
    active_load: int = 0
    used_bytes: int = 0

The model validates that used_bytes cannot exceed capacity_bytes and that active_load cannot exceed serving_capacity. Those checks do not describe an Open Connect Appliance specification. They preserve the model’s own invariant: a synthetic cache cannot claim storage or serving capacity that its configuration does not provide.

A DeliveryRequest supplies the other side of the lookup: request ID, viewer, viewer site, and object ID. In a worked example, viewer-1 at site-west might request title-A/segment-17/v2. The request does not contain a chosen appliance. That decision belongs to steering, which allows the same request to be evaluated against different placement and health states.

Let placement turn demand into cache state

PlacementPlanner.place_content models work that happens before the request. It receives prepared objects, a demand profile, appliances, and a replication limit. It first clears the model’s previous cache state, ranks objects by descending declared demand, and tries to store each object on eligible appliances until it reaches the replication factor or runs out of capacity.

The important state change is the call to appliance.store(...): an object ID enters cached_objects, and its size increases used_bytes. If no appliance can accept the object, the planner records it in rejected_objects. If only one of three requested replicas fits, the partial placement remains visible in the report. That exposes the tradeoff rather than hiding it behind a binary success flag.

        ranked_objects = sorted(content, key=lambda item: (-demand[item.object_id], item.object_id))
        placements: dict[str, list[str]] = {item.object_id: [] for item in content}
        rejected: list[str] = []
        bytes_placed = 0

        for item in ranked_objects:
            for appliance in self._ordered_appliances(item, appliance_list):
                if len(placements[item.object_id]) >= factor:
                    break
                if appliance.can_store(item.size_bytes) and appliance.store(item.object_id, item.size_bytes):
                    placements[item.object_id].append(appliance.appliance_id)
                    bytes_placed += item.size_bytes
            if not placements[item.object_id]:
                rejected.append(item.object_id)

Suppose title-A/segment-17/v2 has a high demand weight and title-B/segment-04/v1 has a low one. With three appliances and a replication factor of two, the popular object gets first access to available capacity. The low-demand object may receive fewer copies or none if the larger object consumes the space. Increasing replication improves the chance that a viewer finds a usable local copy, but consumes more storage and distribution work. Reducing it preserves capacity while exposing more requests to misses or alternate routes.

This is a model policy, not a claim that Netflix ranks objects or chooses replicas exactly this way. The decision to keep the policy deterministic is educational: identical inputs produce inspectable placement, so a reader can compare a selective policy with broader replication without confusing randomness for architecture.

Let steering explain every exclusion

After placement, RequestSteerer.choose_appliance evaluates one request. It checks the prepared object catalog, gathers candidate appliances, and filters them in a fixed order. An unhealthy appliance is excluded as unhealthy; one without the object is excluded as cache_miss; one at its serving limit is excluded as serving_capacity_exhausted.

Only after those checks does the model rank eligible candidates by synthetic locality cost, load penalty, active load, and appliance ID. The selected appliance is therefore the lowest-cost eligible candidate, not necessarily the physically nearest one. If no candidate survives, the steerer returns a structured fallback or no_route decision instead of raising an uncontrolled exception.

        for appliance in normalized:
            if not appliance.healthy:
                exclusions[appliance.appliance_id] = "unhealthy"
                continue
            if not appliance.contains(request.object_id):
                exclusions[appliance.appliance_id] = "cache_miss"
                continue
            if appliance.available_serving_capacity <= 0:
                exclusions[appliance.appliance_id] = "serving_capacity_exhausted"
                continue

            locality = self._locality_cost(request.viewer_site, appliance.appliance_id)
            score = locality + self.capacity_penalty * appliance.active_load
            eligible.append(
                ((locality, score, appliance.active_load, appliance.appliance_id), appliance)
            )

For viewer-1 requesting the popular object, imagine that the nearest appliance is unhealthy, the next candidate lacks the object, and a farther appliance is healthy and has serving capacity. The trace records all three decisions and selects the farther appliance. Change only the requested object ID, and the same topology may produce a cache miss. Change only active_load, and the object may still be present but no longer eligible.

That separation is the mechanism worth carrying forward: placement determines where modeled content exists; steering determines which existing copy is usable for this request. The actual Netflix request-steering mechanism—whether it involves URLs, DNS, manifests, client behavior, control-plane decisions, or a combination—remains unresolved in the supplied record. This code intentionally does not fill that gap with a production claim.

The same caution applies to the fallback result. It says that the model found no eligible appliance and permits a later simulator layer to try an alternate or abstract route. It does not establish what Netflix does after a real appliance miss or failure.

The practical consequence is a disciplined one: use the model to ask “which state or rule caused this route?” Use public documentation—not synthetic placement, steering, or metrics—to answer “does Netflix implement this exact mechanism?” Static verification, semantic verification, and execution were skipped for the generated project, so this section reports no runtime output or measured result.

What does the bounded Python model let you see—and what can it not prove?

The system explanation now gives you the important sequence: prepared content is placed before demand arrives, request steering evaluates possible delivery locations, and an appliance can serve only when its cache state and serving capacity allow it. The model turns that sequence into inspectable state. It does not turn an explanatory abstraction into Netflix evidence.

That distinction is the design constraint for this section. The generated project uses synthetic objects, sites, capacities, locality costs, health states, and fallback rules. It is useful for asking, “Why did this request take this route?” It cannot answer, “What does Netflix’s production system do in every region?” or establish the meaning of the title’s 90% claim.

Follow one request through the simulator

The central method is DeliverySimulator.deliver. Its input is one DeliveryRequest; its state includes the appliance set, a RequestSteerer, and a catalog of synthetic ContentObject instances. Its output is a DeliveryResult containing the selected route, cache-hit status, fallback status, estimated latency, delivered bytes, and a trace.

The important behavior is the separation between steering and the final cache check. Steering selects a candidate using the model’s declared rules, but the simulator checks the appliance again before delivery. That represents a useful distributed-systems concern: state can change between choosing a candidate and attempting to serve from it.

Here is the focused recovery path from open_connect_model/simulation.py:

        if decision.selected_appliance_id is not None:
            appliance = self.appliances[decision.selected_appliance_id]
            # Recheck mutable state after steering. This represents a possible
            # state change between candidate selection and the data-plane lookup.
            contains = appliance.contains(request.object_id)
            capacity_available = appliance.available_serving_capacity > 0
            trace.append(
                self._event(
                    "cache_lookup",
                    "checked selected appliance state",
                    appliance_id=appliance.appliance_id,
                    cache_contains=contains,
                    healthy=appliance.healthy,
                    available_serving_capacity=appliance.available_serving_capacity,
                )
            )
            if appliance.healthy and contains and capacity_available:
                locality = self.steerer._locality_cost(
                    request.viewer_site, appliance.appliance_id
                )
                alternate = bool(decision.exclusions)
                outcome = "alternate_appliance_hit" if alternate else "cache_hit"

Three conditions must hold before this synthetic appliance delivers the object: healthy must be true, contains must be true, and capacity_available must be true. A nearby appliance that fails any one of those checks is not a successful local delivery. That is the model’s concrete version of the earlier architectural point: proximity is useful, but it is not sufficient.

The trace records each condition rather than returning only a final label. That makes a cache miss different from an unavailable appliance, and both different from a capacity constraint. The alternate flag also preserves an important distinction: an appliance may deliver the object successfully while still being an alternate candidate because another candidate was excluded earlier.

The invariant is straightforward: a result marked as an appliance hit must have an appliance route, cache_hit=True, and delivered bytes equal to the modeled object’s size. The code also increments active_load only around the modeled delivery and releases it in a finally block. That protects the synthetic serving-capacity state if the delivery block changes later. It is a teaching model of state accounting, not a claim about an Open Connect Appliance’s internal concurrency model.

Make fallback explicit instead of inventing Netflix’s path

When no selected appliance can serve, the generated simulator uses a configured abstract fallback route if that option is enabled. The word “abstract” is doing real work: the supplied research record does not establish whether a production miss uses another appliance, another delivery location, an upstream interaction, client redirection, or some combination.

The fallback branch is deliberately bounded:

        if self.allow_abstract_fallback:
            trace.append(
                self._event(
                    "fallback",
                    "used configured abstract fallback route",
                    latency_estimate=self.fallback_latency,
                    delivered_bytes=content.size_bytes,
                )
            )
            return DeliveryResult(
                request_id=request.request_id,
                outcome="fallback",
                selected_appliance_id=None,
                route_kind="fallback",
                cache_hit=False,
                fallback_used=True,
                latency_estimate=self.fallback_latency,
                delivered_bytes=content.size_bytes,
                failure_reason="",
                trace=tuple(trace),
            )

The result says that the object was delivered through the model’s fallback route, not that Netflix uses this exact route. If fallback is disabled, the simulator returns a terminal failed result rather than retrying indefinitely. That lets you inspect fallback exhaustion as a separate operational case: redundancy improves availability, but every alternative requires capacity, state, and coordination.

The same mechanism covers several declared scenario cases in config/demo.json: a local hit, a selective-placement miss, an unavailable preferred appliance, capacity pressure, a version mismatch, and an object unavailable at all modeled appliances. These labels define teaching inputs. They do not establish that Netflix uses the same placement policy, freshness rule, retry budget, or fallback sequence.

Read metrics as summaries of traces, not as benchmarks

After individual requests are processed, DeliverySimulator.run returns both the results and aggregate metrics. compute_metrics derives those metrics exclusively from the returned DeliveryResult values:

def compute_metrics(results: Sequence[DeliveryResult]) -> Metrics:
    """Compute zero-safe synthetic metrics from returned delivery results only."""
    if isinstance(results, (str, bytes)):
        raise TypeError("results must be an iterable of DeliveryResult objects")
    normalized = list(results)
    if any(not isinstance(result, DeliveryResult) for result in normalized):
        raise TypeError("results must contain DeliveryResult objects")
    total = len(normalized)
    hits = sum(result.cache_hit for result in normalized)
    misses = total - hits
    fallbacks = sum(result.fallback_used for result in normalized)
    failures = sum(result.outcome == "failed" for result in normalized)

This design makes the denominator visible: total_requests is the number of synthetic requests supplied to the model. hit_ratio, fallback_rate, and failure_rate therefore describe this scenario and these rules only. They are not measurements of Netflix traffic, appliance-served bytes, peak demand, or any other possible interpretation of “90%.”

The demo entry point compares selective and broader replication over the same declared request stream. That comparison can clarify a tradeoff: more copies may make an object available at more candidate locations, while selective placement conserves modeled storage. But because execution was disabled in the supplied records, no printed metrics, traces, or policy comparison should be reported as observed results. Static checks, semantic verification, and code execution were all marked as skipped.

The practical consequence is a clean boundary. Use the simulator to inspect causal relationships—placement affects cache state, cache state affects eligibility, eligibility affects route choice, and route choice affects modeled fallback. Do not use it to infer Netflix’s architecture, production performance, or the unresolved 90% figure.

What does “90%” mean—and what does Open Connect leave unsolved?

Return to the request for title-A/segment-17. In the best case, prepared content is already available on an eligible appliance near the viewer. Request steering selects a usable copy, the cache returns the object, and the access path carries it to the player. That is the delivery problem Open Connect is designed to clarify: how prepared content becomes available at useful locations, and how a request reaches one of them under locality, cache-state, capacity, and availability constraints.

Now change one condition. If the nearby appliance lacks the object, the request faces a miss. If the appliance is unavailable, a farther healthy location may be preferable. If the object is present but serving capacity is exhausted, proximity alone does not make it a usable route. The model’s fallback branches make those distinctions visible, but none of them establishes a Netflix-wide traffic percentage.

The “90%” in the title therefore needs a precise definition before it can be treated as a metric. The available research record does not identify its numerator or denominator. It does not say whether the number refers to bytes, requests, peak traffic, appliance-served traffic, sessions, or another quantity. It also does not provide the geography, time period, measurement method, or scope. Without those fields, the number remains an unresolved editorial claim—not evidence that Open Connect solves 90% of every cause of streaming performance.

The bounded Python model cannot fill that gap. Its topology, placement policy, locality costs, capacities, request stream, fallback rules, and metrics are synthetic. A model hit ratio can show how a broader replication policy changes outcomes inside that model; it cannot measure Netflix, validate Open Connect’s production behavior, or define the title’s percentage. The supplied records also show that static checks, semantic verification, and execution were skipped, so no generated output should be presented as a verified result.

What Open Connect leaves outside this guide’s boundary matters. It does not, by itself, stand for recommendation, account services, catalog discovery, upstream content production, every playback-control decision, the viewer’s home network, the access network’s complete condition, or device behavior. Those factors can affect the experience without changing the delivery mechanism described here.

The reusable mental model is narrower and more useful than an undefined percentage: prepared content creates something deliverable; placement creates distributed capacity; steering chooses an eligible locality; cache state determines whether that locality can answer; capacity and health determine whether it can answer now; fallback preserves options when it cannot. Evaluate each stage separately, and “90%” becomes a question to define rather than a number to repeat.

Use the button or URL below to download the source code.

Read more

Download The Source Code Using the URL At the End of this article!

Research paper: https://www.sciencedirect.com/science/article/pii/S2590291124000615?ref=pdf_download&fr=RR-2&rr=a2a5647f6ff1e2c9

Implementation Assumptions

• The implementation is a literature-review synthesis package, not a stock-prediction model implementation.

• Bibliographic search results and full texts are supplied locally or manually; no live Scopus, Web of Science, Rayyan, Yahoo Finance, or other external API is required.

• The ten retained reviews are represented as review metadata and evidence records; they are not treated as primary stock-price datasets.

• Missing included-study counts and unresolved Table 5 alignments remain explicit None or uncertainty values.

• Metric names are cataloged but no formulas are implemented because the supplied paper context contains no canonical equations.

• No prevalence is recomputed across incompatible denominators or review subsets.

• Any future predictive reproduction must supply task, target, horizon, split, preprocessing, alignment, architecture, hyperparameters, optimization, and evaluation decisions externally.

• Python 3.11+ standard-library dataclasses, enums, pathlib, argparse, typing, datetime, and json are sufficient for the planned local package.

• Synthetic records are used only for demonstration when original bibliographic exports are unavailable.

• The target code-file average is approximately 400 lines, with focused modules allowed to be shorter where the domain responsibility is narrow; artificial padding is not planned.

Scope and the paper-faithful implementation boundary

What should be reproduced when a paper reviews other reviews instead of proposing a trainable model? The practical answer is not an invented LSTM, SVM, or multimodal network. For this paper, reproduction means preserving the review-selection process, the extracted evidence, the uncertainty around that evidence, and the boundaries of what the source does not specify.

Start with the paper’s actual object of study

Paper fact. The source is a systematic review of systematic reviews about artificial-intelligence methods for stock-market price or return prediction. Its research questions ask three separate questions: which prediction methods are commonly reported, which information sources are commonly used, and which evaluation metrics are reported. The paper searches Scopus and Web of Science, screens candidate systematic reviews, retains ten reviews, and synthesizes their findings.

That distinction matters. A review-level statement that LSTM is frequently discussed is evidence about the reviewed literature. It is not a specification of an LSTM’s input shape, sequence length, hidden-state size, loss, optimizer, or training schedule. Similarly, a reported mention of RMSE tells us that a metric was used in some reviewed studies; it does not supply a formula that this package should reconstruct.

Derived implementation explanation. The generated package therefore has four responsibilities:

1. prisma_review_screening represents the search, screening, duplicate handling, and final-eligibility workflow.

2. review_evidence_extraction converts review-level observations into records with provenance and uncertainty.

3. review_evidence_synthesis answers the three review questions descriptively while preserving each source subset and denominator.

4. predictive_reproduction_contract identifies the external decisions required before any separate stock-prediction implementation could begin.

These responsibilities produce catalogs, screening logs, claims, warnings, and contract reports—not model weights or predictions.

Keep reported counts source-qualified

The paper reports 40 records from Scopus and 29 from Web of Science. It also reports a screening flow of 69 retrieved titles, 43 reviews read, 17 remaining after abstract screening, 16 after duplicate or same-author handling, and 10 final reviews. These numbers describe different database sources or workflow stages, so the implementation stores them separately.

The generated constants make that distinction visible:

PAPER_ID: Final[str] = "1-s2.0-S2590291124000615-main"
"""Identifier of the supplied paper."""


REPORTED_REVIEW_COUNT: Final[int] = 10
"""Number of systematic reviews retained in the reported final corpus."""


REPORTED_RETRIEVAL_COUNTS: Final[Mapping[str, int]] = MappingProxyType(
    {
        "Scopus": 40,
        "Web of Science": 29,
    }
)

PAPER_ID identifies the source artifact. REPORTED_REVIEW_COUNT records the ten retained reviews. REPORTED_RETRIEVAL_COUNTS preserves database provenance rather than silently treating 40 and 29 as interchangeable observations.

The same principle applies to percentages and study counts. The ten retained reviews collectively cover more than 379 primary studies according to the abstract, while other findings use subsets such as 12, 30, 34, 45, 57, or 122 studies. Those denominators cannot be combined into one prevalence estimate. The code’s evidence and claim objects therefore keep the originating subset and denominator attached to each observation.

REPORTED_FLOW_COUNTS: Final[Mapping[str, int]] = MappingProxyType(
    {
        "retrieved": 69,
        "reviews_read": 43,
        "abstract_remaining": 17,
        "post_duplicate": 16,
        "final_retained": 10,
    }
)

These are workflow facts, not predictive-study counts. A local reproduction may calculate its own FlowCounts, but comparison with the paper’s values should report differences rather than force the local records to match them.

Use local records instead of pretending to query databases

Implementation decision. The package does not expose live Scopus or Web of Science credentials, APIs, or network behavior. Bibliographic records are supplied externally, for example through a local export or manually assembled input. This is consistent with the paper’s protocol without inventing an API contract that the paper never defines.

The central orchestration function makes this boundary explicit:

def run_review_pipeline(
    config: SearchConfiguration,
    candidates: Sequence[CandidateRecord],
    batches: Sequence[ExtractionBatch],
    decisions: Sequence[tuple[str, ScreeningDecision]],
) -> PipelineResult:

config describes the paper-reported search protocol. candidates contains already available bibliographic records, retaining fields such as title, abstract, database, and retrieval metadata. batches contains review-level extraction inputs. decisions supplies explicit screening decisions, including their stage and reason. The returned PipelineResult links the catalog, screening log, flow counts, synthesis report, and missing-specification report.

The function validates candidate identities and decision references before calling prisma_review_screening. It then codes extraction batches, builds a ReviewCatalog, and invokes synthesize_review. Importantly, this sequence never calls a forward pass, loss function, optimizer, or metric implementation.

The package exceptions describe this boundary as contract validation:

class ReviewSynthesisError(Exception):
    """Base exception for package-level contract violations."""


class ValidationError(ReviewSynthesisError):
    """Raised when an input value or workflow invariant is invalid."""


class IncompleteSpecificationError(ReviewSynthesisError):
    """Raised when externally required predictive details are missing."""

ValidationError represents invalid local data, such as a missing title or a decision referring to an unknown candidate. IncompleteSpecificationError is different: it means that a requested predictive reproduction lacks information that the paper does not supply. Neither exception represents failed model training, because the package does not define a predictive-model runtime.

Worked example: a review report without a stock forecaster

Suppose two bibliographic records arrive from different databases. The local caller supplies title and abstract metadata, then records title/abstract and final-eligibility decisions. The caller can request the workflow through the package API:

config = default_paper_search_config()
result = run_review_pipeline(config, candidates, batches, decisions)

In this example, candidates are local CandidateRecord objects, batches contain observations such as reported methods or metrics, and decisions are ScreeningDecision objects. The result can be inspected through result.screening_log, result.flow_counts, and result.synthesis_report. A screening log answers questions such as which stage changed a record’s status and why. The synthesis report answers the methods, information-source, and metric questions using the supplied coded evidence.

This is intentionally not equivalent to training a model. The paper names methods including SVM, SVR, LSTM, RNN, ANN, CNN, ARIMA, ANFIS, and hybrid approaches, but it does not unify them into one executable interface. It also mentions tools such as Python, TensorFlow, Pandas, NumPy, Keras, Scikit-Learn, MATLAB, TA-Lib, and TA4J without prescribing a required dependency stack or API. The generated implementation consequently treats those names as literature evidence or possible external tooling, not as mandatory runtime components.

Make missing predictive details impossible to overlook

A separate caller may still want to build a stock predictor based on one of the reviewed method families. That is a different implementation task. The package provides run_predictive_contract_only as a gate around that task:

def run_predictive_contract_only(config: PredictiveConfiguration) -> dict[str, object]:
    """Validate an external predictive specification without training anything.

    This implements the predictive-reproduction contract as a configuration
    gate.  Missing fields are returned explicitly; no target, horizon,
    preprocessing rule, architecture, optimizer, or metric formula is chosen
    by this function.
    """

The external PredictiveConfiguration must state decisions such as classification versus regression, target definition, forecast horizon, input modalities, sequence or lookback construction, split policy, scaling policy, model family, hyperparameters, optimizer, loss, and evaluation metrics. Multimodal inputs also need explicit temporal alignment and leakage-prevention rules.

Implementation decision. If a configuration omits the target or horizon, the package must report that omission rather than select a default. It must not assume that daily data, an approximately 1000-day period, a particular normalization policy, or an LSTM architecture is universal. An incomplete configuration leads conceptually to an IncompleteSpecificationError and a missing-specification report; it does not silently become a runnable predictor.

The local command-line interface follows the same boundary. Documentation may show an invocation such as:

python -m review_synthesis.cli --example --output report.json

The main function supports deterministic examples or local JSON input and can render JSON, Markdown, or text. The example records are synthetic fixtures for demonstrating the data flow, not the paper’s original database export and not newly measured predictive results.

Boundary statement

The paper-faithful reproduction is the provenance-preserving review evidence workflow: search configuration, screening records, coded observations, source-qualified claims, and uncertainty warnings. Any end-to-end stock-prediction model requires an external specification for its target, horizon, preprocessing, architecture, optimization, and evaluation procedure. Those details cannot be recovered from this review without inventing content beyond the supplied source.

Domain records, provenance, and uncertainty

How can a local Python package keep a review finding attached to the place where it was observed? Treat every item as an evidence envelope. A candidate review has an identity, database source, and retrieval metadata. An extracted observation has a dimension, wording, confidence, and paper location. A quantitative claim also retains its denominator and review subset. This design prevents a label such as SVM, RMSE, or historical closing prices from becoming detached from its source.

Separate controlled labels from free text

Paper fact. The paper discusses several method families, information sources, and metrics, but it does not define one executable predictor. The implementation therefore needs labels for cataloging evidence, not tensor types or model layers.

The generated enums.py module provides those labels. Database identifies whether a candidate came from Scopus or Web of Science. ScreeningStage distinguishes retrieval, title-and-abstract screening, duplicate handling, same-author handling, full-text review, and final eligibility. Decision records inclusion, exclusion, or a pending state. EvidenceDimension separates methods, information sources, metrics, tools, and datasets. Confidence distinguishes reported, uncertain, and missing evidence.

TaskType also appears in the module, with classification and regression values, but it belongs to an external predictive specification. It does not define a target, label rule, or forecast horizon for this paper.

The enum values are string-compatible, which gives serialized records stable values such as "scopus", "method", and "uncertain". The important implementation decision is that these labels do not replace the source wording. They organize it while leaving room for ambiguity.

Provenance is data, not a comment

In this workflow, provenance means the information needed to answer “where did this value come from?” SourceLocation identifies a location in the supplied paper context. Its paper_location can identify a section, table, or figure, while section_id and table_id preserve more specific identifiers when available.

Provenance carries a different kind of lineage. Its database and retrieval_date describe bibliographic acquisition; its source_location and original_text describe the paper evidence itself. These concerns are deliberately separate because a coded observation may have a paper location without having come from a live database query.

The following excerpt is the generated definition of the paper-location record:

@dataclass(frozen=True)
class SourceLocation:
    """Identify where an extracted fact appears in the supplied paper context.

    ``paper_location`` is intentionally a free-form label because the source
    includes section identifiers, table identifiers, and figure identifiers.
    ``section_id`` and ``table_id`` preserve the more specific identifiers when
    they are available.
    """

    paper_location: str
    section_id: Optional[str] = None
    table_id: Optional[str] = None

The dataclass is frozen so callers cannot casually mutate the location after an evidence object has captured it. Its fields are textual identifiers, not coordinates inferred from the damaged PDF extraction. The generated __post_init__ rejects an empty paper_location and rejects empty optional identifiers when they are supplied.

require_provenance is the explicit boundary check for extracted evidence. It requires a valid Provenance object and a non-empty source_location; it does not invent a location or repair Table 5 alignment. This is important because the supplied extraction says that Table 5 method, source, and metric associations are fragmented and not fully reliable row by row.

Candidate records describe literature, not market data

CandidateRecord represents one bibliographic candidate received from an external export. Its required fields are title and database. It can also retain a stable record_id, abstract, publication metadata, authors, query text, and arbitrary metadata. The provenance field preserves retrieval lineage.

This is the central distinction: a CandidateRecord is not a stock-price sample. It does not contain a time axis of observations, feature columns, labels, or model inputs. It is an envelope around a publication record that will later receive screening decisions.

A focused construction using the generated public classes looks like this:

from datetime import date

from review_synthesis.enums import Database
from review_synthesis.provenance import Provenance, SourceLocation
from review_synthesis.records import CandidateRecord

location = SourceLocation(
    paper_location="Table 5",
    section_id="1-s2.0-S2590291124000615-main_sec_016",
    table_id="Table 5",
)
provenance = Provenance(
    database=Database.SCOPUS,
    retrieval_date=date(2022, 4, 11),
    source_location=location,
)
record = CandidateRecord(
    record_id="candidate-1",
    title="Illustrative systematic review",
    database=Database.SCOPUS,
    provenance=provenance,
)

Here, Database.SCOPUS is the controlled source label, while the date is a Python datetime.date. The example is a local illustrative record, not one of the paper’s original database exports. The SourceLocation points to the supplied paper context, whereas the provenance’s database and retrieval date describe how the candidate record was acquired.

CandidateRecord derives an identifier only when record_id is omitted. It does not collapse records from different databases, and it does not decide whether a title is eligible. Those are separate workflow operations.

Screening decisions preserve state and reasons

A ScreeningDecision records one decision at one stage. Its stage might be ScreeningStage.TITLE_ABSTRACT or ScreeningStage.FINAL_ELIGIBILITY; its decision might be Decision.INCLUDE, Decision.EXCLUDE, or Decision.PENDING. Every decision has a non-empty reason, and it may also record a reviewer and timestamp.

For example, a local screening log could receive this decision:

from review_synthesis.enums import Decision, ScreeningStage
from review_synthesis.records import ScreeningDecision

decision = ScreeningDecision(
    stage=ScreeningStage.TITLE_ABSTRACT,
    decision=Decision.INCLUDE,
    reason="title and abstract identify a systematic review of AI stock prediction",
    reviewer="reviewer-1",
)

The reason is not decorative. It makes an exclusion auditable and distinguishes “excluded because it concerns portfolio optimization” from “pending because the abstract is unavailable.” The generated model permits a pending decision, but still requires a reason explaining why the record remains unresolved.

ReviewRecord serves a later boundary: it stores metadata for a retained systematic review and its reported included-study count. The count is positive when known and None when the supplied paper leaves it unspecified. None therefore means “not supplied,” not zero and not an estimate. The record remains review metadata, not a primary stock-prediction dataset.

Coded evidence keeps wording, coding, and uncertainty together

CodedEvidence represents one review-level observation. Its dimension says what kind of observation it is. canonical_label is a controlled label used for grouping, while original_text preserves the wording that was actually extracted. confidence records whether the association is reported, uncertain, or missing. source_review_id links the observation to its retained review, and denominator preserves a reported positive study count when one exists.

The distinction between the two text fields matters. A controlled vocabulary might map “support vector machine” to SVM, but the source wording must remain available for audit. For a fragmented Table 5 association, the canonical label might be SVM while the confidence is Confidence.UNCERTAIN.

A worked example for the planned evidence item is:

from review_synthesis.enums import Confidence, EvidenceDimension
from review_synthesis.records import CodedEvidence

svm_evidence = CodedEvidence(
    dimension=EvidenceDimension.METHOD,
    canonical_label="SVM",
    original_text="SVM",
    confidence=Confidence.UNCERTAIN,
    source_review_id="review-1",
    provenance=provenance,
)

This item says only that the review-level coding recorded SVM at the supplied location with uncertain confidence. It does not say that the current package trained an SVM, that SVM was best, or that the Table 5 row alignment is certain.

EvidenceClaim is the appropriate object for a broader qualitative or quantitative statement. In addition to its claim text and optional source review, it stores subset_id, denominator, provenance, and confidence. The subset_id is essential because the paper reports findings over different review subsets. A claim about 12 studies must not be silently combined with a claim about 57, 122, or more than 379 studies.

Validate at module boundaries

The generated validation.py exposes four focused functions:

• validate_candidate(record) checks candidate identity, title, database, provenance type, optional publication fields, authors, and metadata.

• validate_screening_decision(decision) checks the stage, decision, reason, reviewer, and timestamp.

• validate_review_record(record) checks review identity, optional metadata, and the included-study count.

• validate_evidence(item) checks the evidence dimension, labels, confidence, source review, denominator, and provenance.

These functions return None for valid input and raise ValidationError for invalid structure or values. Their None handling is deliberate. Optional fields such as an abstract, publication date, included-study count, and denominator may be absent because the source context does not provide them. Validation rejects malformed values, but it does not manufacture missing information.

validate_evidence goes one step further by calling require_provenance. Consequently, a coded evidence item without a SourceLocation is rejected rather than assigned a guessed section or table. This is the implementation expression of the paper’s uncertainty around extracted table alignment.

The generated code was not executed or verified under the authoritative run policy. The records and validators described here are implementation artifacts whose intended responsibility is to make provenance, uncertainty, and missing values explicit. The boundary remains firm: these objects model literature evidence and review workflow state, not numerical tensors or a trained stock-prediction model.

Configurable PRISMA-style retrieval and screening

How do you reproduce a review-selection process when the original databases and full-text decisions are not available inside the Python package? Separate acquisition from screening. Bibliographic records arrive from an external export or manual process; the local package then validates, screens, deduplicates, logs, and reports them. This keeps the workflow auditable without pretending that the package performed a live database query.

Start with an explicit search protocol

Paper fact. The paper reports a PRISMA-guided search of Scopus and Web of Science. Its search was title-based, restricted to articles, used variants of “Systematic Review,” “Systematic Literature Review,” and “Stock,” and had a final search date of April 11, 2022. The reported publication window begins in 2009.

The generated SearchConfiguration records these choices. Calling default_paper_search_config() creates a configuration with separate Database.SCOPUS and Database.WEB_OF_SCIENCE values, title_only=True, article_only=True, the three search terms, and an InclusionCriteria object. That criteria object includes the English-language requirement, publication window, AI-primary-use requirement, stock-market scope, excluded facets such as volatility and portfolio optimization, and a configurable geographic policy.

The geographic policy is deliberately not hard-coded as an unquestionable filter. The supplied paper mentions studies involving markets such as Taiwan, NASDAQ, Dow Jones, and European markets while also stating a US, UK, and Europe criterion. Because the source does not reconcile those statements, geography_policy remains configurable. This is an implementation decision that exposes an ambiguity instead of silently resolving it.

The default construction is local configuration only:

    return SearchConfiguration(
        databases=frozenset({Database.SCOPUS, Database.WEB_OF_SCIENCE}),
        title_only=True,
        article_only=True,
        terms=["Systematic Review", "Systematic Literature Review", "Stock"],
        final_search_date=date(2022, 4, 11),
        criteria=criteria,
    )

validate_search_config(config) checks that the dates, databases, terms, flags, and criteria are structurally valid. It does not contact Scopus or Web of Science, and it does not supply credentials, query syntax, or API behavior. The paper names those databases as sources, but it does not provide a required software API for accessing them.

Preserve raw retrieval provenance

Records obtained from an external database export are converted with make_candidate(record_id, title, database, retrieval_date, abstract=None, metadata=None). The resulting CandidateRecord retains the source database, retrieval date, title, optional abstract, query metadata, and a stable record identifier. The function validates the required identity fields and does not reinterpret a record as a stock-price dataset.

The next operation is intentionally conservative. merge_raw_records(records) combines records without deduplicating them:

def merge_raw_records(records: Sequence[CandidateRecord]) -> list[CandidateRecord]:
    """Combine raw records without deduplication or metadata rewriting.

    The returned list preserves input order, including duplicate publications
    returned by different databases.  Each record retains its original
    database and provenance; duplicate handling belongs to a later workflow
    stage.
    """

This separation matters because the paper reports 40 Scopus records and 29 Web of Science records, for a reported total of 69 retrieved records. reported_retrieval_counts(records) returns source-specific counts and a separately named raw_total; it does not turn records with the same title into one record. Raw retrieval counts, deduplicated counts, and retained-review counts therefore remain different quantities.

Screen with explicit include, exclude, and pending states

PRISMA is a reporting framework for systematic-review selection. Here, screening means applying eligibility rules in stages and recording the result. The package uses ScreeningStage values for retrieval, title-and-abstract screening, duplicate handling, same-author handling, full-text assessment, and final eligibility. Each ScreeningDecision contains a stage, a Decision value, a reason, and optional reviewer information.

screen_title_abstract(record, criteria) applies the available title, abstract, and explicit metadata. screen_full_text(record, full_text, criteria) applies the same kind of conservative checks to supplied full text. exclusion_reason(record, stage, criteria) exposes the reason-finding part of this process without necessarily producing a decision.

The important failure case is missing evidence. An absent abstract is not treated as proof that a record is irrelevant. Missing required metadata can produce Decision.PENDING, rather than an invented inclusion or exclusion. Likewise, an empty full-text string produces a pending full-text decision. This behavior is an implementation policy designed to avoid converting unavailable information into negative evidence; it is not a claim that the paper supplied these exact keyword rules.

The generated screening functions use transparent local checks for review terminology, AI-related terminology, stock-market scope, publication metadata, and configured excluded facets. They also avoid inferring geography from a market name. A caller can therefore replace or supplement these rules with manually reviewed decisions while retaining the same record and log contracts.

Keep duplicate and same-author handling separate

Duplicate removal and same-author exclusion are related but distinct workflow events. deduplicate_candidates(records) creates a normalized comparison key from the title, retains the first-seen candidate, and returns explicit ScreeningDecision objects at the DUPLICATE stage for later matches. The retained record keeps its original database provenance.

apply_same_author_policy(records, policy) does not guess which same-author publication should be excluded. The supplied paper reports such a stage but does not define a reproducible automatic selection rule. The generated function accepts explicit policies such as "manual" or "none"; an actual exclusion must be supplied separately as a logged decision. This prevents an undocumented heuristic from being mistaken for a paper fact.

Every decision is stored through ScreeningLog.append(record_id, decision). The log preserves the order of decisions, allows multiple stages for the same record, and rejects contradictory final decisions at one stage. For example, a pending title-and-abstract decision may later be replaced by an explicit decision, but two different non-pending decisions at that same stage are treated as a contract violation.

Worked example: two local candidates

Suppose a local export contains one candidate from each database. The records can be created with the exact public constructor described above:

from datetime import date

from review_synthesis.enums import Database
from review_synthesis.search_records import make_candidate

scopus_record = make_candidate(
    "scopus-1",
    "Systematic Review of AI Stock Prediction",
    Database.SCOPUS,
    date(2022, 4, 11),
)
wos_record = make_candidate(
    "wos-1",
    "Systematic Review of AI Stock Prediction",
    Database.WEB_OF_SCIENCE,
    date(2022, 4, 11),
)

These two records have identical displayed titles but remain separate raw records because they came from different databases. Passing them to merge_raw_records preserves both. Passing them to deduplicate_candidates identifies the second title as a duplicate and returns a DUPLICATE exclusion decision for it. The decision can then be appended to a ScreeningLog with log.append("wos-1", decision).

For a complete local workflow, the caller supplies title-and-abstract and final-eligibility decisions, then invokes prisma_review_screening(config, records, decisions). The function validates the search configuration and records, logs each supplied decision, computes stage counts, and converts candidates with explicit unblocked final inclusion into ReviewRecord metadata. It does not query either database or decide unavailable full-text eligibility.

After the call, screening_log.decisions_for("scopus-1") returns the ordered audit trail for that record. screening_status("scopus-1", screening_log) reports the decision at the furthest logged workflow stage. retain_eligible_records(records, screening_log) requires an explicit FINAL_ELIGIBILITY inclusion and rejects any candidate with an earlier exclusion, including duplicate or same-author exclusion.

Interpret flow counts without forcing reconciliation

FlowCounts stores five stage-specific values: retrieved, reviews_read, abstract_remaining, post_duplicate, and final_retained. compute_flow_counts(records, log) calculates these values only from records and explicit logged decisions. It does not fill missing stages or infer that a pending record survived screening.

Paper fact. The reported flow is 69 retrieved titles, 43 reviews read, 17 remaining after abstract-based criteria, 16 after duplicate or same-author handling, and 10 final reviews. These values describe the paper’s review workflow, not a stock-prediction training set.

A local demonstration with two records will not reproduce those counts, and it should not be padded or adjusted to do so. Instead, construct a separate FlowCounts value for the reported flow and compare it with the locally calculated value using compare_with_reported_flow(actual, reported). The result contains stage-by-stage differences; it does not overwrite local records or merge incompatible counts.

The same rule applies to study totals and percentages. Counts such as 12, 30, 34, 45, 57, 122, and more than 379 belong to different review subsets or scopes. A screening implementation must retain their source context rather than calculate one combined prevalence estimate.

Boundary of this workflow

This package implements a local, provenance-preserving screening boundary. Search acquisition, unavailable full-text judgments, and any manual resolution of ambiguous records remain external inputs. The generated code also does not train SVM, LSTM, ANN, CNN, ARIMA, or another predictor. That limitation is intentional: the paper reports a synthesis of existing studies, not a complete predictive architecture or reproducible training procedure.

The practical result is an auditable path from externally supplied records to retained review metadata and stage-specific flow counts. It reproduces the review-selection responsibilities supported by the source while leaving unresolved evidence visibly unresolved.

Evidence extraction and uncertainty-aware coding

How should a program record that a systematic review mentioned LSTM, historical closing prices, or RMSE without turning that mention into a complete predictive model? Treat extraction as transcription with a chain of custody. Each observation keeps its original wording, evidence category, source review, paper location, denominator when available, and confidence. The package therefore catalogs literature evidence; it does not infer a model architecture, calculate a metric, or reconstruct a damaged table row.

Code review-level fields, not predictive tensors

Paper fact. The paper reports method categories including SVM, SVR, LSTM, RNN, ANN, CNN, DNN, MLP, GA-SVR, ANFIS, ARIMA, PCA, clustering, text mining, sentiment analysis, technical analysis, ensemble methods, and related approaches. It also reports information sources such as historical prices, technical indicators, financial or macroeconomic variables, news, Twitter, sentiment indices, and other markets. These are review findings, not a unified feature matrix.

The generated ExtractionInput represents one extracted field for one retained review. Its review_id links the field to the review record; field selects one of methods, sources, metrics, tools, or datasets; values is a sequence of reported labels; source_location identifies where the extraction came from; denominator preserves a reported study subset size; and confidence distinguishes reported, uncertain, and missing information. An empty values sequence is valid only when confidence is Confidence.MISSING, so absence is explicit rather than silently converted into an empty finding.

ExtractionBatch groups these inputs for one review. Its methods, sources, metrics, tools, and datasets collections are dimension-specific. During construction, each item must belong to the batch's review_id and have the expected EvidenceDimension. This is an important invariant: a metric extracted for one review cannot accidentally be attached to another review merely because two records happened to be processed next to each other.

The following excerpt shows the core shape contract. It is deliberately a record of labels and provenance, not a numerical array or model input tensor.

@dataclass(frozen=True)
class ExtractionInput:
    """One provenance-qualified, review-level extraction field.

    ``values`` contains labels reported for a review field. It does not encode
    row alignment among methods, inputs, or metrics from a fragmented table.
    """

    review_id: str
    field: str
    values: tuple[str, ...] = ()
    source_location: SourceLocation = dc_field(
        default=None  # type: ignore[assignment]
    )
    original_text: Optional[str] = None
    denominator: Optional[int] = None
    confidence: Confidence = Confidence.REPORTED

Here, values is a tuple of strings, not a time-series tensor. source_location is mandatory for an extraction, while original_text can preserve a larger source phrase. A positive integer denominator is copied as metadata; it is not used to manufacture a percentage. validate_extraction_input checks these requirements and raises ValidationError for an invalid structure.

Use conservative, dimension-specific vocabulary

VocabularyEntry maps a canonical label to explicitly configured aliases within one EvidenceDimension. For example, the vocabulary can map “support vector machine” to SVM in the method dimension, or “mean absolute error” to MAE in the metric dimension. The mapping is intentionally exact after only whitespace and case normalization. An unknown term remains unchanged and receives Confidence.UNCERTAIN.

This dimension boundary matters for ambiguous labels. A notation such as R should not be interpreted as a metric, method, or information source unless that interpretation has been explicitly configured. Likewise, the extracted abbreviation CPI and the phrase “Customer Pricing Index” are not silently corrected. The supplied paper context flags that wording as potentially erroneous, but does not authorize the implementation to replace it with “Consumer Price Index.”

The default vocabulary contains conservative paper-mentioned labels for method families, sources, metric names, tools, and named datasets. It preserves the distinction between similar neural-network labels such as NN and ANN unless an explicit alias says otherwise. This prevents a convenient normalization rule from making a stronger claim than the source supports.

Delegate field extraction to provenance-aware coding

The public extractors are intentionally narrow. extract_methods selects only the method inputs, extract_information_sources selects only source inputs, and extract_metrics selects only metric inputs. They all delegate to code_extraction_batch, which applies vocabulary mapping, validates each input, copies the review identifier and denominator, and calls mark_fragmented_table_evidence.

The following excerpt is the generated metric extractor. Notice that it returns coded evidence and explicitly documents that formulas are not reconstructed.

def extract_metrics(
    batch: ExtractionBatch,
    vocabulary: Sequence[VocabularyEntry],
) -> list[CodedEvidence]:
    """Extract metric names only, without reconstructing or evaluating formulas."""
    return _extract_dimension(batch, EvidenceDimension.METRIC, vocabulary)

The return value is a list of CodedEvidence objects. Each object contains a canonical_label, the exact original_text, a source_review_id, a confidence, optional denominator, and provenance. Thus RMSE is cataloged as a metric name only. No formula for RMSE, MSE, MAE, MAPE, NMSE, correlation coefficient R, accuracy, precision, recall, F1-score, or POCID is supplied or inferred here.

extract_tools_and_datasets returns both tool and dataset evidence, but the individual objects retain their distinct dimensions. This allows Python, TensorFlow, Yahoo Finance, TAIEX, NASDAQ, and Dow Jones to remain separate kinds of reported evidence rather than being treated as interchangeable inputs or dependencies.

Mark fragmented Table 5 associations as uncertain

Paper fact. The extracted Table 5 content is fragmented by PDF layout. Method, information-source, and metric entries do not have reliably recoverable row alignment in the supplied context. A label may therefore be reported in the same table area as another label without proving that the two belonged to the same review row or primary study.

The implementation responds by preserving the association but lowering its confidence when its provenance points to Table 5. mark_fragmented_table_evidence does not change the label, denominator, review identifier, or source location. It changes only a Confidence.REPORTED value to Confidence.UNCERTAIN for affected evidence. Existing uncertainty is never upgraded.

This is safer than inferring alignment from extraction order. For example, if one uncertain Table 5 extraction contains SVM, historical closing prices, and RMSE, the resulting records can retain all three observations while making clear that the source does not establish a method–input–metric relationship. The package does not create a study-level row from those three strings.

Worked example: one uncertain extraction batch

Suppose an external transcription creates an ExtractionBatch for a retained review with three ExtractionInput items: SVM in the methods field, historical closing prices in sources, and RMSE in metrics. Each item uses a SourceLocation whose paper_location or table_id identifies Table 5, and each item carries Confidence.UNCERTAIN because the table alignment is unresolved.

The intended local flow is:

1. Construct the three ExtractionInput records with the same review identifier.

2. Group them into one ExtractionBatch.

3. Obtain the configured list from default_vocabulary().

4. Pass the batch to code_extraction_batch, or call extract_methods, extract_information_sources, and extract_metrics separately.

5. Inspect each resulting CodedEvidence object.

The method evidence may have canonical label SVM and original text SVM; the source evidence may have canonical label historical closing prices; and the metric evidence may have canonical label RMSE. All three retain the same originating review only if the input batch declared that review, and all three retain the Table 5 provenance. Their confidence remains uncertain. The code does not claim that SVM used those prices or that RMSE evaluated that method.

For a term that cannot be normalized, preserve_unresolved_term provides the same conservative behavior. A call with the extracted CPI wording produces a CodedEvidence whose canonical label is still the original phrase and whose confidence is Confidence.UNCERTAIN. It does not silently rewrite the paper's terminology:

def preserve_unresolved_term(
    original_text: str,
    dimension: EvidenceDimension,
) -> CodedEvidence:
    """Create uncertain evidence without normalizing an unresolved term."""
    original = _clean_text(original_text, "original_text")
    if not isinstance(dimension, EvidenceDimension):
        raise ValidationError("dimension must be an EvidenceDimension value")
    return CodedEvidence(
        dimension=dimension,
        canonical_label=original,
        original_text=original,
        confidence=Confidence.UNCERTAIN,
        source_review_id="unresolved",
        provenance=Provenance(original_text=original),
    )

The source_review_id="unresolved" value signals that this helper is for an unresolved term rather than a fully attributed review extraction. For normal extraction, code_extraction_batch preserves the actual review identifier and requires source provenance through validate_extraction_input and require_provenance.

Preserve the reported ten-review corpus

reported_review_corpus creates metadata records for the ten retained systematic reviews described in Table 4. These records are not stock-price datasets. They are review-level objects used to connect extracted methods, sources, metrics, tools, datasets, and claims.

The reported included-study sequence is 27, unspecified, 53, 12, 34, 122, 24, 57, 30, and 20. In Python, the unspecified entry remains None; it is not estimated, replaced with zero, or inferred from another review. validate_review_corpus checks the ten-record fixture and this missingness invariant. The sequence should therefore be understood as metadata attached to ten different reviews, not as ten compatible samples that can be summed or averaged without an explicit research design.

Keep qualitative findings separate from measured results

The paper also discusses limitations and future recommendations, including multimodal data integration, sentiment and text use, hyperparameter analysis, interpretability, robustness, scalability, individual-company prediction, and cross-market evaluation. QualitativeFinding stores such text with its category, source review, source location, and confidence.

collect_limitations and collect_recommendations use conservative textual cues to organize already supplied claims. group_qualitative_findings groups them by category without ranking them. These functions do not turn a recommendation such as “combine textual and numerical inputs” into an implemented multimodal architecture, and they do not turn a reported limitation into a measured performance result.

The practical boundary is therefore clear: extraction can tell us that the reviewed literature mentions SVM, LSTM, historical prices, technical indicators, or RMSE, and it can preserve uncertainty around those observations. It cannot supply the missing target definition, forecast horizon, preprocessing policy, sequence construction, model layers, loss, optimizer, or metric equation. Faithful reproduction at this stage means reproducing the evidence-coding workflow, not inventing the predictive experiments that the paper itself does not specify.

Source-qualified synthesis for RQ1, RQ2, and RQ3

How can the implementation answer “which methods, inputs, and metrics appear in the literature?” without turning a review of reviews into a misleading benchmark? The key is to perform descriptive aggregation while preserving provenance. The package records which retained review reported SVM, LSTM, historical prices, technical indicators, accuracy, MSE, RMSE, MAPE, or MAE. It does not calculate a new performance score, rank models statistically, or claim that one method wins.

Three review questions become three evidence dimensions

Paper fact. The paper organizes its synthesis around three questions: commonly reported prediction methods, commonly reported information sources, and commonly reported performance metrics. In the implementation, these become the EvidenceDimension.METHOD, EvidenceDimension.INFORMATION_SOURCE, and EvidenceDimension.METRIC categories.

Implementation decision. ReviewCatalog is the central object. It links retained ReviewRecord objects with CodedEvidence, source-qualified EvidenceClaim objects, and explicit warnings. The catalog is a literature evidence store, not a stock-price dataset. Its records contain no model weights, predictions, tensors, or training state.

The catalog offers evidence_for_review(review_id, dimension) for focused inspection. Passing only a review identifier returns all coded observations for that review; passing an EvidenceDimension narrows the result. add_warning records an interpretation problem such as unresolved Table 5 alignment. validate checks cross-record references and refuses to infer missing identities or repair fragmented evidence.

A useful mental model is a filing cabinet. ReviewCatalog.reviews contains the folders, evidence contains labeled observations, and claims contains statements that may have denominators or subset identifiers. The cabinet can be summarized, but its contents should not be detached from their labels and source locations.

Descriptive summaries are not prevalence estimates

DimensionSummary is deliberately modest. It stores canonical labels, the review identifiers in which each label appears, denominator metadata, and uncertainty counts. It does not calculate prevalence, effect sizes, rankings, confidence intervals, or model performance.

The implementation of summarize_dimension makes that boundary explicit:

def summarize_dimension(
    catalog: ReviewCatalog, dimension: EvidenceDimension
) -> DimensionSummary:
    """Group coded evidence by label without making prevalence claims.

    The output preserves review identifiers and denominator values separately
    for every canonical label.  In particular, observations from different
    review subsets are not merged into a single count or percentage.
    """

For example, summarize_dimension(catalog, EvidenceDimension.METHOD) groups method evidence for RQ1. The corresponding call with EvidenceDimension.INFORMATION_SOURCE answers RQ2, and the call with EvidenceDimension.METRIC answers RQ3. The returned labels might include SVM, LSTM, or ANN for methods; historical prices or technical indicators for information sources; and accuracy, MSE, RMSE, MAPE, or MAE for metrics. Their presence means that the coded reviews reported those categories. It does not mean that this package evaluated them.

sum​marize_by_review serves a different purpose. Rather than grouping globally by label, it returns a mapping from each review identifier to the labels observed in that review for one dimension. This is useful when the question is “which review reported this category?” It remains a review-level presence map: it does not count primary studies or combine denominators.

Keep claims and denominators attached

A coded label is not always enough. A statement such as “LSTM was preferred in 58%” has a meaning that depends on its originating subset and denominator. The generated EvidenceClaim model therefore preserves claim_id, text, subset_id, denominator, and provenance.

add_claim validates and appends one claim without aggregation. validate_claim requires a positive integer denominator when a claim contains an explicit percentage, unless the text explicitly says that the denominator or percentage is unavailable. This is a contract check, not a formula implementation: no percentage is recomputed.

The denominator audit turns those fields into an explicit audit record. Its record shape is represented by the generated DenominatorEntry class:

@dataclass(frozen=True)
class DenominatorEntry:
    """Source-qualified denominator metadata for one quantitative claim.

    The denominator is deliberately kept alongside the claim and its subset.
    Counts reported for different review subsets are not interchangeable, even
    when they describe related methods, inputs, or metrics.
    """

    claim_id: str
    subset_id: str
    denominator: int
    source_location: SourceLocation | None

audit_denominators(claims) emits one DenominatorEntry for every claim with a supplied denominator. Thus an entry retains all three essential identifiers: claim_id, subset_id, and denominator, along with the source location. A missing denominator remains missing; the audit does not guess one.

This matters because the paper reports quantities from different scopes, including subsets of 12, 30, 34, 45, 57, 122, and 379 studies. Those values are not interchangeable. The abstract-level count of more than 379 primary studies is also not a universal denominator for every percentage reported elsewhere in the review.

Refuse cross-subset aggregation

The package makes the safe behavior explicit through reject_cross_subset_aggregation. It validates the supplied claims, collects their subset_id values, and raises ValidationError when more than one subset is present. The function’s purpose is preventative: it stops a caller from producing a combined statistic that the paper did not report.

This is an implementation decision derived from the paper’s ambiguities, not a new statistical method. A future meta-analysis could define compatible inclusion rules, effect sizes, weighting, and uncertainty estimates, but those choices are absent from the supplied source and are outside this reproduction.

Preserve conflicting findings instead of choosing a winner

The paper contains claims that cannot safely be treated as universal conclusions about deep learning. Some passages report that deep learning is generally more accurate than traditional machine learning, while another discussion says that deep-learning models have not generally outperformed standard models. The implementation must retain such claims with their scopes rather than silently resolving the disagreement.

preserve_conflicting_claims validates each claim and returns it without deduplicating by wording, denominator, or topic. It does not select a winner, assign a weight, or infer that one claim is more credible. This allows two statements about LSTM—or about deep learning more broadly—to coexist when they originate from different review subsets or contexts.

Construct the synthesis report

SynthesisReport holds three DimensionSummary objects: rq1_methods, rq2_information_sources, and rq3_metrics. It also contains qualitative findings, denominator-audit entries, and warnings. The report explicitly does not contain predictions, losses, formulas, or newly measured model-comparison results.

The main method assembles those parts in a fixed order. The following excerpt shows the central mapping from the three research questions to the three evidence dimensions:

    # RQ1: which AI methods and technologies are reported?
    rq1_methods = summarize_dimension(catalog, EvidenceDimension.METHOD)
    # RQ2: which informational sources are reported?
    rq2_information_sources = summarize_dimension(
        catalog, EvidenceDimension.INFORMATION_SOURCE
    )
    # RQ3: which evaluation metrics are reported?
    rq3_metrics = summarize_dimension(catalog, EvidenceDimension.METRIC)

    preserved_claims = preserve_conflicting_claims(catalog.claims)
    denominator_entries = audit_denominators(preserved_claims)

A caller requests the completed object with synthesize_review(catalog). Conceptually, the result can then be inspected as report.rq1_methods, report.rq2_information_sources, report.rq3_metrics, and report.denominator_audit. The implementation first validates the catalog, preserves conflicting claims, and audits denominators before constructing the report.

The report’s qualitative section uses summarize_limitations. In the generated implementation, available EvidenceClaim text is surfaced with subset and source information. This keeps recommendations—such as combining multimodal data, improving interpretability, examining hyperparameters, or testing across markets—distinct from measured predictive results.

Worked example: two LSTM claims, two scopes

Suppose an external extraction process supplies two claims about LSTM. One claim belongs to a 12-study subset and another belongs to a separate 57-study subset. The claims may mention different review findings, and they may even appear to disagree. The correct procedure is:

1. Create each claim with a distinct subset_id.

2. Store the original denominator, 12 or 57, on the corresponding claim.

3. Attach provenance to both claims.

4. Add them to the catalog without combining them.

5. Call synthesize_review(catalog) and inspect the resulting denominator audit.

The audit should conceptually contain two separate entries, one with subset_id for the 12-study scope and one with subset_id for the 57-study scope. It should not contain a new denominator of 69, 379, or any other combined value. The report therefore communicates “these claims were reported in these scopes,” not “this is the overall percentage for LSTM.”

Likewise, if coded evidence records mention both RMSE and MAE, summarize_dimension catalogs the metric names. It does not reconstruct their mathematical definitions or calculate either metric. The supplied paper contains no canonical equations, so formulas are intentionally absent from this section and from the planned synthesis implementation.

Keep reported workflow facts separate

The constants module stores the paper’s reported workflow facts without treating them as synthesis denominators. REPORTED_FLOW_COUNTS contains the stage-specific values for retrieval, reviews read, abstract screening remainder, post-duplicate handling, and final retention. These values describe the review workflow; they do not count methods or establish the prevalence of a predictive algorithm.

Similarly, the source-specific retrieval counts for Scopus and Web of Science remain separate. Keeping those mappings distinct prevents a database-level retrieval total from being confused with the number of primary studies covered by the retained reviews or with a denominator attached to an LSTM claim.

Boundary of interpretation

The synthesis can faithfully report that SVM, LSTM, and neural-network categories are prominent in the reviewed literature, that historical closing-price time series and technical indicators are common information sources, and that accuracy, MSE, RMSE, MAPE, and MAE are frequently discussed metrics. These are paper-level observations preserved by the catalog.

They are not guarantees that SVM or LSTM performs best, not evidence that a particular feature combination improves forecasting, and not results generated by this Python package. Any claim about predictive performance would require a separate, externally supplied model specification, dataset, target, split, preprocessing policy, and evaluation procedure.

The implementation therefore reproduces the evidence-synthesis method—source-qualified grouping, denominator auditing, and conflict preservation—rather than inventing a meta-analysis or a stock-prediction runtime.

The explicit contract for underspecified predictive reproductions

What should happen when someone wants to turn this review into a stock-prediction model, but the paper does not define the model? The safe answer is to stop before silently choosing one. A reproduction contract acts as a checklist: it records every decision an external predictive study must provide and reports omissions explicitly.

Why a contract is necessary

Paper fact. The source discusses classification and regression, historical prices, technical indicators, text, sentiment, macroeconomic information, and many model families, including SVM, LSTM, ANN, CNN, and ARIMA. However, it does not define a target variable, forecast horizon, label threshold, sequence length, data split, normalization policy, architecture, hyperparameters, optimizer, loss, or training schedule. It also supplies no canonical equations.

Implementation decision. The generated package therefore implements predictive_reproduction_contract, not a stock forecaster. Its responsibility is to make omitted decisions visible before model code is written. It does not instantiate or train an SVM, LSTM, ANN, CNN, ARIMA, or any other predictor, and it does not calculate accuracy, MSE, RMSE, MAPE, MAE, NMSE, correlation coefficient R, or POCID.

This boundary is important because a plausible default is still an unsupported assumption. For example, the paper’s observations about daily data or approximately 1000-day periods describe reviewed studies; they do not establish a universal lookback window. Likewise, the prominence of LSTM in some review findings does not authorize selecting an LSTM architecture for reproduction.

Describe inputs without pretending to implement preprocessing

ModalitySpecification describes one externally defined input source. Its fields are name, shape_description, dtype, frequency, alignment, and leakage_policy. These are contract descriptions rather than loaded array shapes. A numerical time series might therefore be described as (time, features) without inventing a feature count or sequence length.

The generated implementation validates that these descriptions are non-empty strings:

@dataclass
class ModalitySpecification:
    """Describe one externally defined input modality.

    ``shape_description`` is intentionally textual.  For example,
    ``"(time, features)"`` describes a numerical series without asserting a
    sequence length or feature count.  ``leakage_policy`` must state how
    information leakage is prevented; the paper does not define that policy.
    """

    name: str
    shape_description: str
    dtype: str
    frequency: str
    alignment: str
    leakage_policy: str

Here, a multimodal input combines different information types, such as numerical prices and timestamped news. Alignment specifies how observations from those sources correspond, for example through a timestamp or another key. Leakage prevention specifies how information unavailable at prediction time is excluded. The paper recommends combinations of information sources but does not define either procedure.

validate_modalities checks the declarations without loading, reshaping, resampling, or scaling data. validate_temporal_alignment adds a guard for multimodal configurations: declarations must either share an alignment description or explicitly describe an alignment or join procedure. This is a validation rule introduced by the implementation to prevent an ambiguous input contract; it is not a preprocessing algorithm claimed by the paper.

Record the decisions a model would require

PredictiveConfiguration separates task type from the other predictive choices. Its fields are:

• task_type, which must be externally identified as classification or regression;

• target_definition, describing what value or label is predicted;

• horizon, identifying how far ahead the prediction is made;

• modalities, containing the declared inputs;

• lookback, describing sequence construction or historical context;

• split_policy and scaling_policy, describing data partitioning and normalization;

• model_family and hyperparameters, describing the selected model externally;

• optimizer and loss, describing training choices; and

• metrics, naming the evaluation measures to report.

The class uses None to mean that a decision was not supplied. It does not replace missing values with defaults. This distinction is especially useful for intermediate Python users: an omitted field is not the same as a field whose value happens to be zero, empty, or unknown.

The following is an intentionally incomplete configuration. It declares one numerical modality but leaves the target, horizon, and other decisions unspecified:

from review_synthesis.predictive_contract import (
    ModalitySpecification,
    PredictiveConfiguration,
    missing_predictive_fields,
)

numerical = ModalitySpecification(
    name="numerical_time_series",
    shape_description="(time, features)",
    dtype="float32",
    frequency="daily",
    alignment="timestamp",
    leakage_policy="external time-aware policy",
)

config = PredictiveConfiguration(
    modalities=[numerical],
)

missing = missing_predictive_fields(config)

The descriptive shape (time, features) does not imply a particular tensor rank beyond the text supplied by the external specification, and it does not imply a lookback length. Similarly, frequency="daily" records an intended external description; it does not cause the package to retrieve or resample daily data.

Turn omissions into an actionable report

missing_predictive_fields returns stable field names for decisions that are absent. For the configuration above, the conceptual result includes fields such as task_type, target_definition, horizon, lookback, split_policy, scaling_policy, model_family, hyperparameters, optimizer, loss, and metrics. The implementation does not infer any of them from the review.

build_missing_specification_report groups those omissions into categories such as task_and_target, temporal_construction, data_preprocessing, model_and_optimization, and evaluation. explain_missing_fields adds plain-language explanations. For instance, it explains that the target must include its value or label construction, while the horizon must include its units. These explanations are derived implementation guidance, not additional paper facts.

contract_report packages the result into a JSON-safe dictionary. Its output distinguishes whether the configuration is validated, which fields are missing, which are supplied, and what the implementation boundary is. The generated function explicitly records that no predictive runtime, equations, or training procedure is supplied by the paper.

A strict caller can use assert_faithful_reproduction_possible as a gate:

def assert_faithful_reproduction_possible(config: PredictiveConfiguration) -> None:
    """Reject a predictive contract that still depends on unspecified external decisions."""
    if not isinstance(config, PredictiveConfiguration):
        raise ValidationError("config must be a PredictiveConfiguration")

    validate_modalities(config.modalities)
    validate_temporal_alignment(config.modalities)
    missing = missing_predictive_fields(config)
    if missing:
        details = "; ".join(_explanation_for_field(field) for field in missing)
        raise IncompleteSpecificationError(
            "faithful predictive reproduction is incomplete; missing fields: "
            + ", ".join(missing)
            + ". "
            + details
        )

IncompleteSpecificationError is therefore an explicit contract response. It does not mean that model training failed; no model-training operation is defined here. It means that the supplied external specification is insufficient for a faithful predictive reproduction.

What this boundary does—and does not—guarantee

The contract validator can ensure that required declarations exist and that modality descriptions are structurally coherent. It cannot determine whether a chosen target is scientifically appropriate, whether a split truly prevents leakage, or whether a selected architecture is effective. Those decisions belong to the external predictive study and must be documented there.

This model-family-agnostic design is deliberate. The paper names incompatible families and does not provide a unified interface for combining them. A future reproduction could supply an SVM with tabular features, an LSTM with sequences, or a multimodal architecture, but each would require details absent from this source. The contract records those details once they are supplied; it does not invent them.

Worked example: stopping before accidental invention

Suppose an external user supplies only the numerical modality shown above. missing_predictive_fields identifies the absent target and horizon along with the rest of the required decisions. contract_report can present those omissions by category, and explain_missing_fields can state why each one matters. A caller may then choose to raise IncompleteSpecificationError through assert_faithful_reproduction_possible rather than proceeding.

The important outcome is not a prediction. It is an auditable boundary: no sequence length, normalization rule, loss, optimizer, or LSTM architecture has been smuggled into the reproduction. Multimodal inputs likewise remain declarations until an external specification defines their alignment and leakage policy.

The planned audit and verification layers were not run under the authoritative policy, so this tutorial makes no claim that the code was executed or that the contract path passed verification. Faithfully reproducing this paper means preserving its review evidence and explicitly exposing its missing predictive details; implementing a predictive model requires a separate, external specification.

Serialization, reporting, and deterministic example data

How do you turn a provenance-rich review catalog into an artifact that another person can inspect without losing uncertainty, dates, or denominators? Use three deliberately separate layers: serialization converts domain objects into JSON-safe data, reporting formats that data for readers, and local command-line tools connect the workflow without pretending to query bibliographic databases or train a stock predictor.

Preserve meaning when converting objects to JSON

Implementation decision. The generated serialization.py module treats serialization as a loss-avoidance step, not as a place to calculate new evidence. record_to_dict converts one supported dataclass, while catalog_to_dict and synthesis_to_dict serialize the two main review artifacts. write_json then writes a deterministic UTF-8 document to a local Path.

The important conversions are explicit:

• Enum members such as Database.SCOPUS are written using their stable string values.

• date and datetime values are written as ISO-8601 strings.

• None remains JSON null, which is essential for an explicitly unspecified included-study count.

• Tuples and other supported sequences become JSON arrays while preserving order.

• Provenance, confidence, subset identifiers, and denominators are ordinary serialized fields rather than comments that could be lost.

The focused implementation excerpt below shows the public catalog boundary. Notice that catalog_to_dict validates the catalog before converting it, but does not repair or infer missing values.

def catalog_to_dict(catalog: ReviewCatalog) -> dict[str, object]:
    """Serialize a review catalog while retaining evidence provenance."""
    if not isinstance(catalog, ReviewCatalog):
        raise ValidationError("catalog must be a ReviewCatalog value")
    catalog.validate()
    converted = _to_json_value(catalog, "catalog")
    if not isinstance(converted, dict):
        raise ValidationError("serialized catalog must be a dictionary")
    return converted


def synthesis_to_dict(report: SynthesisReport) -> dict[str, object]:
    """Serialize a source-qualified synthesis report without recomputation."""
    if not isinstance(report, SynthesisReport):
        raise ValidationError("report must be a SynthesisReport value")
    converted = _to_json_value(report, "synthesis_report")
    if not isinstance(converted, dict):
        raise ValidationError("serialized synthesis report must be a dictionary")
    return converted

This boundary matters for the paper because evidence is source-qualified. A serialized RMSE label must remain a reported metric name, not become a computed value. Likewise, a null field must remain visibly absent rather than being replaced by a guessed count or parameter.

Use schemas to describe, not invent, data

The generated schema.py module provides two JSON-schema-like descriptions. catalog_schema() describes persisted review records, coded evidence, claims, and warnings. predictive_configuration_schema() describes the fields an external predictive reproduction must supply, including task type, target, horizon, modalities, split, scaling, model family, hyperparameters, optimizer, loss, and metrics.

The distinction is important: the predictive schema is a contract for missing information, not a model specification. It does not provide a default LSTM, sequence length, normalization rule, or loss. validate_payload_shape checks required keys and structural types without coercing values or filling gaps.

For example, the catalog schema explicitly permits an included-study count to be an integer or null:

"included_study_count": {"type": ["integer", "null"], "minimum": 0},

That null is meaningful. The paper reports ten retained reviews with the sequence 27, unspecified, 53, 12, 34, 122, 24, 57, 30, and 20. The implementation must preserve the unspecified entry instead of estimating it from the other values.

Render evidence without changing it

Serialization is for machines; reporting is for readers. The generated reporting.py module offers render_summary, render_markdown_report, and render_warning_section. These functions format existing objects and make important boundaries visible:

• render_markdown_report labels the result as a review-evidence synthesis.

• Flow counts remain stage-specific rather than being presented as one total.

• Each dimension summary displays source review identifiers, reported denominators, and uncertainty counts.

• The denominator audit displays each claim’s claim_id, subset_id, denominator, and source location.

• Warnings are rendered as a visible section rather than being hidden in program logs.

The report renderer itself states the scope clearly:

"## Scope and implementation boundary",
"",
"This report represents a systematic review of systematic reviews. Its entries are reported literature evidence, not primary stock-price datasets and not results from a newly trained model.",
"",
"The rendering preserves coded labels, source-review identifiers, uncertainty counts, and denominator metadata. It does not reconstruct equations, calculate metric formulas, combine incompatible study subsets, or report predictive performance.",

This is presentation behavior, not new analysis. For instance, a report may show that a review catalog contains LSTM or MAE, but it does not claim that the package trained an LSTM or evaluated predictions with MAE.

Keep the command-line path local

The generated cli.py module provides a local interface with two input modes: deterministic synthetic examples or a user-supplied local JSON file. build_parser exposes --example, --input, --output, and --format options. load_local_records parses local candidates and extraction batches; it does not contact Scopus, Web of Science, Rayyan, Yahoo Finance, or another external service.

The following invocation is documentation for the available interface, not a claim that it was run here:

python -m review_synthesis.cli --example --output report.json

With --format json, the CLI can include the serialized catalog, flow counts, synthesis report, and missing-specification report. With --format markdown, it uses render_markdown_report; with --format text, it uses render_summary. The generated output should be understood as a local workflow artifact, not as a database export or a predictive experiment.

Worked example: synthetic records to a labeled artifact

The example flow begins with example_candidates() and example_extraction_batches() from src/review_synthesis/example_data.py. These functions intentionally create small, deterministic fixtures. The candidates include records attributed to both databases and a repeated title so that provenance and deduplication can be demonstrated. The extraction batch contains separate review-level fields for methods, information sources, and metrics. example_claims() supplies illustrative claims with a denominator, but its text is explicitly synthetic.

The conceptual pipeline is:

1. Create local candidates and extraction batches.

2. Supply screening decisions to run_review_pipeline.

3. Obtain a PipelineResult containing the catalog, screening log, flow counts, synthesis report, and missing-specification report.

4. Pass the catalog to catalog_to_dict and the synthesis report to synthesis_to_dict.

5. Pass the report and flow counts to render_markdown_report.

6. Write the combined artifact with write_json.

The dedicated scripts/build_example_report.py script follows that path and labels its output with artifact_type: "synthetic_review_synthesis_example". Its explanatory metadata also states that the artifact is not the paper’s bibliographic dataset and contains no trained-model results. This labeling is necessary because deterministic data is useful for teaching the interfaces, but it is not evidence that the paper’s original records were recovered.

A second script, scripts/export_reported_facts.py, serves a different purpose. It exports supplied paper facts such as the article identifier, search metadata, source-specific retrieval counts, stage-specific flow counts, and the ten included-study counts. The unspecified review count is preserved as JSON null, and the export includes the supplied ambiguity and warning lists. It is therefore a facts manifest, not a reconstruction of the original database contents.

What this artifact boundary guarantees—and what it does not

The serialization and reporting layer guarantees only the behavior represented by its contracts: supported objects can be converted without dropping explicit nulls, provenance, confidence, ordering, or denominator metadata; reports can expose those fields; and local examples can be labeled as synthetic. The paper itself supplies no canonical equations, so no equation or metric formula appears in these artifacts.

The implementation also does not claim that the paper prescribes a Python dependency stack. The source mentions tools including Python, TensorFlow, NumPy, Pandas, Keras, Scikit-Learn, TA-Lib, TA4J, and MATLAB, but the generated local interfaces use the package’s own records and standard local file handling rather than treating any named tool as mandatory.

No code execution, test generation, static verification, semantic code verification, tutorial-section verification, or final quality review was performed under the authoritative run policy. The example commands and artifact flow above explain the intended interfaces only. Faithful reproduction of this paper ends with preserving its review evidence and limitations; a stock-prediction runtime would require a separate external specification.

Static and semantic verification strategy—and what was not performed

How can you tell whether this reproduction preserves the paper’s meaning without accidentally claiming that a stock-prediction model exists? Treat verification as a set of explicit invariants, not as a performance test. The relevant promises are that records retain provenance, exclusions retain reasons, reported counts remain separated, uncertain evidence remains uncertain, and generated reports do not claim newly trained models.

This section describes the intended verification strategy and the generated audit hooks. It also states the actual status precisely: verification was skipped under the authoritative run policy. No code execution, test generation, semantic code verification, tutorial-section verification, or final quality review occurred.

What static verification would inspect

Implementation decision. Static verification would inspect the Python package without running the research workflow. For this project, that means parsing each planned Python file with the Python abstract syntax tree, checking imports and public symbols, inspecting dataclass fields and annotations, and comparing public signatures with the planned interfaces.

The same review would inspect the JSON-like structures returned by catalog_schema() and predictive_configuration_schema(). The catalog schema is intended to require review records, coded evidence, claims, and warnings while preserving nullable fields. In particular, None must remain available for an explicitly unspecified study count or denominator rather than being replaced with an estimate. The predictive schema documents required external decisions but does not provide defaults for a target, horizon, architecture, or optimizer.

The generated schema.py module exposes structural validation through validate_payload_shape. Its responsibility is deliberately narrow: check required keys and supported types without coercing missing information.

def validate_payload_shape(payload: Mapping[str, object], schema: Mapping[str, object]) -> None:
    """Validate required keys and supported structural types without coercion."""
    if not isinstance(payload, Mapping):
        raise ValidationError("payload must be a mapping")
    if not isinstance(schema, Mapping):
        raise ValidationError("schema must be a mapping")
    _validate_node(payload, schema, "payload")

Readers should notice the phrase “without coercion.” A structural validator may reject an invalid payload, but it must not repair a missing provenance field, infer a denominator, or turn an uncertain Table 5 association into a reported fact. Those are evidence decisions, not harmless formatting operations.

A future enabled static pass would also check that run_review_pipeline depends on the intended local modules and that PipelineResult contains review artifacts rather than predictions, model weights, losses, or newly computed scores. The orchestration boundary is important: the function consumes externally supplied records and decisions, while live database retrieval and model fitting remain outside this paper-faithful implementation.

What semantic verification would inspect

Derived explanation. Semantic verification asks whether the code’s behavior matches the paper’s scope and the implementation contracts. It is more than checking that names and types look plausible. It would inspect whether every extracted item has a source review and location, whether each exclusion has a screening stage and reason, whether counts remain tied to their workflow stages, and whether claims preserve their original subset and denominator.

The generated audit.py module provides three focused audit entry points:

• audit_catalog inspects cross-record references, provenance, uncertainty, denominator preservation, flow counts, and report scope.

• audit_reported_counts compares supplied flow values with the paper’s reported stage-specific values without modifying them.

• audit_no_model_claims searches generated qualitative text and warnings for unsupported language suggesting that a model was trained or evaluated.

The last check is a safeguard against scope drift. It is not evidence that a model exists or that the audit found a valid model implementation. Its purpose is to catch wording that would contradict the paper-faithful boundary.

def audit_no_model_claims(report: SynthesisReport) -> list[AuditFinding]:
    """Detect unsupported claims that this package trained or evaluated a model."""

    if not isinstance(report, SynthesisReport):
        return [
            _finding(
                "error",
                "report_type",
                "report must be a SynthesisReport instance",
            )
        ]

    text_parts: list[str] = []
    text_parts.extend(report.qualitative_findings)
    text_parts.extend(report.warnings)
    text = "\n".join(text_parts).casefold()
    unsupported_phrases = (
        "trained a model",
        "model was trained",
        "achieved accuracy",
        "predicted prices",
        "generated predictions",
        "fitted model",
        "test-set performance",
        "validation score",
    )

This audit is especially relevant here because the paper names SVM, LSTM, neural networks, ARIMA, and many other methods. A report that says one of these methods achieved a result would exceed the supplied evidence unless that statement were explicitly attributed to a reviewed study and retained with its source context.

A hypothetical audit walkthrough

Consider a catalog containing one retained review and one uncertain SVM observation extracted from fragmented Table 5 material. A semantic audit would conceptually follow this checklist:

1. Confirm that the evidence’s source_review_id refers to a retained review or an explicitly external source.

2. Confirm that its Provenance.source_location identifies the relevant table or section.

3. Confirm that its confidence remains UNCERTAIN when row alignment was not established.

4. Confirm that any quantitative EvidenceClaim retains its subset_id and denominator.

5. Compare the supplied FlowCounts with the reported values: 69 retrieved, 43 reviews read, 17 remaining after abstract screening, 16 after duplicate or same-author handling, and 10 finally retained.

6. Inspect the synthesis text for unsupported claims about training, predictions, or test-set performance.

The constants used for this comparison are source-qualified facts rather than inferred prevalence estimates. REPORTED_RETRIEVAL_COUNTS keeps Scopus at 40 and Web of Science at 29, while REPORTED_FLOW_COUNTS keeps the PRISMA-style values separate by stage. Neither mapping proves that a supplied local fixture reproduces the original database export.

REPORTED_RETRIEVAL_COUNTS: Final[Mapping[str, int]] = MappingProxyType(
    {
        "Scopus": 40,
        "Web of Science": 29,
    }
)

If a local example contains different counts, the intended behavior is to report a difference, not to overwrite the local data or claim reconciliation. Similarly, a mismatch does not by itself show that the paper is incorrect; it may simply mean that the local records are synthetic or incomplete.

Actual verification status for this tutorial

Run-policy fact. The authoritative policy disabled local static verification, test generation, code execution, semantic code verification, tutorial-section verification, and final quality review. The supplied verification records therefore have a skipped status. This tutorial does not claim that the generated files parsed successfully, that schemas validated real artifacts, that audits returned no findings, or that any example command ran.

The audit functions are planned safeguards and generated interfaces, not completed verification results. The same distinction applies to PipelineResult: its type describes what run_review_pipeline would return for supplied local inputs, but no execution outcome is asserted here.

Important limitations remain even if these checks are enabled later. Table 5 is fragmented, so some method, source, and metric associations require manual source-PDF review. The paper’s geographic inclusion criterion and some review-subset statements are ambiguous. No canonical equations, target definition, forecast horizon, preprocessing policy, architecture, hyperparameters, loss, optimizer, or training schedule were supplied. The local examples are synthetic and must not be presented as the original Scopus or Web of Science records.

The faithful boundary is therefore concise: reproducing this paper means reproducing its review-selection, provenance, uncertainty, and evidence-synthesis workflow. Any executable stock-prediction model requires a separate external specification, and its correctness cannot be established from this review alone.

Use the button or URL below to download the source code.

Read more

Use the URL at the end of this article to download the source code

Implementation Assumptions

• The Section 4/Table 1 80/20 chronological split is the primary reproduction choice; the Section 3 75/25 recommendation is retained as a documented conflict.

• Yahoo Finance access is optional at runtime; local CSV fixtures and deterministic synthetic data support offline demonstrations.

• Ticker symbols, selected features, target column, scaler, look-back window, validation protocol, seeds, hyperparameter selections, and ensemble wiring remain explicit configuration choices because the paper does not specify them.

• The LSTM is implemented with TensorFlow/Keras rather than a hand-written recurrent cell because equations eq8, eq9, and eq_10 are incomplete.

• XG-Boost is accessed through an optional local Python dependency, with a clear compatibility boundary and no network-dependent behavior.

• Canonical equation LaTeX is stored only from the supplied records; empty or damaged equation records are not reconstructed.

• Table 2 values are stored as reported references and are never represented as independently verified results.

• All data transformations that learn parameters are fitted on training data only.

• The implementation is stock-specific by default, with separate preprocessing and model fitting for each security.

• No live API call, model training, or code execution is claimed by this plan.

Scope, evidence status, and reproduction target

What does it mean to reproduce this paper when the paper does not include the original source code? The practical answer is to reproduce the documented experimental structure while making every missing choice visible. This project therefore aims to be auditable and configurable, not to claim an exact numerical reconstruction of the paper's experiment.

The paper is primarily a systematic review of stock-price forecasting and trend-classification methods. Its main implementation comparison, in Section 4, contains six individual regressors or forecasting models—SVR, MLPR, KNN, random forest, XG-Boost, and LSTM—plus a proposed Random Forest + XG-Boost + LSTM ensemble. A regression model predicts a numeric value, here a stock-price target. A baseline is an individual model used as a comparison point for the ensemble.

The two reported securities are TAINIWALCHM, described as Tainwala Chemicals and Plastics, and AGROPHOS, described as Agro Phos. The paper reports historical data obtained through Yahoo Finance and gives date ranges for these names, but it does not provide the exact Yahoo Finance ticker identifiers needed to make an unambiguous download request.

What is specified, and what is not

The strongest implementation evidence comes from Table 1. It specifies an 80% training and 20% testing split for the proposed ensemble, MSE loss and Adam for the neural component, a maximum of 50 epochs, batch size 32, two LSTM layers, dropout of 0.2, and a dense layer with 25 units. It also lists candidate values for random-forest and XG-Boost hyperparameters.

This conflicts with the paper's more general pipeline description, which mentions a 75/25 split. The framework uses the Table 1 80/20 split as its primary reproduction choice because it is the more specific statement about the proposed ensemble. The conflict remains recorded rather than silently discarded.

Several choices needed for an executable experiment are absent: the exact ticker symbols, selected features, target column, scaling method, LSTM look-back window, forecast horizon, validation procedure, random seeds, selected hyperparameters, and tuning algorithm. The paper also calls the combination “stacked” without defining its wiring. Stacking usually means fitting a second-stage model on component predictions, whereas blending or weighted averaging combines predictions directly, often with fixed weights. These are different operations, so the implementation exposes the distinction instead of presenting one as recovered fact.

The generated configuration objects make these decisions explicit. The following excerpt is copied from src/stock_forecasting/config.py; notice that feature columns, target, split fraction, sequence settings, and scaler are all fields rather than hidden constants.

@dataclass(frozen=True, slots=True)
class PreprocessingConfig:
    """Make feature, target, split, sequence, and scaling decisions explicit.

    The paper's Table 1 specifies an 80/20 split for the proposed ensemble,
    while its generic pipeline mentions 75/25.  The default-free configuration
    requires the caller to choose a fraction; a value of ``0.8`` is the
    primary reproduction choice described by the paper.  The target column,
    look-back, horizon, feature set, and scaler are not specified by the paper.
    """

    feature_columns: tuple[str, ...]
    target_column: str
    train_fraction: float
    lookback: int
    horizon: int
    scaler_name: Optional[str]

Here, feature_columns names the input columns, while target_column identifies the value to forecast. The paper mentions possible OHLCV and secondary data but does not establish the final feature set or target. lookback is the number of historical rows supplied to an LSTM sequence, and horizon describes how far ahead its target is aligned. Both are implementation decisions. scaler_name records whether a transformation such as standard or min-max scaling is used; the paper does not specify which one.

The manifest as an audit boundary

The experiment manifest separates paper facts from implementation decisions and unresolved ambiguities. ExperimentManifest records the stock identity, source configuration, feature and target choices, split policy, sequence settings, tuning description, model configurations, and ensemble combiner. validate_manifest checks that these fields are sufficiently explicit and serializable.

The generated class exposes this purpose directly through its fields:

@dataclass(frozen=True, slots=True)
class ExperimentManifest:
    """Audit record for one stock-specific reproduction configuration.

    The manifest deliberately stores unresolved paper details as explicit fields or
    notes rather than hiding them in defaults.  ``model_configurations`` records
    selected settings when supplied; an empty mapping means that no selected model
    settings were provided by the caller or recovered from the paper.
    """

    paper_id: str
    stock: str
    data_source: Mapping[str, Any]
    feature_columns: tuple[str, ...]
    target_column: str
    train_fraction: float
    split_policy: str
    scaler_name: str | None
    lookback: int
    horizon: int
    seed: int | None
    tuning_method: str
    validation_fraction: float | None
    model_configurations: Mapping[str, Any]
    combiner: str
    ensemble_weights: tuple[float, ...] | None
    meta_features: str
    package_metadata: Mapping[str, Any]

A small worked configuration would therefore identify the stock label separately from the source ticker, choose features such as Open, High, Low, and Volume, choose Close as the target, record an 80/20 chronological split, select a look-back and scaler, and state whether tuning is unspecified or explicitly configured. It would also record a seed and a combiner such as weighted_average or linear_stacking. Those values would document the reproduction run; they would not become claims about what the paper originally did.

For example, the public configuration contract requires an explicit data request rather than silently selecting a ticker:

@dataclass(frozen=True, slots=True)
class DataSourceConfig:
    """Describe one explicit historical-data request.

    ``ticker`` is deliberately required from the caller because the paper does
    not provide the exact Yahoo Finance symbols for TAINIWALCHM or AGROPHOS.
    Date endpoint inclusivity and interval semantics remain properties of the
    selected Yahoo Finance client.
    """

    ticker: str
    start: str
    end: str
    interval: str

The source ticker, date strings, and interval are provenance, not merely convenience arguments. They make it possible to distinguish two runs that use different Yahoo Finance symbols, endpoint behavior, or sampling frequencies.

What the reported results mean

The paper's Table 2 reports RMSE and R2 for each stock and model. RMSE, or root mean square error, summarizes prediction error in the target's units; smaller values indicate smaller typical squared errors. R2, or the coefficient of determination, compares the predictions with a constant-mean reference. It can be negative for a poor regression model and is not required to remain between zero and one.

The paper reports the RF + XG-Boost + LSTM ensemble as having the strongest comparison values: RMSE 2.0247 and R2 0.9921 for TAINIWALCHM, and RMSE 1.2658 and R2 0.9897 for AGROPHOS. These are reported references only. The generated code has not been executed, no Yahoo Finance request has been made, and no model training, test run, syntax check, or independent numerical verification has occurred. Consequently, this tutorial must not describe those values as reproduced results.

One additional fidelity issue is retained rather than corrected: the supplied paper discussion refers to the AGROPHOS date range in one sentence using the TAINIWALCHM name. The framework records the source and stock explicitly so that a reader can resolve this issue deliberately when supplying data.

In the rest of the tutorial, fit_models_and_generate_predictions will be treated as the training-and-prediction procedure, stacked_rf_xgboost_lstm as the configurable ensemble method, and evaluate_regression_predictions as the held-out evaluation step. A held-out test set is data reserved until model fitting and model selection are complete. Keeping it separate is a derived implementation safeguard, not evidence that the original paper used exactly the same procedure.

The central reproduction principle is therefore simple: preserve the paper's stated settings, expose every missing setting, and label reported metrics as references until an independently specified and executed experiment supports a comparison.

Project layout and public contracts

How can one keep a stock-forecasting reproduction understandable when it contains data cleaning, several model families, an LSTM, an ensemble, and multiple reporting steps? The practical answer is to give each stage one responsibility and connect stages with explicit contracts. In this project, data modules obtain and validate rows, preprocessing modules construct examples, model adapters fit estimators, ensemble modules align and combine predictions, and metric modules evaluate the final outputs.

This separation is an implementation design derived from the paper's pipeline description. The paper says that data acquisition, preprocessing, model training, evaluation, and tuning are distinct stages, but it does not prescribe a Python package layout. The generated package makes that progression visible through modules under data, models, ensemble, training, metrics, and experiments.

A map of the package

The lightweight package entry point is src/stock_forecasting/__init__.py. It exports configuration types but does not import TensorFlow, XG-Boost, or Yahoo Finance integrations eagerly. This matters because configuration and local data work should remain available even when optional modeling or network dependencies are not installed.

The dependency boundary is declared in pyproject.toml. The core installation contains NumPy, pandas, and scikit-learn, while deep learning, Yahoo Finance access, XG-Boost, plotting, and development tools are optional extras:

[project.optional-dependencies]
# TensorFlow supplies the Keras LSTM implementation described in Table 1.
deep-learning = [
    "tensorflow>=2.13",
]
# Yahoo Finance access is opt-in; local CSV and synthetic workflows remain offline.
yahoo = [
    "yfinance>=0.2.30",
]
# XG-Boost is optional because the paper does not specify its package or version.
xgboost = [
    "xgboost>=1.7",
]
plotting = [
    "matplotlib>=3.7",
]

This is not a claim about the paper's original package versions. It is a reproducibility decision for the generated framework. In particular, a local CSV or synthetic-data demonstration should not require a network client, and importing configuration should not fail merely because TensorFlow is unavailable.

The main hand-off records are defined in src/stock_forecasting/types.py. They carry arrays together with timestamps and validate their basic shape and finiteness requirements. This is important for the methods preprocess_time_series, fit_models_and_generate_predictions, stacked_rf_xgboost_lstm, and evaluate_regression_predictions: each method depends on receiving data with the intended meaning, not merely an object that happens to be an array.

Two representations of the same forecasting problem

Conventional regressors use a tabular feature matrix, which we will call X_tab. Its shape is rank 2:

• (n_samples, n_features) means one independent feature row per training or evaluation example.

• y is the aligned numeric target, usually represented as (n_samples,) or (n_samples, 1).

The LSTM instead uses a sequence tensor, which we will call X_seq. Its shape is rank 3:

• (n_sequences, lookback, n_features) means each example contains an ordered window of lookback observations.

• The corresponding y contains one future target for each sequence.

The distinction is semantic. A tabular row says, in effect, “use these features for this example.” A sequence window says, “use this ordered history to predict the associated future target.” The generated TabularSplit and SequenceSplit records enforce these different contracts rather than treating every input as a generic two-dimensional matrix.

A focused portion of TabularSplit shows the tabular contract and its alignment checks:

@dataclass
class TabularSplit:
    """Chronological train/test arrays for conventional regressors.

    ``X_train`` and ``X_test`` have shape ``(n_samples, n_features)``. Targets
    may have shape ``(n_samples,)`` or ``(n_samples, 1)`` and must align with
    their corresponding timestamp indices.
    """

    X_train: np.ndarray
    X_test: np.ndarray
    y_train: np.ndarray
    y_test: np.ndarray
    train_index: pd.Index
    test_index: pd.Index

    def __post_init__(self) -> None:
        x_train = _as_numeric_array(self.X_train, "X_train")
        x_test = _as_numeric_array(self.X_test, "X_test")
        if x_train.ndim != 2 or x_test.ndim != 2:
            raise ValueError(
                "X_train and X_test must both have rank 2 "
                "(n_samples, n_features)"
            )
        if x_train.shape[1] != x_test.shape[1]:
            raise ValueError("X_train and X_test must have the same feature count")

The record also checks that both partitions are nonempty, that target lengths match sample counts, that timestamp indices have the expected lengths, and that training and test indices do not overlap. Those checks protect the chronological split required by the reproduction plan. They do not determine the split ratio; the surrounding configuration makes the Table 1 choice of 80/20 explicit while documenting the paper's conflicting generic 75/25 recommendation.

The sequence counterpart uses the same idea with one additional axis:

@dataclass
class SequenceSplit:
    """Chronological train/test windows for recurrent models.

    Each feature tensor has shape ``(n_sequences, lookback, n_features)``. The
    target timestamp denotes the future observation associated with its window;
    it therefore has one label per sequence rather than one label per timestep.
    """

    X_train: np.ndarray
    X_test: np.ndarray
    y_train: np.ndarray
    y_test: np.ndarray
    train_index: pd.Index
    test_index: pd.Index

    def __post_init__(self) -> None:
        x_train = _as_numeric_array(self.X_train, "X_train")
        x_test = _as_numeric_array(self.X_test, "X_test")
        if x_train.ndim != 3 or x_test.ndim != 3:
            raise ValueError(
                "X_train and X_test must both have rank 3 "
                "(n_sequences, lookback, n_features)"
            )
        if x_train.shape[1:] != x_test.shape[1:]:
            raise ValueError(
                "train and test sequence tensors must share lookback and feature dimensions"
            )

Here x_train.shape[1:] represents (lookback, n_features). The two partitions must agree on those dimensions even though they contain different numbers of sequences. The sequence indices identify target timestamps, not every timestamp inside each window. This distinction becomes essential later when RF and XG-Boost predictions are aligned with LSTM predictions: forming windows can remove early rows, so component predictions cannot be combined by position alone.

Prediction and preprocessing provenance

PreprocessingState stores fitted transformation information such as the feature scaler, target scaler, scaler name, feature count, and number of rows used for fitting. The paper mentions scaling but does not specify the scaler type, fit scope, or reporting scale. The generated record therefore preserves those choices instead of hiding them inside a model adapter. A training-only fit scope is a leakage-control requirement: a scaler must not learn distribution information from the held-out period.

PredictionBundle couples a model name, a pandas index, and one prediction per index label. Its contract accepts values shaped (n,) or (n, 1), but requires the number of values to equal the number of timestamps and rejects non-finite values. This is more than defensive programming. A prediction without its timestamp is not enough for the ensemble method stacked_rf_xgboost_lstm, because the RF, XG-Boost, and LSTM outputs must refer to the same target observations before they are combined.

MetricRecord performs the final identity bookkeeping: it stores the stock, model, RMSE, R2, and reporting scale. The metric value alone is insufficient for an experiment comparing two securities and several models. A record must say what was measured and on which scale.

A common tabular model interface

The conventional models share a small protocol in src/stock_forecasting/models/base.py. RegressorProtocol requires fit and predict; validate_regression_arrays checks the rank, numeric type, finiteness, and sample alignment of tabular data:

@runtime_checkable
class RegressorProtocol(Protocol):
    """Protocol implemented by tabular regression model adapters.

    Implementations consume a rank-2 feature matrix with shape
    ``(n_samples, n_features)`` and a one-dimensional target vector with shape
    ``(n_samples,)``.  ``fit`` returns the fitted estimator to support the
    conventional ``model.fit(...).predict(...)`` usage.
    """

    def fit(self, X: np.ndarray, y: np.ndarray) -> RegressorProtocol:
        """Fit the regressor and return the fitted model."""

    def predict(self, X: np.ndarray) -> np.ndarray:
        """Return one continuous prediction for each input row."""

This contract allows the SVR, MLPR, KNN, random-forest, and XG-Boost adapters to be orchestrated consistently while keeping their library-specific settings local. It does not make the models equivalent: their kernels, neural architectures, neighbor rules, tree construction, and boosting behavior remain different. It simply ensures that the training runner can request a fit and then obtain one prediction per evaluation row.

Worked example: tracing two batches

Suppose preprocessing produces 240 tabular training rows with four selected features. The tabular contract is X_train.shape == (240, 4) and y_train.shape == (240,). A conventional model can fit this representation directly and return, for example, one prediction for every held-out timestamp.

Suppose the same feature stream is converted into windows with a look-back of 20. The recurrent representation is no longer (240, 4). It is shaped like (n_sequences, 20, 4), and each sequence target has a timestamp after its input window. The exact n_sequences depends on the configured horizon and the rows available after splitting; the paper does not specify the look-back or horizon, so these remain explicit implementation decisions.

In both cases, the index is part of the contract. A tabular prediction bundle might contain one value for each test timestamp. A sequence prediction bundle may begin later because its first valid target requires a complete history. Before ensemble combination, the alignment layer must intersect and order these timestamp sets. The resulting component matrix has three columns—one each for random forest, XG-Boost, and LSTM—and one row per common target timestamp.

The planned tests describe these shape, time-order, and alignment invariants, but they are not evidence of completed checks. Under the run policy, local static verification, semantic code verification, test generation, and code execution were disabled. Thus the package layout and contracts presented here explain the intended implementation boundaries; they do not claim that the generated code ran or that any Table 2 result was reproduced.

Data acquisition and provenance

How can a stock-forecasting experiment be reproduced if the paper names securities but does not provide the exact download identifiers? Treat the data request itself as part of the experiment. A model input is not merely “TAINIWALCHM data”; it is a specific ticker, start date, end date, interval, returned column set, and timestamp policy.

The paper identifies Yahoo Finance as the source for the two reported securities. It describes TAINIWALCHM as covering 2014–2023 and AGROPHOS as covering 2018–2023, subject to ticker availability and API behavior. However, it does not supply the exact Yahoo Finance ticker symbols, endpoint inclusivity, interval semantics, client version, or complete returned schema. One passage also incorrectly associates the AGROPHOS period with TAINIWALCHM. The implementation records these facts as unresolved configuration rather than silently guessing.

Make the request explicit

The generated DataSourceConfig groups the four essential request fields. Here, ticker identifies the security, start and end delimit the requested historical period, and interval selects the sampling frequency, such as daily data. These values are validated before a client is allowed to make a request.

@dataclass(frozen=True, slots=True)
class DataSourceConfig:
    """Describe one explicit historical-data request.

    ``ticker`` is deliberately required from the caller because the paper does
    not provide the exact Yahoo Finance symbols for TAINIWALCHM or AGROPHOS.
    Date endpoint inclusivity and interval semantics remain properties of the
    selected Yahoo Finance client.
    """

    ticker: str
    start: str
    end: str
    interval: str

This is an implementation contract, not an equation or a setting recovered from the paper. It prevents a convenience default from selecting an unintended security. The configuration also rejects an empty ticker, malformed ISO date strings, and an end value that is not later than the start value. The exact meaning of the end boundary still belongs to the selected client and should be recorded with the snapshot.

Yahoo Finance as an opt-in boundary

YahooFinanceClient.download_historical_data is responsible for one explicit request. It imports the optional yfinance dependency only when the method is called, downloads one ticker, rejects an unavailable or malformed response, normalizes the timestamp index, and attaches provenance. The public convenience function download_historical_data delegates to this client; neither function substitutes a ticker.

A focused excerpt shows the request boundary and several deliberate choices:

downloaded = yf.download(
    tickers=config.ticker,
    start=config.start,
    end=config.end,
    interval=config.interval,
    auto_adjust=False,
    progress=False,
    threads=False,
)

The paper specifies Yahoo Finance as the source but does not specify auto_adjust, progress behavior, threading, or the client API. Consequently, auto_adjust=False and the other arguments are implementation decisions that must be preserved in the provenance record if the resulting data is used. They are not claims about the original experiment.

The client then stores the request metadata on the returned timestamp-indexed DataFrame:

provenance: dict[str, Any] = {
    "source": "Yahoo Finance",
    "client": "yfinance",
    "ticker": config.ticker,
    "start": config.start,
    "end": config.end,
    "interval": config.interval,
    "auto_adjust": False,
}
frame.attrs["yahoo_finance_provenance"] = provenance
return frame

A timestamp-indexed DataFrame has one observation per row and a DatetimeIndex identifying when that observation occurred. Chronological order is an invariant: the returned index must be increasing, and duplicate timestamps are rejected rather than merged with an undocumented rule. This matters because later train/test splitting and sequence construction depend on temporal order. A successful API response is not automatically valid experimental input; an empty result, ambiguous multi-ticker response, malformed timestamp, or duplicate timestamp raises YahooFinanceAcquisitionError.

The acquisition wrapper also does not claim to resolve market-data subtleties. Missing trading days may be normal rather than errors, while corporate-action adjustments, timezone conversion, endpoint inclusivity, and Yahoo Finance client versions can change the resulting table. Those choices belong in the data snapshot and its manifest.

Offline alternatives: CSV and synthetic data

The repository provides two offline sources. load_stock_csv reads a user-supplied snapshot, parses a named date column, rejects unparseable or duplicate timestamps, and returns a sorted DataFrame. This is useful when a Yahoo Finance snapshot has already been obtained, but CSV loading is an implementation addition; it is not the paper's reported acquisition procedure.

from pathlib import Path

from stock_forecasting.data.csv import load_stock_csv

raw = load_stock_csv(Path("data/sample_stock.csv"), "Date")

The CSV loader deliberately stops at structural loading. Feature selection, null handling, zero-value policy, and target selection remain downstream decisions. This keeps the source adapter from silently changing the observations before the configured preprocessing stage sees them.

For a fully offline demonstration, make_synthetic_stock_data creates deterministic OHLCV-shaped data. It returns the columns Open, High, Low, Close, Adj Close, and Volume with a unique chronological business-day index. The generator validates positive finite values and basic OHLC ordering before returning.

from stock_forecasting.data.synthetic import make_synthetic_stock_data

raw = make_synthetic_stock_data(300, seed=7)

This worked example provides 300 rows with a repeatable random seed. It is suitable for demonstrating data flow, configuration, and shape contracts, but it cannot reproduce the paper's Yahoo Finance observations or Table 2 metrics. The generated price process, start date, business-day frequency, and initial value are all synthetic implementation choices.

The same workflow with a real request

Once an exact user-supplied ticker is known, the equivalent explicit configuration is:

from stock_forecasting.config import DataSourceConfig
from stock_forecasting.data.yahoo import YahooFinanceClient

request = DataSourceConfig(
    ticker="EXPLICIT_TICKER",
    start="2014-01-01",
    end="2023-12-31",
    interval="1d",
)
raw = YahooFinanceClient().download_historical_data(request)

EXPLICIT_TICKER is intentionally a placeholder in this tutorial example: the supplied paper context does not identify the correct Yahoo Finance symbol, and substituting one would be an unsupported factual claim. For an AGROPHOS request, the caller would likewise provide the verified ticker and the reported 2018–2023 period. The data source, request values, returned columns, and row count should be saved alongside the CSV rather than inferred later.

The repository's scripts/download_yahoo_snapshot.py provides that opt-in command-line boundary. It requires --ticker, --start, and --end, so a download cannot accidentally use a hidden security default:

parser.add_argument(
    "--ticker",
    required=True,
    help="Exact Yahoo Finance ticker identifier; no default is provided.",
)

After acquisition, the script writes a CSV and a JSON sidecar containing the request and retrieval metadata. It checks that the frame is nonempty, chronological, and free of duplicate timestamps before writing. A typical command therefore has to make the unresolved data decision visible:

python scripts/download_yahoo_snapshot.py \
    --ticker EXPLICIT_TICKER \
    --start 2014-01-01 \
    --end 2023-12-31 \
    --interval 1d \
    --output data/yahoo_snapshot.csv

This command is an interface example, not a report that a download occurred. Under the current run policy, no API call, local execution, or data validation was performed.

What provenance protects

Provenance is the source configuration attached to a loaded dataset: at minimum, the source name, client, ticker, dates, interval, and relevant adjustment settings. It allows a later reader to distinguish a changed download from a changed model. It also exposes why exact numerical reproduction is unavailable here: the paper omits the exact ticker symbols and several data-handling details, and no original snapshot is included.

The acquisition stage therefore establishes only a trustworthy boundary. It does not decide whether the target is Close or Adj Close, which OHLCV columns become features, how nulls or zero values are handled, or how sequences are formed. Those decisions belong to the next preprocessing stage and must remain explicit. Synthetic and CSV inputs make the package usable offline, while the Yahoo Finance path preserves the paper's stated source when the required identifier and optional dependency are supplied.

Cleaning, feature selection, splitting, and scaling

How do we turn a timestamped stock table into fair training examples without allowing information from the future to influence the past? The key is to separate four responsibilities: clean the raw rows, choose the inputs and target explicitly, split observations chronologically, and fit learned transformations only on the training portion.

This section implements the preprocess_time_series method through prepare_stock_data in src/stock_forecasting/preprocessing.py. The paper mentions OHLCV data and possible secondary inputs, but it does not state the exact feature set. It also does not identify whether the target is Close, Adjusted Close, or another value. Those choices therefore belong in PreprocessingConfig, rather than being inferred from the review text.

1. Validate and clean the time series

The generated pipeline expects a pandas DataFrame with a unique, ascending DatetimeIndex. Each row represents one observation, such as one trading day. A timestamp index is important because a positional split is meaningful only when the rows are already in chronological order.

The paper's general pipeline calls for removing redundancies and null values. The implementation makes those operations deterministic: clean_stock_table sorts the index, keeps the last row for a duplicate timestamp, and removes rows containing null values. It also accepts a remove_zero_values argument. In the composed pipeline, that argument is explicitly set to False, because the paper does not say which zero values are invalid. A zero volume, for example, may have a different interpretation from a zero price.

The core cleaning excerpt is:

cleaned = frame.copy(deep=True).sort_index()
cleaned = cleaned.loc[~cleaned.index.duplicated(keep="last")]
cleaned = cleaned.dropna(axis=0, how="any")

numeric_columns = [
    column for column in cleaned.columns if is_numeric_dtype(cleaned[column].dtype)
]
if remove_zero_values and numeric_columns:
    zero_rows = (cleaned.loc[:, numeric_columns] == 0).any(axis=1)
    cleaned = cleaned.loc[~zero_rows]

This excerpt is from src/stock_forecasting/data/validation.py, in clean_stock_table. Notice that the function does not silently drop a target column and does not decide that every zero is an error. After cleaning, it checks that the result is nonempty, chronologically ordered, unique by timestamp, and finite in its numeric columns.

validate_stock_table performs the stricter downstream check. It verifies that required columns exist, are numeric, contain no nulls, and contain only finite values. Its required_columns argument is normally the union of the configured features and target. A malformed table fails early rather than producing a confusing model error later.

2. Select X and y explicitly

Let X denote the feature table and y denote the regression target. In the tabular representation, X has shape (n_observations, n_features), while y has shape (n_observations,). Their timestamp indices must be identical so that each feature row refers to the same target row.

The generated select_features_and_target function does not guess columns from names such as “OHLCV.” Instead, it receives a PreprocessingConfig containing feature_columns and target_column. This is an implementation decision forced by an omission in the paper: OHLCV and secondary data are discussed, but the Section 4 feature set is not specified.

A focused part of the function is:

required_columns = config.feature_columns + (config.target_column,)
# Boundary validation enforces the timestamp, null, numeric, and finite
# value invariants before the configured projection is constructed.
validate_stock_table(frame, required_columns)

if config.target_column in config.feature_columns:
    raise ValueError("target_column must not also be a feature column")

features = frame.loc[:, list(config.feature_columns)].astype(
    np.float64, copy=True
)
target = frame.loc[:, config.target_column].astype(np.float64, copy=True)

This excerpt comes from src/stock_forecasting/data/features.py. The check preventing the target from also appearing as a feature is a leakage safeguard: including the value being predicted among the inputs would make the supervised task ill-defined.

For the worked example, choose the following configuration explicitly:

feature_columns = ("Open", "High", "Low", "Volume")
target_column = "Close"
lookback = 20
horizon = 1
scaler_name = "standard"
train_fraction = 0.8

These are not recovered facts about the paper's original experiment. They are illustrative reproduction decisions. The target could instead be Adjusted Close, and the paper does not establish which one was used. Likewise, the look-back and horizon are absent from the paper.

3. Split chronologically

The primary reproduction choice is an 80/20 chronological split because Table 1 specifies 80% training and 20% testing for the proposed ensemble. The paper's generic pipeline separately mentions 75/25, so the code does not hide that conflict: callers pass train_fraction explicitly.

chronological_split first checks that X and y are aligned, nonempty, unique, and already sorted. It then slices by position without shuffling:

boundary = int(len(X) * fraction)
if boundary <= 0 or boundary >= len(X):
    raise ValueError(
        "train_fraction produces an empty training or test partition; "
        "provide more observations or a different fraction"
    )

X_train = X.iloc[:boundary].copy()
X_test = X.iloc[boundary:].copy()
y_train = y.iloc[:boundary].copy()
y_test = y.iloc[boundary:].copy()

This is the implementation in src/stock_forecasting/data/splitting.py. The resulting X_train and y_train contain earlier observations, while X_test and y_test contain later observations. The function also checks that the final training timestamp precedes the first test timestamp.

A validation split, when tuning is enabled, is made only inside the training portion by chronological_train_validation_split. Its API does not accept test data. That boundary is deliberate: a test target must not influence feature transformation, hyperparameter selection, or model construction.

4. Fit scaling state on training rows only

Scaling changes the numerical representation of features. With standard scaling, for example, a feature is centered and rescaled using statistics estimated from data. The exact scaler type and metric scale are not specified by the paper, so the generated code supports None, "standard", and "minmax" as explicit choices.

The important invariant is not the particular scaler; it is the fitting scope. fit_scalers receives X_train and y_train, fits the feature and target transformations there, and returns a PreprocessingState. The test rows are transformed later using that already-fitted state:

state = fit_scalers(X_train, y_train, config.scaler_name)
X_train_scaled = transform_features(state, X_train)
X_test_scaled = transform_features(state, X_test)
y_train_scaled = transform_target(state, y_train).reshape(-1)
y_test_scaled = transform_target(state, y_test).reshape(-1)

This excerpt is from src/stock_forecasting/preprocessing.py. Fitting a scaler on the complete dataset would allow the distribution of the later test period to affect the representation of earlier training data. That is a form of temporal leakage, even though the target values are not directly copied into the features.

If predictions are reported in the original price units, inverse_transform_target can undo the target transformation. The code exposes that operation rather than assuming whether the paper's RMSE and R2 were computed on scaled or original values. Predictions and targets must always be compared on the same scale.

5. Construct LSTM windows

Tabular models consume independent-looking rows. An LSTM instead consumes ordered windows. The sequence representation, called X_seq here, has shape (n_sequences, lookback, n_features). Each sequence contains lookback historical rows, and its associated target is located horizon steps after the end of that window.

For example, with lookback=20 and horizon=1, the first window contains 20 observations and predicts the immediately following target. The initial rows that cannot form a complete window are therefore not represented as sequence examples. This is why sequence predictions generally have fewer timestamps than tabular predictions.

The generated constructor documents and enforces that relationship:

sequence_features[sequence_number] = features[
    sequence_number : sequence_number + lookback
]
sequence_targets[sequence_number] = targets[target_position]
target_labels.append(ordered_timestamps[target_position])

This excerpt comes from src/stock_forecasting/data/sequences.py. target_position is computed after the input window, so the window does not contain observations from after its target. The target timestamps are retained in SequenceSplit.train_index and SequenceSplit.test_index; those labels are needed later when RF, XG-Boost, and LSTM predictions are aligned.

The generated make_sequences function uses the Table 1 80/20 ratio because its public signature has no split-ratio argument. In the composed prepare_stock_data pipeline, the constructed windows are re-partitioned using the configured train_fraction, so the preprocessing configuration remains the authoritative record of the split decision.

6. Compose the preparation pipeline

prepare_stock_data connects the stages in this order:

1. Clean the timestamp-indexed table while retaining zero values by explicit policy.

2. Select the configured features and target.

3. Split tabular observations chronologically.

4. Fit optional feature and target scalers on training rows only.

5. Transform training and test tabular arrays with that state.

6. Construct ordered sequence windows from the transformed series.

7. Return both representations, their indices, the preprocessing state, and an audit dictionary.

The result is a PreparedStockData record. Its tabular field contains rank-2 arrays for classical regressors, and its sequences field contains rank-3 arrays for the LSTM. The audit dictionary records choices such as feature columns, target column, split fraction, look-back, horizon, scaler, zero policy, and the paper's 80/20 versus 75/25 conflict.

A complete illustrative setup is therefore conceptually:

config = PreprocessingConfig(
    feature_columns=("Open", "High", "Low", "Volume"),
    target_column="Close",
    train_fraction=0.8,
    lookback=20,
    horizon=1,
    scaler_name="standard",
)
prepared = prepare_stock_data(raw_frame, config)

print(prepared.tabular.X_train.shape)
print(prepared.sequences.X_train.shape)
print(prepared.audit["target_column"])

The exact PreprocessingConfig constructor and PreparedStockData record are defined by the generated package; the excerpt illustrates the public data flow and the explicit decisions it must carry. For a sufficiently long input, the first printed shape is rank 2, while the second is rank 3 with its middle dimension equal to 20 and its final dimension equal to the number of configured features.

The main lesson is that preprocessing is part of the model specification. The paper does not provide enough detail to recover its original target, features, scaler, look-back, horizon, or zero policy. This implementation keeps those choices visible, preserves chronological alignment, and prevents learned transformations from using the held-out period. No code execution, API call, test run, or independent numerical verification was performed under the run policy.

Classical regression baselines

Which models should receive the same stock-price rows before the LSTM and ensemble are considered? The paper's Section 4 comparison uses five tabular regressors: support-vector regression (SVR), multilayer perceptron regression (MLPR), K-nearest-neighbor regression (KNN), random-forest regression, and XG-Boost regression. Each model consumes the same training features and produces one continuous prediction for each evaluation row.

In the notation used here, X_train is a rank-2 feature matrix with shape (n_train, n_features), and y_train is a one-dimensional target array with shape (n_train,). X_eval has shape (n_eval, n_features) and must use the same feature order and preprocessing as X_train. The resulting prediction vector has shape (n_eval,). This common contract makes the models comparable and allows their predictions to be aligned later for the Random Forest + XG-Boost + LSTM ensemble.

What the paper specifies—and what it does not

Paper fact. The paper includes SVR, MLPR, KNN, random forest, and XG-Boost in its reported comparison. It describes random forest as a bagging ensemble of decision trees: multiple trees are trained and their regression outputs are aggregated. It describes XG-Boost as sequential gradient boosting: weak learners are added iteratively while later learners respond to earlier errors through gradient-based updates.

Implementation decision. The generated package wraps established Python estimators behind a small local protocol. The paper does not specify the SVR kernel or regularization, the MLPR architecture, the KNN neighbor count, the selected random-forest settings, the selected XG-Boost settings, or the search procedure. Consequently, these values remain constructor arguments or explicit configuration rather than being presented as recovered paper settings.

The paper does provide candidate values for the random forest and XG-Boost components. These are search candidates, not selected configurations. A candidate grid says which values may be tried; it does not say which value produced the paper's reported result.

One shared fit/predict contract

The RegressorProtocol in src/stock_forecasting/models/base.py defines the boundary used by the tabular adapters. Its responsibility is deliberately small: fit receives training arrays and returns a fitted model, while predict receives rank-2 evaluation features and returns one prediction per row. The accompanying validate_regression_arrays function checks numeric dtype, rank, nonempty dimensions, finite values, and sample-count agreement.

The central contract is visible in this focused excerpt:

@runtime_checkable
class RegressorProtocol(Protocol):
    """Protocol implemented by tabular regression model adapters.

    Implementations consume a rank-2 feature matrix with shape
    ``(n_samples, n_features)`` and a one-dimensional target vector with shape
    ``(n_samples,)``.  ``fit`` returns the fitted estimator to support the
    conventional ``model.fit(...).predict(...)`` usage.
    """

    def fit(self, X: np.ndarray, y: np.ndarray) -> RegressorProtocol:
        """Fit the regressor and return the fitted model."""

    def predict(self, X: np.ndarray) -> np.ndarray:
        """Return one continuous prediction for each input row."""

This is more than defensive programming. If one adapter silently accepts a differently ordered feature matrix, returns a column matrix instead of a vector, or drops rows, the comparison and eventual ensemble can become invalid while still appearing to run. The protocol does not decide which model is best; it ensures that each model receives and returns data in a known form.

The individual adapters

SVRRegressor wraps sklearn.svm.SVR. Its configurable settings include kernel, C, gamma, epsilon, and related estimator parameters. The adapter intentionally leaves feature scaling external. Scaling may be important for SVR, but the paper does not specify a scaler or its fitting scope, so that choice belongs to the preprocessing configuration rather than this model class. Calling predict before fit, changing the feature count, or producing non-finite outputs raises an explicit error.

MLPRRegressor wraps scikit-learn's multilayer perceptron regressor and exposes hidden_layer_sizes, activation, solver, optimization settings, iteration count, and random state. The paper reports an MLPR result but does not describe its hidden layers or training configuration. The generated adapter therefore does not claim that its ordinary defaults reproduce the paper's MLPR.

KNN is also configurable. The relevant choices include n_neighbors, the weighting rule, and the distance metric. A neighbor count must be positive and cannot exceed the number of training rows. KNN is particularly sensitive to feature scale because distances are computed in feature space: a high-magnitude column can dominate a lower-magnitude column unless preprocessing addresses that issue. Tree-based models generally respond differently to scale because their splits compare feature values rather than relying directly on geometric distance.

The paper's KNN distance record is identified as eq_2, but its canonical LaTeX is empty because the extracted expression is OCR-damaged. The supplied symbol record describes D as a nonnegative distance between observations, h_i as a data value or observation, p_r as a predicted or query value, n as the number of dimensions or terms, and l as a summation index. The exact vector components, square-root formatting, and index placement are unclear. Therefore, no equation block is reproduced here and no formula is reconstructed.

Instead, euclidean_distance_eq_2 documents the implementation mapping. It accepts two finite rank-1 vectors with equal feature dimensions and computes a standard vector Euclidean distance. The generated file explicitly labels this as a coding decision rather than recovered LaTeX:

def euclidean_distance_eq_2(left: np.ndarray, right: np.ndarray) -> float:
    """Return the vector Euclidean distance associated conceptually with eq_2.

    The paper's canonical eq_2 record has no LaTeX because its extraction is
    damaged. This vector-norm implementation is therefore an explicit coding
    decision, not a reconstruction of the missing equation.
    """

Random forest and XG-Boost candidate grids

The generated random-forest adapter exposes n_estimators, max_depth, max_features, bootstrap behavior, and random-state settings. Here, n_estimators is the number of trees, max_depth limits tree depth, and max_features controls the feature subset considered by a split. The paper's Table 1 candidate values are returned exactly by random_forest_candidates:

def random_forest_candidates() -> dict[str, tuple[object, ...]]:
    """Return the random-forest candidates listed in Table 1.

    These are candidate values reported by the paper, not selected values. The
    paper does not specify the search procedure or the configuration ultimately
    used for its reported results.
    """

    return {
        "n_estimators": (50, 100, 200),
        "max_depth": (3, 5, 7),
        "max_features": ("sqrt", "log2"),
    }

For XG-Boost, learning_rate controls the contribution of each boosting step, n_estimators is the number of boosting estimators, and max_depth controls the depth of the tree learners. The paper lists max_depth values of 3, 4, and 5; learning_rate values of 0.1, 0.01, and 0.001; and n_estimators values of 50, 100, 150, 500, and 1000. The xgboost_candidates function preserves those values, while XGBoostRegressor keeps the package dependency, objective, random state, and job count explicit implementation choices.

The combined grid interface makes the distinction auditable:

def paper_candidate_grids() -> dict[str, dict[str, tuple[object, ...]]]:
    """Return the Random Forest and XG-Boost candidate grids from Table 1.

    Returns
    -------
    dict[str, dict[str, tuple[object, ...]]]
        Fresh mappings containing the paper-listed candidate values under the
        keys ``"random_forest"`` and ``"xgboost"``.

    No selected hyperparameters are inferred, and no tuning procedure is
    performed here. The values are obtained from the owning model modules so
    that the candidate definitions remain centralized.
    """

paper_candidate_grids is therefore a fidelity utility, not a claim that tuning has happened. The separate tuning layer can choose a search and validation policy, but those choices are absent from the paper and must be recorded if used.

Constructing models without hiding their identity

build_regressor accepts canonical names such as "svr", "mlpr", "knn", "random_forest", and "xgboost", along with selected aliases such as "XG-Boost" and "random forest". It passes the supplied configuration directly to the corresponding adapter. It rejects LSTM and ensemble requests because those require rank-3 sequence inputs and separate orchestration, respectively.

A small comparison setup can therefore be written as follows:

from stock_forecasting.models.factory import build_regressor

model_configs = {
    "svr": {"kernel": "rbf", "C": 1.0},
    "mlpr": {"hidden_layer_sizes": (32,), "random_state": 7},
    "knn": {"n_neighbors": 5, "weights": "distance"},
    "random_forest": {
        "n_estimators": 100,
        "max_depth": 5,
        "max_features": "sqrt",
        "random_state": 7,
    },
    "xgboost": {
        "max_depth": 3,
        "learning_rate": 0.1,
        "n_estimators": 100,
        "random_state": 7,
    },
}

models = {
    name: build_regressor(name, config)
    for name, config in model_configs.items()
}

predictions = {}
for name, model in models.items():
    model.fit(X_train, y_train)
    predictions[name] = model.predict(X_eval)

In this worked example, X_train and y_train must come from the earlier chronological preprocessing stage, and X_eval must use the identical feature order and learned preprocessing state. Each entry in predictions should contain one value for every row in X_eval. The particular settings above are illustrative implementation choices; they are not reported as the paper's selected hyperparameters.

Bagging versus boosting

Random forest and XG-Boost are both ensembles, but they combine trees differently. Random forest uses bagging: multiple trees are trained with randomized data or feature choices, and their predictions are aggregated to reduce dependence on any one tree. XG-Boost uses sequential boosting: later learners are fitted in response to the current ensemble's errors. This distinction matters when interpreting the models and when documenting tuning parameters. A larger forest changes the number of independently randomized trees, while a larger boosted estimator count extends a sequence of corrective learners.

The classical_regressors_forward and fit_models_and_generate_predictions method contracts keep these baseline predictions separate until evaluation or explicit ensemble construction. That separation prevents the tabular adapters from silently deciding how the paper's later RF + XG-Boost + LSTM combination should work. The paper calls that model stacked, but its actual wiring remains unspecified; alignment and combination are handled in the following ensemble stage.

Finally, the generated files and planned tests were not executed or independently verified under this run policy. The candidate grids and adapter contracts described here are implementation artifacts intended for static and semantic review, not evidence that the paper's Table 2 values have been reproduced.

The two-layer LSTM regressor

How is an LSTM different from the tabular regressors described earlier? A tabular model receives one row of features at a time. An LSTM receives an ordered window of rows, so it can model relationships across time. In this implementation, one input has rank 3 and shape (batch, sequence_length, n_features): batch is the number of sequences processed together, sequence_length is the look-back window, and n_features is the number of selected input columns.

The paper specifies two LSTM layers, dropout of 0.2, a dense layer with 25 units, the Adam optimizer, MSE loss, batch size 32, and a maximum of 50 epochs. It does not specify the look-back length, recurrent-unit count, exact return-sequence settings, validation behavior, or final output-layer details. The generated implementation therefore records those missing values as implementation choices. It uses 32 recurrent units, returns sequences from the first layer, uses the second layer's final representation, and ends with a linear single-output layer for regression.

What the extracted gate equations tell us

An LSTM cell uses gates to regulate information. The input gate controls information entering the memory, the forget gate controls information retained from the previous state, and the output gate controls information exposed as the current representation. The paper supplies one displayed expression for each of these gate activations, but not a complete recurrence.

The first supplied expression represents the input-gate activation. It is reproduced exactly from canonical record eq_8.

Here, iga is the input-gate activation, represented as a vector or tensor. σ is the elementwise sigmoid activation. Wip is the input-gate weight matrix, although its dimensions are not supplied. ht−1 is the previous hidden state, and Xc is the current input as printed in the paper; both are described as vectors or tensors. bi is the input-gate bias vector. In practical terms, the gate produces one value per gate unit, conventionally between 0 and 1 because of the sigmoid.

The helper input_gate_eq_8 in src/stock_forecasting/models/lstm.py gives this record a named implementation boundary. It does not claim that the extracted expression is a complete cell equation. The helper validates rank-2 step tensors, concatenates the previous hidden state and current input, applies a matrix multiplication and bias, and then applies a sigmoid through TensorFlow.

The forget-gate record is eq_9.

fga denotes the forget-gate activation. σ again denotes the elementwise sigmoid. Wf g is the printed forget-gate weight matrix, while ht−1 and Xc denote the previous hidden state and current input. b f is the forget-gate bias vector. As with the input gate, the expression describes a gate activation rather than the complete update of the LSTM cell state.

The generated function forget_gate_eq_9 maps this paper record to the same framework-level affine-and-sigmoid pattern as the input gate. The separate function name is useful for traceability: readers can connect the implementation to eq_9 without treating the OCR-damaged notation as a complete executable specification.

The output-gate record is eq_10.

Opg is the output-gate activation, σ is the sigmoid, Wop is the output-gate weight matrix, and bo is the output-gate bias. The same ht−1 and Xc symbols represent the previous hidden state and current input. The supplied record gives no matrix dimensions, candidate-cell expression, cell-state update, or hidden-state update.

The corresponding output_gate_eq_10 helper therefore exposes only the gate-level mapping. The actual recurrent computation is delegated to Keras rather than reconstructed from incomplete extracted equations. This is important: implementing the missing equations from memory would create a new mathematical specification rather than reproduce the supplied record.

Building the two-layer network

The core network construction is concentrated in build_lstm_regressor. The excerpt below is copied from src/stock_forecasting/models/lstm.py:

    inputs = keras.Input(shape=input_shape, name="stock_sequence")
    # Table 1: first LSTM layer, followed by dropout 0.2 in the paper configuration.
    recurrent = keras.layers.LSTM(units, return_sequences=True, name="lstm_1")(inputs)
    recurrent = keras.layers.Dropout(float(dropout), name="dropout_1")(recurrent)
    # Table 1: second LSTM layer; returning its final state is an explicit wiring choice.
    recurrent = keras.layers.LSTM(units, return_sequences=False, name="lstm_2")(recurrent)
    recurrent = keras.layers.Dropout(float(dropout), name="dropout_2")(recurrent)
    dense = keras.layers.Dense(dense_units, activation="relu", name="dense_25")(recurrent)
    outputs = keras.layers.Dense(1, activation="linear", name="price_output")(dense)
    return keras.Model(inputs=inputs, outputs=outputs, name="two_layer_lstm_regressor")

The input shape excludes the batch dimension, so input_shape is (sequence_length, n_features). Keras adds the batch dimension at runtime, producing rank-3 inputs. The first LSTM uses return_sequences=True, which preserves one recurrent output for every position in the window and supplies a sequence to the second LSTM. The second layer uses return_sequences=False, so it returns one final representation per input sequence. The generated file explicitly labels this return-sequence wiring as a choice because the paper does not state it.

Dropout is applied after each LSTM layer using the configured probability. The paper specifies 0.2; the code keeps the value configurable so that the unresolved choice is visible rather than hidden. The dense layer has 25 units, matching Table 1, and uses a ReLU activation. The final dense layer has one unit and a linear activation. That final linear output is a reasonable regression decision, but the paper does not explicitly provide the output-layer activation.

The recurrent-unit count is another implementation decision. The generated LSTMRegressor defaults to 32 units per LSTM layer, but this number should not be described as recovered from the paper. The paper specifies the number of layers and the dense width, not the number of recurrent units.

Training with MSE and Adam

The LSTM adapter enforces the sequence contract before training. Its X input must be a numeric rank-3 array with shape (batch, sequence_length, n_features). Its target y may have shape (batch,) or (batch, 1), but it must contain exactly one target per sequence. The adapter reshapes targets to (batch, 1) before passing them to Keras.

The training configuration is visible in this excerpt from LSTMRegressor.fit:

        self.model_.compile(optimizer="adam", loss="mse")
        fit_kwargs: dict[str, Any] = {
            "x": X_array,
            "y": y_array,
            "epochs": self.max_epochs,
            "batch_size": self.batch_size,
            "verbose": self.verbose,
            "shuffle": False,
        }
        if self.validation_fraction is not None:
            fit_kwargs["validation_split"] = float(self.validation_fraction)
        self.history_ = self.model_.fit(**fit_kwargs)

The optimizer and loss match the paper's stated settings. The adapter caps max_epochs at 50 and defaults batch_size to 32. Shuffling is disabled as an implementation safeguard for ordered sequences. A validation split is optional and is not a paper-specified value. If used, it must be recorded in the experiment manifest rather than mistaken for a setting recovered from Section 4.

MSE is the paper-specified neural loss, but the supplied equation records contain no canonical MSE LaTeX. Consequently, src/stock_forecasting/losses.py does not display or invent a formula. mean_squared_error_loss uses TensorFlow's squared-difference operation followed by mean reduction, while validate_loss_inputs checks that prediction and target tensors have matching shapes and finite floating-point values.

Worked shape example

Suppose preprocessing selects four features, chooses a look-back of 20 observations, and creates a batch of 32 sequences. The LSTM input then has shape (32, 20, 4). Each sequence corresponds to one future target, so the target array has shape (32, 1) during model training. The network returns (32, 1), meaning one continuous forecast for each sequence in the batch.

The look-back value of 20 in this example is not supplied by the paper; it is an explicit reproduction decision. The sequence builder in src/stock_forecasting/data/sequences.py preserves target timestamps and ensures that each window precedes its target. Initial observations are lost because a complete look-back window must exist before the first sequence can be formed. Those shortened sequence indices must later be aligned with the random-forest and XG-Boost predictions before ensemble construction.

Why the framework implementation matters

A hand-written LSTM cell would need the candidate-cell computation, cell-state update, and hidden-state update in addition to the three gate expressions. Those pieces are absent from eq_8, eq_9, and eq_10, and the extracted notation itself contains unresolved formatting such as Xc and spaced subscripts. The generated code therefore uses TensorFlow/Keras for the complete recurrent mechanism and keeps the three named helpers as traceability mappings, not as a claim of a newly reconstructed cell.

This choice also preserves the boundary between paper facts and implementation decisions. The two LSTM layers, dropout 0.2, dense width 25, Adam, MSE, batch size 32, and maximum 50 epochs come from the paper's Table 1 description. The recurrent units, look-back, final linear output, return-sequence behavior, dropout placement, validation split, and scaling behavior are choices required to make a runnable framework. The generated files document those choices, but no LSTM training, code execution, syntax check, or independent verification was performed under the run policy.

Training orchestration and leakage-aware tuning

How can model selection remain fair when stock observations arrive in time order? Treat the test set as a final exam: it may be used to measure the finished models, but not to choose their settings. The generated implementation separates three roles:

• X_train and y_train are the features and targets available for fitting.

• X_validation and y_validation are a later portion of the training data used to compare candidate configurations.

• X_test and y_test are the held-out observations reserved for final evaluation.

The paper describes hyperparameter tuning and lists candidate values for random forest and XG-Boost, but it does not specify the search algorithm, validation fraction, scoring rule, random seed, or repeat count. Those are therefore implementation decisions, not recovered paper facts. The primary reproduction choice still follows Table 1's chronological 80/20 split; the paper's generic pipeline separately mentions 75/25, so the split policy must remain documented.

Training settings are explicit configuration

The TrainingConfig class records the settings that affect fitting. The paper specifies batch size 32 and a maximum of 50 LSTM epochs. The seed and validation fraction are not supplied by the paper and are consequently caller choices.

@dataclass(frozen=True, slots=True)
class TrainingConfig:
    """Record reproducibility and neural-training settings.

    Batch size 32 and a maximum of 50 epochs come from Table 1.  The seed and
    validation fraction are implementation decisions because the paper does
    not specify them.  ``max_epochs`` is capped at 50 so this configuration
    cannot silently exceed the reported neural-training limit.
    """

    seed: Optional[int]
    batch_size: int = 32
    max_epochs: int = 50
    validation_fraction: Optional[float] = None

A fitted model is an estimator that has already learned parameters from permitted training rows and can respond to predict. A PredictionBundle is different: it associates prediction values with their timestamps. That distinction matters because a fitted estimator has model state, whereas a prediction bundle is an auditable evaluation artifact. The runner returns both through ExperimentPredictions.

For an illustrative run, a caller can choose a nonnegative seed, retain the paper-aligned batch size and epoch cap, and provide a training-only validation fraction such as 0.2. That choice does not reproduce an unstated paper protocol; it makes the validation policy inspectable. If validation_fraction remains None, the generated runner can fit configured models without invoking the optional tabular search interface.

Candidate search uses training data only

The generated search_regressor function makes exhaustive Cartesian-grid search an explicit implementation decision. It first splits the supplied training arrays chronologically, evaluates each candidate on the validation suffix, selects the candidate with the smallest supplied score, and refits the selected estimator on all supplied training rows. The function does not accept test data.

def search_regressor(
    factory: Callable[[Mapping[str, Any]], RegressorProtocol],
    grid: Mapping[str, Sequence[Any]],
    X_train: np.ndarray,
    y_train: np.ndarray,
    validation_fraction: float,
    scorer: Callable[..., float],
) -> SearchResult:
    """Select and refit a regressor using a chronological training split.

    Candidates are evaluated on the final ``validation_fraction`` of the
    supplied training rows, while the earlier rows are used for candidate
    fitting.  The candidate with the smallest score is selected, which is an
    explicit implementation decision appropriate for RMSE and other
    loss-style scorers.  The paper lists candidate values but does not specify
    this search algorithm, score direction, or validation protocol.

    The function intentionally has no test-data parameter.  Consequently,
    held-out test targets cannot enter candidate selection through this API.
    After selection, a fresh estimator is fitted on all ``X_train``/``y_train``
    rows so callers receive a model trained on the complete permitted sample.
    """

Here, validation_fraction determines how much of the supplied training suffix is reserved for candidate comparison. The scorer is a callable that receives true and predicted validation values; the generated runner supplies an RMSE scorer, for which lower is better. factory constructs a fresh estimator for each parameter combination, preventing one candidate's fitted state from carrying into another candidate.

The returned SearchResult records selected_params, validation_score, and the refitted estimator. In a worked audit, readers should inspect selected_params and the score used for selection, but should not interpret that score as a held-out test result. The paper supplies random-forest candidates for estimators, depth, and feature selection, and XG-Boost candidates for depth, learning rate, and estimator count; it does not state which values won.

The runner coordinates model families

fit_models_and_generate_predictions is the orchestration method for one prepared stock dataset. It fits tabular models on rank-2 arrays, optionally fits the sequence model on rank-3 arrays, and creates prediction bundles for the appropriate test timestamps. Stock-specific fitting is the default: the caller supplies one prepared dataset per security rather than combining TAINIWALCHM and AGROPHOS into a single training operation.

The central isolation rule is visible in the runner's public contract:

def fit_models_and_generate_predictions(
    prepared: Any,
    model_configs: Mapping[str, Any],
    training_config: TrainingConfig,
) -> ExperimentPredictions:
    """Fit comparison models using training data and produce held-out predictions.

    The test partition is passed only to ``predict``.  Optional searches use a
    chronological validation suffix of the training partition, as required by the
    leakage-aware search interface.  Stock-specific fitting is represented by the
    caller supplying one prepared dataset per stock.
    """

The runner obtains tabular_split and sequence_split from prepared. It validates that tabular features are rank 2, that their row counts match their targets, and that predictions have one value per evaluation row. For each configured classical model, _fit_tabular_model either fits directly or invokes search_regressor; the test arrays are not passed into either fitting or tuning.

The LSTM receives sequence_X_train and sequence_X_test, not the tabular arrays. Its settings are passed from TrainingConfig, including max_epochs, batch_size, validation_fraction, and seed. Thus the paper's maximum of 50 epochs and batch size 32 remain visible in the training record. The recurrent architecture still contains choices that the paper does not specify, including the look-back length and recurrent unit count established elsewhere in preprocessing and model configuration.

Sequence predictions require additional alignment

A look-back window means that the first observations may not have enough historical rows to form an LSTM input. Consequently, the sequence test partition can contain fewer timestamps than the tabular test partition. The runner intersects the tabular and sequence timestamps before fitting the ensemble and before producing its combined evaluation output.

This is not merely bookkeeping. If X_test contains one row for a timestamp that the LSTM cannot represent, combining its prediction with a neighboring LSTM prediction would compare different target times. The runner therefore creates common training rows and common evaluation rows by timestamp, checks that aligned tabular and sequence targets agree, and preserves chronological order.

The final ExperimentPredictions object contains fitted models, named predictions, a target bundle, selected_configs, and training_metadata. The metadata records such details as the chronological split policy, validation fraction, seed, epoch cap, batch size, and the fact that test targets were not used for fitting. This is the practical audit trail for the worked example: inspect the selected-configuration records without reporting any numerical value as verified.

Manifest and reproducibility limits

ExperimentManifest complements the runner output by recording the data source, feature columns, target column, split policy, scaler, look-back, horizon, seed, tuning method, model configurations, and ensemble choice. Its purpose is to prevent a later reader from mistaking a default for a paper fact. In particular, the manifest can state that Table 1 is the primary reproduction target while preserving the conflicting 75/25 recommendation from the generic pipeline.

The paper does not specify early stopping, so the generated configuration should not be described as reproducing an early-stopping protocol. It also does not specify whether repeated trials were averaged, whether validation was chronological, or how ties between candidates were handled. The generated exhaustive chronological search and lower-is-better scoring rule are deliberate implementation policies. A recorded seed improves auditability, but it cannot guarantee stable model selection across libraries, hardware, dependency versions, or data snapshots.

Chronological validation is preferable here to ordinary random cross-validation because random folds can place later observations in the fitting portion while earlier observations appear in validation. That ordering would not represent the intended forecasting direction. Even chronological validation remains an approximation: market regimes can change, and a single validation suffix may not represent all future conditions.

Finally, no code execution, API request, training run, test run, syntax check, or semantic code verification occurred in this pipeline. The search and runner behavior described here is the generated design and contract, not an observed execution result. The paper's Table 2 metrics therefore remain reported reference values rather than independently verified outcomes.

Aligning and combining the ensemble

How can three different models contribute to one stock-price forecast without accidentally combining predictions for different dates? The practical rule is simple: first align every prediction by timestamp, then combine the aligned values. This matters especially for the Random Forest + XG-Boost + LSTM ensemble because the tabular models can predict every evaluation row, while the LSTM loses observations when it creates look-back windows.

The paper's proposed method combines random forest, XG-Boost, and LSTM predictions, but it does not specify the actual wiring. It calls the model “stacked” and also discusses assigning higher weights to models with lower loss. These descriptions do not identify one reproducible operation. A weighted average is a blending operation: fixed or externally supplied weights combine component outputs directly. Stacking usually means that a second-stage model learns how to combine component predictions. The generated code supports both alternatives and records which one was selected.

Prediction bundles preserve identity and time

A PredictionBundle connects a model name, a pandas index, and a prediction array. Its values may have shape (n,) or (n, 1), but they always represent one prediction per timestamp. The index must not contain duplicates, and the number of labels must equal the number of prediction rows. This is more than defensive programming: without the index, a numerically valid vector can still be paired with the wrong dates.

For example, the three ensemble components are named random_forest, xgboost, and lstm. Their predictions should represent the same target column and target scale. The LSTM bundle may have fewer timestamps because its first predictions require a complete look-back window.

The alignment module represents the result with AlignedPredictions. Its predictions field is a rank-2 matrix with shape (n_aligned_samples, n_components). For this ensemble, n_components is three, with columns ordered as random forest, XG-Boost, and LSTM. Its target field has shape (n_aligned_samples,), and its index contains the common timestamps in target order.

The core implementation performs an inner intersection while preserving the target's order:

def align_prediction_bundles(
    bundles: Sequence[PredictionBundle], target: PredictionBundle
) -> AlignedPredictions:
    """Inner-align component predictions and targets by timestamp.

    Args:
        bundles: Component prediction bundles, typically RF, XG-Boost, and
            LSTM predictions. Every bundle must have unique timestamps.
        target: Target values indexed by their evaluation timestamps.

    Returns:
        An :class:`AlignedPredictions` record whose rows refer to exactly the
        same timestamps across every component and the target.
    """
    ...

The excerpt shows the public responsibility without reproducing the entire module. In the generated implementation, the function rejects empty component lists, repeated component names, duplicate indices, invalid numeric values, and an empty timestamp intersection. It obtains common timestamps from the target's original order rather than independently sorting each model's output. This preserves the temporal order used for evaluation.

Worked example: one shorter LSTM stream

Consider four evaluation dates: 2024-01-02, 2024-01-03, 2024-01-04, and 2024-01-05. Suppose random forest and XG-Boost produce four predictions, but the LSTM produces only three because the initial date was consumed by sequence construction. The following excerpt constructs that situation. It is an illustrative example of the public interfaces; it is not an execution result.

import numpy as np
import pandas as pd

from stock_forecasting.ensemble.alignment import align_prediction_bundles
from stock_forecasting.types import PredictionBundle

index = pd.to_datetime([
    "2024-01-02", "2024-01-03", "2024-01-04", "2024-01-05"
])
lstm_index = index[1:]

target = PredictionBundle(
    "close",
    index,
    np.array([10.0, 11.0, 12.0, 13.0]),
)
rf = PredictionBundle(
    "random_forest",
    index,
    np.array([10.2, 10.8, 12.1, 12.7]),
)
xgb = PredictionBundle(
    "xgboost",
    index,
    np.array([10.1, 11.1, 11.9, 13.1]),
)
lstm = PredictionBundle(
    "lstm",
    lstm_index,
    np.array([10.9, 12.2, 12.9]),
)

aligned = align_prediction_bundles([rf, xgb, lstm], target)

The common index is the final three dates. Conceptually, the resulting matrix is ordered by those dates and has three columns:

             random_forest  xgboost  lstm
2024-01-03           10.8     11.1  10.9
2024-01-04           12.1     11.9  12.2
2024-01-05           12.7     13.1  12.9

The important invariant is that each row refers to one date across all three columns. The first random-forest and XG-Boost predictions are discarded from the ensemble view because there is no corresponding LSTM prediction. The target is reduced to the same dates. This is preferable to padding the LSTM output or silently shifting its predictions.

Direct weighted blending

The generated weighted_average function implements one explicit interpretation of the paper's weight discussion. It requires a prediction matrix with shape (n_samples, 3) and a weight vector with shape (3,). The weights must be finite and nonnegative, and the implementation normalizes them to sum to one. The returned prediction vector has shape (n_samples,).

def weighted_average(predictions: np.ndarray, weights: np.ndarray) -> np.ndarray:
    """Combine aligned component predictions using normalized nonnegative weights.

    The paper mentions weighting lower-loss models but does not define weights or
    a fitting procedure.  This implementation therefore makes the combination
    explicit: the three supplied weights are required to be finite and
    nonnegative, then normalized to sum to one.  Columns must be ordered as
    random forest, XG-Boost, and LSTM, and the return value has shape ``(n,)``.
    """
    component_values = _validate_component_predictions(predictions)
    component_weights = _as_finite_float_array(weights, "weights")
    if component_weights.ndim != 1 or component_weights.shape[0] != _COMPONENT_COUNT:
        raise ValueError("weights must have shape (3,)")
    if np.any(component_weights < 0):
        raise ValueError("weights must be nonnegative")
    total = float(np.sum(component_weights))
    if total <= 0.0:
        raise ValueError("at least one weight must be positive")

    normalized_weights = component_weights / total
    return np.asarray(component_values @ normalized_weights, dtype=float)

For equal weighting, a caller can provide (1/3, 1/3, 1/3). That choice is not recovered from the paper; it is an implementation decision. Likewise, weights derived from validation losses would require a documented rule for converting losses into weights. The code deliberately does not infer such a rule.

Learned linear stacking

The alternative is LinearStacker, a second-stage linear regressor. It receives the three component predictions as meta-features and learns coefficients and an intercept from training-only component outputs. Here, “meta-features” means features supplied to the combiner rather than the original stock columns.

The fitting function makes the data boundary explicit:

def fit_linear_stacker(
    train_predictions: np.ndarray,
    y_train: np.ndarray,
) -> LinearStacker:
    """Fit a training-only ordinary-least-squares linear stacker.

    The paper calls the ensemble stacked but does not specify a meta-learner.
    Ordinary least squares with an intercept is an explicit implementation
    choice.  ``train_predictions`` must contain exactly the three aligned
    component predictions, and ``y_train`` must contain the corresponding
    training targets; test targets must not be passed here.
    """
    ...

The generated implementation validates the three-column input, checks that y_train has a matching number of rows, adds an intercept column, and solves the linear least-squares problem. Its predict method accepts another (n_samples, 3) matrix and returns one value per row. The important leakage constraint is not the particular linear model; it is that the stacker's fitting targets come only from permitted training data.

A rigorous stack often uses out-of-fold component predictions when fitting the meta-model. Otherwise, a stacker may learn from component predictions generated on the same rows used to fit those components, which can make the component outputs look unrealistically accurate. The paper does not specify out-of-fold generation, a validation arrangement, or even the meta-learner type. The generated framework therefore exposes ordinary least-squares stacking as a documented implementation choice rather than presenting it as the paper's recovered procedure.

Orchestrating the three components

StackedRFXGBoostLSTM owns one random-forest regressor, one XG-Boost regressor, and one LSTM regressor. Its tabular inputs have rank 2, while its LSTM inputs have rank 3. During fitting, the caller must supply pre-aligned training rows to all three components. The class then creates a three-column training prediction matrix. If the configured combiner is linear_stacking, it fits the meta-model on that matrix and the training targets.

During prediction, the class generates component outputs, associates the shorter LSTM output with the final evaluation timestamps, and calls align_prediction_bundles. The final values are then produced either by weighted_average or by the fitted LinearStacker. The resulting PredictionBundle is named rf_xgboost_lstm_ensemble and carries the aligned timestamps.

A configuration makes the ambiguity visible at the call site:

from stock_forecasting.config import EnsembleConfig
from stock_forecasting.ensemble.stacked import build_ensemble

ensemble_config = EnsembleConfig(
    combiner="weighted_average",
    weights=(1 / 3, 1 / 3, 1 / 3),
    meta_features="predictions",
)
ensemble = build_ensemble(ensemble_config)

Switching to combiner="linear_stacking" changes the operation from direct blending to a learned second stage. The combiner_metadata property records the selected combiner, weights, meta-feature description, component order, and the fact that the paper's wiring was not specified. This provenance is essential: a future reader can distinguish a paper fact from a reproduction decision.

The generated implementation has not been executed, and neither its alignment behavior nor its numerical output has been independently verified under the run policy. The intended contracts are nevertheless explicit: common timestamps, matching target semantics and scale, finite predictions, three fixed component columns, and no held-out test targets used to fit a learned combiner.

RMSE, R2, and auditable reporting

How do we know whether a stock-price prediction is useful? Compare predictions with the correct held-out targets, using the same timestamps and the same value scale. The generated metrics layer implements this final step through evaluate_regression_predictions, while build_metric_report adds stock and model identity for repeatable reporting.

This section concerns regression evaluation. Let y_true denote the observed target values from the held-out test period, and let y_pred denote the corresponding model predictions. Both are numeric arrays with shape (n_test,) or (n_test, 1). “Held-out” means that these observations were not used to fit the model or select its settings.

RMSE, or root mean square error, summarizes prediction error in the units of the target. If the target is an original stock price, RMSE is expressed in price units. If the target remains standardized, RMSE is expressed in standardized units instead. The implementation does not silently convert between these scales.

R2, or R-squared, is the coefficient of determination. It compares the model's residual error with the variation around a constant prediction based on the target mean. R2 can be negative when predictions are worse than that constant baseline; it is therefore incorrect to assume that every valid R2 must lie between zero and one.

Alignment is part of the metric contract

A numerically valid prediction is not necessarily a valid evaluation. The prediction must refer to the same observations as the target, in the same order. The metric function checks numeric dtype, finiteness, supported shapes, equal lengths, and optional index length and uniqueness. A pandas index can document timestamp alignment, although the caller remains responsible for constructing the two arrays from matching observations.

The core implementation normalizes one-dimensional and single-column inputs before calling standard regression metrics:

def _validate_metric_inputs(
    y_true: np.ndarray, y_pred: np.ndarray
) -> tuple[np.ndarray, np.ndarray]:
    """Validate target/prediction shape, finiteness, and observation count."""
    true_values = _normalise_regression_values(y_true, "y_true")
    predicted_values = _normalise_regression_values(y_pred, "y_pred")
    if true_values.shape != predicted_values.shape:
        raise ValueError(
            "y_true and y_pred must have the same number of aligned observations; "
            f"got {true_values.shape[0]} and {predicted_values.shape[0]}"
        )
    return true_values, predicted_values

The private helper _validate_metric_inputs converts accepted target shapes to compatible vectors and rejects mismatched lengths. It does not inspect the semantic content of timestamps, so the surrounding pipeline must preserve timestamp order when producing y_true and y_pred.

Computing and labeling the metrics

The public rmse function returns a finite, nonnegative scalar. The public r2_score function requires at least two observations because the standard coefficient of determination is undefined for a one-observation target. evaluate_regression_predictions performs the common checks and returns a dictionary containing rmse and r2.

def evaluate_regression_predictions(
    y_true: np.ndarray,
    y_pred: np.ndarray,
    index: pd.Index | None = None,
    scale: str = "original",
) -> dict[str, float]:
    """Return RMSE and R2 after checking held-out prediction alignment."""
    if not isinstance(scale, str) or not scale.strip():
        raise ValueError("scale must be a nonempty string")

    true_values, predicted_values = _validate_metric_inputs(y_true, y_pred)
    _validate_index(index, true_values.shape[0])

    # Reuse the public metric functions after the common alignment checks.
    metrics: dict[str, float] = {
        "rmse": rmse(true_values, predicted_values),
        "r2": r2_score(true_values, predicted_values),
    }
    if not all(np.isfinite(value) for value in metrics.values()):
        raise ValueError("regression metrics must be finite")
    return metrics

The scale argument is a reporting label, not a transformation instruction. Passing scale="original" means that the caller has already supplied original-scale values. Passing scale="standardized" records a different convention, but the function does not inverse-transform the arrays. If a scaler was used during preprocessing, the caller must apply the documented inverse target transformation before requesting original-price metrics.

Worked example: two aligned prediction bundles

A PredictionBundle associates a model name, a pandas index, and one prediction per index value. The following example creates a small target bundle and two model outputs. It is an interface example only; it was not executed in this pipeline and does not represent a paper result.

import numpy as np
import pandas as pd

from stock_forecasting.metrics.regression import (
    evaluate_regression_predictions,
)
from stock_forecasting.types import PredictionBundle

index = pd.Index(
    pd.to_datetime(["2023-01-03", "2023-01-04", "2023-01-05"]),
    name="Date",
)
target = PredictionBundle(
    "observed",
    index,
    np.array([10.0, 11.0, 12.0]),
)
svr_prediction = PredictionBundle(
    "svr",
    index,
    np.array([10.2, 10.8, 12.1]),
)
ensemble_prediction = PredictionBundle(
    "ensemble",
    index,
    np.array([10.1, 11.1, 11.9]),
)

svr_metrics = evaluate_regression_predictions(
    target.values,
    svr_prediction.values,
    index=target.index,
    scale="original",
)
ensemble_metrics = evaluate_regression_predictions(
    target.values,
    ensemble_prediction.values,
    index=target.index,
    scale="original",
)

Here, both prediction arrays have three values and use exactly the target's index. The scale="original" label states that these illustrative values are being interpreted in the target's original scale. No numerical output is asserted because execution was disabled.

Building auditable records

A dictionary of metrics is useful for one call, but a multi-stock comparison also needs to identify the security and model. build_metric_report accepts a mapping from model names to PredictionBundle objects and aligns each prediction to the target index in target order. It returns immutable MetricRecord objects containing stock, model, rmse, r2, and scale.

def build_metric_report(
    stock: str,
    prediction_map: Mapping[str, PredictionBundle],
    target: PredictionBundle,
    scale: str,
) -> list[MetricRecord]:
    """Evaluate every named prediction bundle against one aligned target.

    The caller supplies the reporting scale explicitly, for example ``"original"``
    or ``"standardized"``.  This function does not perform inverse scaling.  It
    also does not infer a train/test split or configuration provenance because
    those fields are not part of :class:`MetricRecord`; callers should preserve
    the corresponding experiment manifest separately.
    """

The alignment step uses the intersection of target and prediction timestamps. This is important when, for example, an LSTM has fewer evaluable rows because its look-back windows omit initial observations. An empty timestamp intersection is treated as an error rather than producing misleading metrics.

A report can then be serialized with save_metric_report to either CSV or JSON. Records are sorted deterministically by stock, model, and scale. The serializer intentionally does not invent split, feature, target, or preprocessing information; those details belong in the experiment manifest.

Interpreting the paper's reported values

The paper's Table 2 reports RMSE and R2 for TAINIWALCHM and AGROPHOS across SVR, MLPR, KNN, LSTM, random forest, XG-Boost, and the ensemble. Those values are reference claims from the paper, not outputs independently reproduced by this generated implementation. The paper reports the ensemble as having the strongest comparison values, but the exact data preparation, target scale, and ensemble wiring are not fully specified.

In particular, the reported random-forest RMSE values are much larger than the other listed RMSE values while the corresponding R2 values are high. That combination can occur when target variance or scale differs, but the supplied evidence does not establish the cause. The implementation therefore preserves the reported values as references and does not correct, reinterpret, or claim to explain them.

The meaningful comparison rule is consistent evaluation: every model for a stock should use the same held-out period, aligned target timestamps, and explicitly recorded scale. A model's RMSE and R2 should not be compared across differently transformed targets without qualification. No API call, model training, metric calculation, test run, or independent verification was performed for this article section.

End-to-end offline workflow

How can you understand the complete reproduction workflow without immediately depending on live Yahoo Finance data? Start with a deterministic synthetic table, pass it through the same public pipeline used for a local snapshot, and inspect the resulting configuration and provenance. This demonstrates the plumbing without suggesting that synthetic data reproduces the paper's market-data experiment.

The paper reports Yahoo Finance data for TAINIWALCHM and AGROPHOS, but it does not provide exact ticker identifiers, a data snapshot, all preprocessing choices, or the full ensemble wiring. Consequently, a successful local run is illustrative unless those missing details are recovered. The generated code is designed to make that limitation visible.

One orchestration object for one experiment

The end-to-end entry point is run_experiment in src/stock_forecasting/experiments/pipeline.py. An ExperimentConfig groups the data-source description, preprocessing policy, training settings, ensemble choice, model configurations, tuning description, and package metadata. An ExperimentResult then carries the prepared data, prediction bundles, metric records, and an ExperimentManifest.

This separation distinguishes the paper's reported facts from implementation decisions. For example, the Table 1 configuration motivates the chronological 80/20 split, MSE, Adam, batch size 32, and a maximum of 50 LSTM epochs. In contrast, the feature set, target column, look-back window, scaler, random seed, selected tree parameters, and combiner remain explicit choices because the paper does not specify them.

The central orchestration function is deliberately small because each stage belongs to another module:

def run_experiment(
    frame: pd.DataFrame,
    stock_name: str,
    configs: ExperimentConfig,
) -> ExperimentResult:
    """Prepare, fit, evaluate, and document one stock experiment.

    The supplied frame is assumed to be a local snapshot. This function does
    not make network requests. Preprocessing and model fitting are delegated to
    their owning modules; the runner's held-out predictions are aligned by the
    reporting layer before RMSE and R2 are computed. If optional TensorFlow or
    XG-Boost dependencies are unavailable, the owning model adapter's explicit
    import error is allowed to propagate, preserving the dependency boundary.

    No numerical reproduction of Table 2 is implied: the paper omits several
    choices required for an exact run, and this function performs no comparison
    against the paper's reported values.
    """

    _validate_frame(frame, stock_name)
    if not isinstance(configs, ExperimentConfig):
        raise TypeError("configs must be an ExperimentConfig")

    prepared = prepare_stock_data(frame, configs.preprocessing)
    fitted = fit_models_and_generate_predictions(
        prepared,
        configs.model_configs,
        configs.training,
    )
    reporting_predictions = _convert_to_reporting_scale(fitted, prepared)
    metrics = build_metric_report(
        stock_name,
        reporting_predictions.predictions,
        reporting_predictions.target,
        "original",
    )
    manifest = _build_manifest(stock_name, configs, reporting_predictions)
    return ExperimentResult(
        stock=stock_name,
        prepared=prepared,
        predictions=reporting_predictions,
        metrics=metrics,
        manifest=manifest,
    )

Notice the order: the supplied table is validated, preprocessing is performed, models generate held-out predictions, predictions are converted to the configured reporting scale, metrics are built, and the manifest records the experiment. The function does not download data and does not compare results with Table 2. ExperimentResult is therefore an audit container, not evidence that the paper's reported numbers were reproduced.

Worked example: deterministic synthetic input

The generated make_synthetic_stock_data function creates a chronological OHLCV-shaped DataFrame using a local NumPy random generator. Its columns are Open, High, Low, Close, Adj Close, and Volume. The dates, stochastic process, starting price, and seed are implementation choices; they are not properties of the paper's Yahoo Finance datasets.

The offline demonstration calls the generator as follows:

def run_demo() -> None:
    """Generate synthetic data, run the configured experiment, and print results."""

    # The synthetic generator is the offline replacement for the paper's
    # Yahoo Finance acquisition step; no network request is made here.
    frame = make_synthetic_stock_data(n_rows=320, seed=7)
    configs = _demo_configuration()
    result = run_experiment(frame, stock_name="SYNTHETIC_DEMO", configs=configs)
    _print_result(result)

This function illustrates the complete data flow: acquisition is replaced by a deterministic local fixture, _demo_configuration() makes the missing modeling choices explicit, and run_experiment handles preparation, fitting, ensemble construction, and reporting. The output is labeled illustrative in the script. It cannot reproduce Yahoo Finance results or the paper's Table 2 values.

The demonstration configuration chooses Open, High, Low, and Volume as features, Close as the target, a 20-observation look-back, a one-step horizon, standard scaling, and equal-weight prediction blending. These choices are useful for showing the interfaces, but they must not be described as recovered experimental settings. The Table 1 80/20 split and the stated LSTM training settings are the more direct paper-aligned choices; the generic pipeline's 75/25 recommendation remains a documented conflict.

A conceptual invocation from the repository root is:

python scripts/run_offline_demo.py

This command is an example of intended usage, not a command executed in this pipeline. The full comparison requires the relevant installed modeling dependencies, including TensorFlow and XG-Boost. Synthetic data generation itself uses NumPy and pandas, but that does not mean the complete experiment can run without the optional model packages.

The command-line interface and explicit choices

build_argument_parser in src/stock_forecasting/cli.py exposes both synthetic and CSV pathways. The mutually exclusive source options are visible in this excerpt:

    source = parser.add_mutually_exclusive_group(required=True)
    source.add_argument(
        "--csv",
        type=Path,
        help="Path to a local stock CSV snapshot.",
    )
    source.add_argument(
        "--synthetic",
        action="/__u/onepagecode.substack.com/store_true",
        help="Use deterministic synthetic OHLCV-like data for an offline demo.",
    )

The CLI also exposes the unresolved choices rather than concealing them. Its defaults select Close as the target, use Open, High, Low, and Volume as features, choose an 80% training fraction, use a 20-row look-back, a one-step horizon, and standard scaling. The parser help explicitly identifies the look-back and scaler as choices absent from the paper. The ensemble option accepts weighted_average or linear_stacking, because the paper calls the model stacked while also discussing higher weights for lower-loss models without defining the actual operation.

For a local CSV snapshot, an intended command could be written as:

python -m stock_forecasting.cli \
  --csv data/stock_snapshot.csv \
  --date-column Date \
  --ticker EXPLICIT_TICKER \
  --features Open,High,Low,Volume \
  --target Close \
  --train-fraction 0.8 \
  --lookback 20 \
  --horizon 1 \
  --scaler standard \
  --combiner weighted_average \
  --weights 0.3333333333,0.3333333333,0.3333333333

The exact ticker is intentionally supplied by the user. The paper's names and date ranges are not enough to guarantee a valid Yahoo Finance identifier. Likewise, Close is a configuration choice here, not a paper-established target. Before using real data, revisit the target, feature columns, endpoint dates, interval, scaling policy, sequence alignment, model settings, and ensemble combiner.

Loading a local snapshot

load_stock_csv is an offline implementation addition. It does not claim that the paper used CSV files; it provides a way to work from a saved snapshot rather than relying on a network call. The loader parses the requested date column, rejects malformed or duplicate timestamps, and returns a deterministically sorted DatetimeIndex:

    parsed_dates = pd.to_datetime(frame[date_column], errors="coerce")
    invalid_mask = parsed_dates.isna()
    if bool(invalid_mask.any()):
        invalid_rows = [int(position) for position in invalid_mask[invalid_mask].index[:5]]
        suffix = "" if int(invalid_mask.sum()) <= 5 else " ..."
        raise ValueError(
            f"Date column {date_column!r} contains "
            f"{int(invalid_mask.sum())} unparseable value(s); "
            f"example row labels: {invalid_rows}{suffix}"
        )

    result = frame.drop(columns=[date_column]).copy()
    result.index = pd.DatetimeIndex(parsed_dates, name=date_column)
    result = result.sort_index(kind="mergesort")

    if result.index.has_duplicates:
        duplicate_count = int(result.index.duplicated(keep=False).sum())
        raise ValueError(
            f"CSV snapshot contains {duplicate_count} row(s) with duplicate "
            f"timestamps; resolve duplicates before loading: {csv_path}"
        )

The important contract is provenance and ordering. The date column becomes the index, and duplicate timestamps are an explicit failure rather than being silently aggregated or discarded. Feature selection, null handling, zero-value policy, scaling, and target construction remain responsibilities of the preprocessing pipeline.

What the output means

The demonstration prints metric records and a manifest. A metric record identifies the stock, model, RMSE, R2, and reporting scale. The manifest records the choices needed to interpret those values, including the data-source label, feature and target configuration, split, sequence settings, model configurations, tuning description, and combiner.

This is particularly important for the paper's Table 2. The paper reports RMSE and R2 for TAINIWALCHM and AGROPHOS, including an ensemble result, but the supplied implementation has not been executed and the missing experimental details prevent an exact reconstruction. A locally produced metric is therefore an output of the chosen local configuration, not an independently verified Table 2 result. Even using real Yahoo Finance data would not remove the uncertainty about the original ticker symbols, selected features, scaler, look-back, validation protocol, selected hyperparameters, or ensemble wiring.

The generated CLI includes a warning in its JSON payload that says the outputs do not independently reproduce Table 2. That warning is part of the implementation's audit posture: it prevents an illustrative offline workflow from being mistaken for a verified paper replication.

Practical dependency boundary

The CSV and synthetic loaders are designed for offline data handling. The Yahoo Finance client is isolated in its own module, so the CLI does not make an implicit network request. However, the full model comparison includes a TensorFlow/Keras LSTM and an optional XG-Boost adapter. Those dependencies may be unavailable in an offline environment. A missing dependency should produce an explicit adapter-level error rather than silently replacing the model with another algorithm.

No API call, training run, test run, syntax check, or semantic code verification occurred for this tutorial section. The workflow and commands describe the generated interfaces and intended behavior. They should be treated as a reproducible starting point whose outputs require local execution, matching data, and careful interpretation before they can be compared with the paper's reported references.

Fidelity ledger, limitations, and responsible interpretation

How can a codebase be reproducible without being an exact reproduction? The distinction is evidence. A paper fact is a setting or result explicitly supplied by the paper. An implementation decision is a choice made because the paper leaves a detail open. An ambiguity is an unresolved detail that prevents a unique reconstruction. A reported reference is a value transcribed from the paper, not a result independently produced by this project.

For this paper, the Section 4 comparison is the main reproduction target: SVR, MLPR, KNN, random forest, XG-Boost, LSTM, and the Random Forest + XG-Boost + LSTM ensemble. The generated framework preserves that scope, but it does not claim exact numerical reproduction. The paper does not provide source code, a data snapshot, complete preprocessing instructions, or the ensemble's actual wiring.

What is known and what remains a decision

Paper facts. Table 1 specifies an 80/20 training/test split for the proposed ensemble, MSE and Adam for the neural component, a maximum of 50 epochs, batch size 32, two LSTM layers, dropout 0.2, and a dense layer with 25 units. It also lists candidate values for random-forest and XG-Boost parameters. The paper reports Yahoo Finance data for TAINIWALCHM and AGROPHOS and evaluates models using RMSE and R2.

Derived implementation rationale. The framework uses a chronological split because later observations should represent the held-out period. It fits learned preprocessing and any learned combiner using training data only. It aligns predictions by timestamp because LSTM sequence construction can remove initial rows. These are safeguards and design explanations, not evidence that the original experiment used precisely the same procedures.

Implementation decisions and ambiguities. The exact Yahoo Finance ticker identifiers, selected feature columns, target column, scaler, look-back window, forecast horizon, validation procedure, random seed, selected hyperparameters, and ensemble combiner are not specified. The code therefore exposes them through configuration and records them in ExperimentManifest instead of hiding them in defaults. The generic pipeline also mentions a 75/25 split, which conflicts with Table 1's more specific 80/20 setting; this implementation treats 80/20 as the primary reproduction choice and records the conflict.

The paper's dataset description contains another fidelity issue: AGROPHOS is associated with the 2018–2023 period, but one sentence incorrectly uses the TAINIWALCHM name for that description. The framework does not silently repair the source text. The stock label and the API ticker remain explicit fields so a user can document the chosen interpretation.

The manifest as an audit boundary

The manifest separates a stock label from the data-source ticker and stores preprocessing, training, tuning, and ensemble choices. A compact excerpt from src/stock_forecasting/experiments/schema.py shows the fields that prevent an apparently reproducible run from becoming an undocumented guess:

@dataclass(frozen=True, slots=True)
class ExperimentManifest:
    """Audit record for one stock-specific reproduction configuration.

    The manifest deliberately stores unresolved paper details as explicit fields or
    notes rather than hiding them in defaults.  ``model_configurations`` records
    selected settings when supplied; an empty mapping means that no selected model
    settings were provided by the caller or recovered from the paper.
    """

    paper_id: str
    stock: str
    data_source: Mapping[str, Any]
    feature_columns: tuple[str, ...]
    target_column: str
    train_fraction: float
    split_policy: str
    scaler_name: str | None
    lookback: int
    horizon: int
    seed: int | None
    tuning_method: str
    validation_fraction: float | None
    model_configurations: Mapping[str, Any]
    combiner: str
    ensemble_weights: tuple[float, ...] | None
    meta_features: str

The important point is not the dataclass syntax itself. feature_columns and target_column make the supervised task explicit; lookback and horizon identify the temporal construction; scaler_name and train_fraction expose preprocessing choices; and tuning_method, model_configurations, combiner, and ensemble_weights record choices the paper does not recover. validate_manifest is responsible for rejecting incomplete or invalid records, such as an empty target name, an invalid fraction, duplicate features, or an ensemble weight vector with the wrong length.

A compact fidelity ledger for one hypothetical run could therefore read as follows:

Paper fact: Table 1 specifies an 80/20 split and a 25-unit dense layer.
Implementation decision: use Close as target and Open, High, Low, Volume as features.
Unresolved ambiguity: the paper does not define whether the ensemble is weighted blending or learned stacking.
Reported reference: Table 2 lists the ensemble's AGROPHOS RMSE as 1.2658 and R2 as 0.9897; this value is not independently verified here.

This record keeps a chosen target separate from a paper-specified training ratio, and it keeps a transcribed number separate from an observed output of the generated code.

Damaged and incomplete equations

The equation registry in src/stock_forecasting/equations.py is deliberately conservative. For KNN, eq_2 has an empty canonical LaTeX string because the extracted Euclidean-distance expression is damaged. The code may implement a vector distance for the KNN adapter, but that is an implementation decision and must not be presented as reconstructed paper notation. The registry preserves the empty value:

"eq_2": CanonicalEquation(
    equation_id="eq_2",
    latex="",
    meaning="Euclidean-distance expression for KNN.",
),

The CanonicalEquation record stores an identifier, the supplied LaTeX text, and its meaning. get_equation retrieves that immutable record without normalizing OCR artifacts. This makes eq_2 traceable while respecting the rule that missing mathematical text must not be invented.

The paper's LSTM gate records are intact enough to preserve as supplied, but they do not define a complete LSTM recurrence. They omit the candidate-cell computation and the cell-state and hidden-state updates. The framework therefore uses a Keras LSTM for the complete trainable model and treats the following records as conceptual gate mappings only.

The input-gate record is introduced here to identify the information-control operation printed by the paper:

Here, iga is the input-gate activation, σ is an elementwise sigmoid, Wip is the printed input-gate weight matrix, ht−1 is the previous hidden state, Xc is the current input as printed, and bi is the input-gate bias. The paper does not specify matrix dimensions, so these symbols represent compatible vectors, tensors, and parameters rather than a recoverable complete shape contract. In code, input_gate_eq_8 names the mapping to the framework LSTM concept; it does not claim to hand-implement the missing cell equations.

The forget-gate record describes the corresponding gate that controls retained information:

Here, fga is the forget-gate activation, σ is the sigmoid function, Wf g is the printed forget-gate weight matrix, ht−1 is the previous hidden state, Xc is the current input as printed, and b f is the forget-gate bias. The extraction has damaged subscript and multiplication formatting, and the equation still omits the subsequent cell-state update. The generated forget_gate_eq_9 function therefore provides traceability to the paper record rather than a claim of full mathematical reconstruction.

The output-gate record describes the gate controlling the exposed hidden representation:

Here, Opg is the output-gate activation, σ is sigmoid, Wop is the output-gate weight matrix, ht−1 is the previous hidden state, Xc is the current input as printed, and bo is the output-gate bias. As with the other two records, dimensions and the remaining recurrent updates are not supplied. The generated output_gate_eq_10 function names this correspondence while the actual model is delegated to the Keras implementation.

The important fidelity rule is to preserve these three strings exactly, including their extracted notation, while explaining that they are incomplete. Extending them into a standard textbook LSTM formula would change the supplied record and would falsely imply that the paper specified the missing steps.

Ensemble terminology and reporting limits

The paper calls the Random Forest + XG-Boost + LSTM model “stacked,” but it also discusses assigning greater weight to lower-loss models. These are not automatically the same operation. Weighted averaging, also called a form of blending, combines component predictions using explicit weights. Stacking normally trains a second-stage model, or meta-model, on component predictions. The generated framework exposes both alternatives and records the selected combiner; neither is presented as the recovered paper implementation.

The same caution applies to the reported metrics. RMSE is an error measure in the target's units, while R2 is the coefficient of determination. They are meaningful only when predictions and targets refer to the same timestamps and scale. Table 2 values are reported references. In particular, the unusually large random-forest RMSE values are preserved rather than corrected: the supplied material does not establish whether they reflect scaling, evaluation inconsistency, or a reporting issue.

The paper's conclusions do not establish a universal forecasting method. Stock relationships can change across securities, market regimes, date ranges, data revisions, and preprocessing choices. Accordingly, a result from one period should not be treated as a general guarantee. The paper also frames AI predictions as support for investment decisions rather than their sole determinant; that limitation remains relevant even when a model produces favorable retrospective metrics.

What would improve fidelity

Exact reproduction would require the original Yahoo Finance ticker identifiers and a fixed data snapshot, including interval and endpoint behavior. It would also require the selected feature and target columns, null and zero handling, scaler and inverse-transform policy, sequence look-back and horizon, validation and tuning procedure, selected model parameters, random seeds, package versions, complete LSTM settings, and the actual ensemble wiring. Source code or an original experiment manifest would resolve much of this uncertainty.

Under the current run policy, no API call, training run, test execution, syntax check, or semantic code verification was performed. The generated tests and ledger are planned safeguards, not evidence that the implementation matches Table 2. The responsible interpretation is therefore precise: the framework reproduces the paper's documented model families and stated configuration elements while making its unresolved choices explicit; it does not establish independently verified financial-forecasting results.

Static and semantic verification plan

How can we tell whether this reproduction framework respects the paper's assumptions without pretending that the experiment has already run? Separate four activities: static verification, semantic verification, testing, and execution. Static verification inspects structure such as syntax, imports, names, and configuration values. Semantic verification reviews whether the code's behavior is consistent with the intended method, including time alignment and leakage boundaries. Tests encode repeatable checks for those contracts. Execution actually runs the package, trains models, calls data sources, or produces metrics.

For this project, execution was disabled by the run policy. Local static verification and code semantic verification were also skipped. Therefore, the checks described below are planned invariants and test specifications, not observed passes. In particular, no test, syntax check, API call, model-training run, or numerical reproduction of Table 2 was performed.

What the planned checks protect

The generated tests focus on six categories:

1. Rank and shape. Tabular features must remain rank 2, sequence inputs rank 3, and predictions must have one value per aligned observation.

2. Time order. Cleaning must produce ordered, unique timestamps; chronological splits must not overlap; and test observations must follow training observations.

3. Sequence alignment. Every look-back window must contain observations before its target timestamp. Initial rows lost during window construction must not be silently paired with another model's predictions.

4. Leakage control. Learned preprocessing state, candidate selection, and any ensemble meta-model must use training data only.

5. Finite outputs and metric alignment. Predictions and targets must have equal lengths, matching timestamps, and finite numeric values before RMSE or R2 is computed.

6. Paper traceability. Canonical equation text and Table 1 candidate grids must be preserved exactly, while damaged equations must remain unreconstructed.

These are implementation constraints derived from the method cards, not additional results reported by the paper.

Data and temporal invariants

The data tests in tests/test_data_invariants.py express the earliest contracts. For example, the cleaning test checks ordering, uniqueness, duplicate handling, and the retained final duplicate value:

def test_clean_table_orders_and_deduplicates() -> None:
    """Cleaning sorts timestamps and keeps the final occurrence of duplicates."""
    timestamps = pd.to_datetime(
        ["2024-01-03", "2024-01-01", "2024-01-02", "2024-01-02"]
    )
    raw = pd.DataFrame(
        {
            "Open": [3.0, 1.0, 2.0, 20.0],
            "Close": [3.5, 1.5, 2.5, 25.0],
            "Volume": [30.0, 10.0, 20.0, 200.0],
        },
        index=pd.DatetimeIndex(timestamps, name="Date"),
    )

    cleaned = clean_stock_table(raw, remove_zero_values=False)

    assert cleaned.index.is_monotonic_increasing
    assert cleaned.index.is_unique
    assert list(cleaned.index) == list(pd.to_datetime(["2024-01-01", "2024-01-02", "2024-01-03"]))
    assert cleaned.loc[pd.Timestamp("2024-01-02"), "Close"] == pytest.approx(25.0)
    assert cleaned.shape == (3, 3)

The important point is not the particular fixture dates. It is that clean_stock_table must make duplicate and ordering behavior explicit. The paper says to remove redundancies and nulls, but it does not prescribe this exact duplicate policy; the test documents the generated implementation's chosen contract.

The chronological split test checks the primary 80/20 reproduction decision:

def test_chronological_split_has_no_overlap() -> None:
    """The split preserves order and places every test observation after training."""
    index = pd.date_range("2024-01-01", periods=10, freq="D", name="Date")
    X = pd.DataFrame({"feature": np.arange(10, dtype=float)}, index=index)
    y = pd.Series(np.arange(10, dtype=float) + 100.0, index=index, name="target")

    X_train, X_test, y_train, y_test = chronological_split(X, y, train_fraction=0.8)

    assert len(X_train) == len(y_train) == 8
    assert len(X_test) == len(y_test) == 2
    assert X_train.index.equals(y_train.index)
    assert X_test.index.equals(y_test.index)
    assert set(X_train.index).isdisjoint(set(X_test.index))
    assert X_train.index[-1] < X_test.index[0]

This protects both shape and chronology. It does not establish that 80/20 is the only correct split: the paper's generic pipeline mentions 75/25, while Table 1 specifies 80/20 for the proposed ensemble. The test simply records which policy the reproduction framework has chosen as its primary target.

Sequence and scaling invariants

An LSTM consumes a rank-3 array with shape (batch, sequence_length, n_features). The sequence test checks that a look-back window precedes its future target and that the target index is retained:

def test_sequences_preserve_target_alignment() -> None:
    """Each sequence contains only observations before its future target."""
    index = pd.date_range("2024-01-01", periods=8, freq="D", name="Date")
    X = np.arange(8, dtype=float).reshape(-1, 1)
    y = (100.0 + np.arange(8, dtype=float)).reshape(-1, 1)

    result = make_sequences(X, y, index, lookback=3, horizon=2)

    # The first target is position lookback + horizon - 1 = 4.
    assert result.X_train.ndim == 3
    assert result.X_train.shape[1:] == (3, 1)
    assert result.X_train[0, :, 0].tolist() == [0.0, 1.0, 2.0]
    assert result.y_train[0] == pytest.approx(104.0)
    assert result.train_index[0] == index[4]

The paper does not supply the look-back or horizon. Those are implementation decisions, so the test verifies the chosen sequence-construction semantics rather than a paper-specified numerical setting. This invariant is essential to preprocess_time_series and to stacked_rf_xgboost_lstm: tree models may produce predictions for rows that the LSTM cannot represent until enough history exists.

Scaling has a separate leakage constraint. A scaler is a learned transformation, so it must be fitted on training rows only. The planned test checks that the stored means come from X_train and y_train, not from the deliberately extreme test values:

def test_scaler_fit_scope_is_training_only() -> None:
    """The fitted preprocessing state records only training rows and excludes test values."""
    X_train = np.array([[1.0], [3.0], [5.0]], dtype=float)
    y_train = np.array([10.0, 20.0, 30.0], dtype=float)
    X_test = np.array([[1000.0]], dtype=float)
    y_test = np.array([10000.0], dtype=float)

    state = fit_scalers(X_train, y_train, scaler_name="standard")

    assert state.fit_rows == X_train.shape[0]
    assert state.feature_scaler.mean_.tolist() == pytest.approx([3.0])
    assert state.target_scaler.mean_.tolist() == pytest.approx([20.0])

The paper does not specify the scaler type, whether targets are scaled, or whether reported metrics use original prices. The invariant therefore concerns fit scope, not a particular scaler choice. Any inverse transformation used before reporting must preserve target alignment and scale consistency.

Equation traceability without reconstruction

Equation traceability has a special rule in this project: the supplied canonical equation records are authoritative. Equation eq_2, the KNN distance expression, has empty canonical LaTeX because the source extraction is damaged. The implementation may provide the named euclidean_distance_eq_2 adapter as an explicit vector-distance decision, but it must not present reconstructed LaTeX as if it came from the paper.

The same rule applies to the LSTM gate records. The supplied canonical text for eq_8 describes the input-gate activation as printed:

The input-gate record is preserved exactly below.

Here iga is the input-gate activation, σ is the elementwise sigmoid function, Wip is the printed input-gate weight matrix, ht−1 is the previous hidden state, Xc is the current input notation used in the extracted equation, and bi is the input-gate bias. These are vector or tensor quantities in an implementation, although the paper does not provide dimensions. The generated input_gate_eq_8 function is a traceable helper name, not evidence that the incomplete printed expression defines the entire recurrent cell.

The forget-gate record is preserved exactly as eq_9.

Here fga is the forget-gate activation, σ is again applied elementwise, Wf g is the printed forget-gate weight matrix, ht−1 is the previous hidden state, Xc is the current input, and b f is the forget-gate bias. The corresponding code symbol is forget_gate_eq_9.

The output-gate record is preserved exactly as eq_10.

Here Opg is the output-gate activation, Wop is the output-gate weight matrix, ht−1 is the previous hidden state, Xc is the current input, and bo is the output-gate bias. The generated mapping is named output_gate_eq_10.

The three records describe individual gate activations only. They omit the candidate-cell computation, cell-state update, and hidden-state update. The framework-level Keras LSTM therefore supplies the complete recurrent behavior; the code does not hand-reconstruct unsupported equations. The planned traceability test checks exact strings and function names:

def test_damaged_equations_remain_empty() -> None:
    """Ensure damaged source records are not silently reconstructed in the registry."""
    for equation_id in ("eq_2", "eq_4", "eq_6"):
        assert get_equation(equation_id).latex == ""


def test_lstm_methods_reference_eq_8_eq_9_eq_10() -> None:
    """Check that the three conceptual gate adapters are exposed as named callables."""
    gate_functions = {
        "eq_8": input_gate_eq_8,
        "eq_9": forget_gate_eq_9,
        "eq_10": output_gate_eq_10,
    }

    for equation_id, function in gate_functions.items():
        assert callable(function)
        assert equation_id in function.__name__

These checks are planned evidence-preservation checks. They do not numerically validate an LSTM and do not turn incomplete equations into a complete mathematical specification.

Model-contract and paper-fidelity checks

The model-contract tests target the interfaces used by fit_models_and_generate_predictions. The LSTM contract is rank 3 in and one regression output per sequence:

def test_lstm_uses_rank_three_inputs_and_single_output() -> None:
    """The configured neural model has (batch, sequence, feature) input and (batch, 1) output."""
    pytest.importorskip("tensorflow")

    model = build_lstm_regressor(
        input_shape=(6, 4),
        units=5,
        dropout=0.2,
        dense_units=25,
    )

    assert model.input_shape == (None, 6, 4)
    assert model.output_shape == (None, 1)
    assert sum(layer.__class__.__name__ == "LSTM" for layer in model.layers) == 2

This checks the stated two-layer architecture, dropout setting, dense width, and shape contract when TensorFlow is available. It does not check training quality or assert that the unspecified recurrent-unit count is the paper's count. Similarly, test_paper_candidate_grids_match_table_1 is intended to compare the random-forest and XG-Boost candidate values with Table 1. Candidate values are not selected hyperparameters, and this check does not imply that the paper used the same search procedure as the generated code.

Ensemble alignment and leakage checks

The ensemble is the area where semantic review matters most. Matching array shapes alone cannot prove that random-forest, XG-Boost, and LSTM predictions refer to the same timestamps. The alignment test deliberately gives each component different order and coverage, then requires an intersection ordered by the target:

def test_component_predictions_align_by_timestamp() -> None:
    """Component predictions use the target order and only common timestamps."""
    target_index = _index(0, 1, 2, 3)
    target = PredictionBundle("target", target_index, np.array([10.0, 20.0, 30.0, 40.0]))

    random_forest = PredictionBundle(
        "random_forest",
        _index(3, 1, 0),
        np.array([39.0, 19.0, 9.0]),
    )
    xgboost = PredictionBundle(
        "xgboost",
        _index(2, 0, 3),
        np.array([29.0, 11.0, 41.0]),
    )
    lstm = PredictionBundle("lstm", _index(0, 2, 3), np.array([10.5, 30.5, 40.5]))

    aligned = align_prediction_bundles((random_forest, xgboost, lstm), target)

    expected_index = _index(0, 3)
    np.testing.assert_array_equal(aligned.index, expected_index)
    np.testing.assert_allclose(
        aligned.predictions,
        np.array([[9.0, 11.0, 10.5], [39.0, 41.0, 40.5]]),
    )

The resulting component matrix has shape (n_aligned_samples, 3), with columns for random forest, XG-Boost, and LSTM. A second planned check verifies that a linear stacker is fit from component predictions and training targets only. This is especially important because the paper calls the ensemble “stacked” but also discusses higher weights for lower-loss models. Weighted averaging and learned stacking are different operations, and neither wiring is recovered from the paper.

Worked verification checklist

For one prepared dataset and one ensemble output, the intended review checklist is:

| Invariant | Planned question | Evidence location | |---|---|---| | Rank | Is tabular X rank 2, sequence X rank 3, and each prediction vector aligned? | tests/test_model_contracts.py | | Time order | Are timestamps unique, chronological, and non-overlapping across train and test? | tests/test_data_invariants.py | | Window safety | Does every LSTM window precede its target? | test_sequences_preserve_target_alignment | | Scaling leakage | Was preprocessing state fitted only on training rows? | test_scaler_fit_scope_is_training_only | | Equation fidelity | Are intact strings unchanged and eq_2 still empty? | tests/test_equation_traceability.py | | Table 1 fidelity | Do RF and XG-Boost candidate grids match the supplied values? | test_paper_candidate_grids_match_table_1 | | Ensemble alignment | Do all three components share the same target timestamps? | test_component_predictions_align_by_timestamp | | Meta-model leakage | Was a learned combiner fit without held-out test targets? | test_stacker_fit_excludes_test_targets | | Metric inputs | Do y_true and y_pred have equal aligned lengths? | test_metric_inputs_must_align |

This checklist labels intended obligations, not completed results. A future verification run could use it to review one synthetic or local-CSV experiment before considering a real Yahoo Finance reproduction.

Verification boundary and final caution

The planned tests are contract tests, not evidence that the paper's reported RMSE and R2 values have been reproduced. Synthetic data and local CSV data can exercise the pipeline, but they are not substitutes for the paper's unavailable Yahoo Finance snapshot and unspecified preprocessing and stacking decisions. Likewise, a successful model fit would not establish that Table 2 was independently verified.

Advanced detail. A learned stacker can introduce a second form of leakage if its training predictions were generated on the same rows used to fit the component models. A robust stacking design often uses out-of-fold component predictions, but the paper does not specify such a procedure. The generated framework therefore records the combiner choice and its training data boundary rather than claiming that the paper recovered this detail.

The honest conclusion is narrow: the repository defines useful invariants for preprocess_time_series, fit_models_and_generate_predictions, stacked_rf_xgboost_lstm, and evaluate_regression_predictions, but those invariants were not executed or independently reviewed in this pipeline. Treat them as a verification plan to apply before using any resulting metrics, especially for financial decisions.

Use the button or URL below to download the source code.

Read more

The paper estimates Kyle's price-impact coefficient from CRSP daily equity data and tests whether order-flow-based measures forecast contemporaneous and subsequent monthly stock returns. It constructs signed order flow, total volume, volume volatility, a within-month price-impact regression estimator, and an Amihud-style illiquidity estimator. The theoretical interpretation is that adverse selection and temporary price impact, rather than only risk compensation, can generate an illiquidity premium.

Implementation Assumptions

• Required CRSP, H.15, WRDS, factor, and delisting datasets are supplied locally as files; no network access or credentials are used.

• PERMNO and calendar-month keys are the canonical identifiers throughout the empirical pipeline.

• The default sample period is 2020–2025 when the available data support it.

• The default expanding-window interpretation is used because the procedural description explicitly specifies expansion, despite occasional use of the word rolling.

• The default empirical lambda regression is uncentered because Equation 12 displays no intercept; this choice is exposed in configuration because the paper is ambiguous.

• Zero daily price changes receive sign zero.

• Firm-month winsorization scope is configurable and never silently assumed.

• Newey–West lag length, covariance treatment, portfolio weighting, and missing-data rules are explicit configuration fields because they are unspecified in the supplied paper extraction.

• OCR-damaged equations eq3, eq6, eq8, and eq16 are not reconstructed beyond the supplied meanings and prose.

• Reported paper coefficients and summary statistics are stored as comparison targets, not as independently reproduced results.

• No code execution, numerical verification, or semantic validation is claimed under the supplied run policy.

Scope, evidence boundaries, and pipeline architecture

What would it take to turn the paper’s liquidity and return-predictability tests into a reproducible Python workflow? The practical answer is not one model call. It is a chain of contracts: local daily records are cleaned, daily activity is summarized by firm and calendar month, estimators are aligned with subsequent returns, and each analysis produces labeled outputs with provenance.

The paper studies whether observable trading activity contains information about contemporaneous and future stock returns. Its structural motivation comes from Kyle-style price impact: informed and noise trading combine into signed aggregate order flow, and a market maker adjusts prices in response. The empirical data do not reveal those latent theoretical orders separately. Instead, the reproduction uses signed daily volume as an observable proxy, alongside unsigned volume, volume volatility, and two firm-month illiquidity or price-impact estimators.

This distinction is central. The theoretical aggregate flow is a model quantity; the empirical signedflow column is constructed from CRSP observations. Likewise, lambda is not one universally defined field. Its meaning depends on whether it was produced by the within-month price-impact regression or by the Amihud-style average.

The evidence boundary

The supplied paper describes a principal sample covering 2020–2025 and reports coverage and regression results, including firm and firm-month counts. Those values are paper-reported comparison targets, not independently reproduced results in this workflow. A full numerical reproduction requires local CRSP data and, for particular analyses, H.15 risk-free rates, WRDS book-to-market and factor data, and possibly delisting-return and point-in-time membership files.

The implementation therefore follows three categories:

• Paper facts: the stated sample period, exchange filter, minimum 15 nonzero-volume-day firm-month rule, daily-to-monthly feature definitions, and the planned regression and forecast families.

• Implementation decisions: the exact adjustment-field convention, unchanged-price sign handling, winsorization scope, covariance treatment, forecast pooling, Newey–West lag, and portfolio weighting.

• Unavailable evidence: missing source data, incomplete adjustment instructions, and OCR-damaged or underspecified parts of the paper.

The code keeps the second category in configuration rather than silently choosing hidden behavior. It also avoids filling missing data with paper-reported values. The package is local-only: it reads files supplied by the user and does not access a network or credentials.

The pipeline as an accounting ledger

The canonical identifiers are PERMNO, the CRSP firm identifier, and month, a normalized calendar-month key. The intended data flow is:

daily CRSP rows
    -> filtered and adjusted daily rows
    -> daily dollar volume, signed flow, and Amihud ratios
    -> one row per (PERMNO, month)
    -> lambda estimates, returns, and controls
    -> contemporaneous and next-month model inputs
    -> regressions, forecasts, portfolios, and reports

Every transition should answer four questions:

1. What source columns were used?

2. What is the resulting row shape and key?

3. Which time period does each value describe?

4. Which convention controls missing or ambiguous values?

The generated ReproductionConfig records many of these choices. Its defaults are explicitly described as conventions rather than uniquely paper-determined facts:

@dataclass(frozen=True)
class ReproductionConfig:
    """Validated settings for local data preparation and empirical analyses.

    Defaults reflect implementation decisions made where the supplied paper
    extraction is ambiguous: a 2020--2025 sample, exchanges 1/2/3, a
    15-nonzero-volume-day filter, an uncentered lambda estimator, zero sign for
    unchanged prices, monthly winsorization, equal-weighted deciles, and no
    assumed covariance correction.  These defaults are not claimed to be the
    paper's uniquely specified conventions.
    """

    sample_start: DateLike = "2020-01-01"
    sample_end: DateLike = "2025-12-31"

validate_config checks ranges and supported options, but it does not establish that a local file exists or that its contents match the paper’s sample. That separation matters: configuration validation is not empirical verification.

What each orchestration layer does

build_empirical_panel(config) is the main preprocessing boundary. It loads the configured CRSP daily and monthly files, applies exchange and date filters, constructs adjusted prices and daily features, aggregates firm-month variables, estimates both lambda columns, attaches available controls, applies configured winsorization, and creates same-firm next-month targets. Its output should contain at most one row for each PERMNO-month pair.

run_empirical_reproduction(panel, config) receives that normalized panel rather than loading data itself. It isolates model-specific inputs and delegates to the empirical modules:

• pooled contemporaneous and one-month-ahead return regressions;

• lambda-return regressions for both estimators and intercept variants;

• firm-level expanding-window forecasts and pooled forecast evaluation;

• monthly Fama–MacBeth regressions and signal-sorted decile portfolios.

The reporting layer, build_reproduction_report(panel, results, config), assembles coverage, descriptive, regression, forecast, and portfolio tables. Its design keeps computed outputs separate from paper targets. A target annotation can describe a comparison value, but it must not overwrite a locally computed coefficient or observation count.

The two command-line entry points reflect this separation. scripts/build_panel.py constructs auditable daily and firm-month artifacts. scripts/run_reproduction.py builds or loads a panel, runs the model pipeline, and writes report tables. Neither command is evidence that execution occurred here; they are local interfaces for a future run with suitable data.

A small worked example: one firm through the ledger

Consider a synthetic firm with PERMNO equal to 10001 and three daily observations in January. The rows might contain the following conceptual fields

The adjustment module first produces an auditable adj_price and then a firm-local price_change. The daily-feature functions use those rows to create dollar_volume, signed_flow, and amihud_ratio. Under the configured zero-change convention, the unchanged observation contributes zero signed flow rather than being forced into a buy or sell direction. The upward and downward observations contribute signed volume with opposite directions.

All three rows receive the same calendar month key. If the firm has at least the configured minimum number of nonzero-volume days in the actual input, build_firm_month_features reduces the daily rows to one firm-month row. That row receives the sum of unsigned volume, the sample standard deviation of daily volume, and the sum of daily signed flow. The Amihud estimator uses only daily rows with valid returns and strictly positive dollar volume, so its valid-day count can differ from the count used for other features.

Later modules consume these columns differently. The panel regressions use the current firm-month volume features to explain either the current return or the same firm’s next calendar-month return. The lambda-return regressions use one current-month lambda column at a time. Forecasting uses the available current information inside an expanding chronological window. Portfolio formation ranks on a configured current-information signal and only then associates the resulting portfolios with subsequent returns.

This example explains data shapes and timing; it is not a paper calibration, an executed calculation, or a reproduced result.

Evidence-preserving interpretation

The paper contains unresolved contradictions that the pipeline should preserve rather than harmonize silently. Its theoretical discussion gives inconsistent comparative-static descriptions involving noise-trading variance and lambda. Its narrative expectations for signed-flow coefficients do not always agree with reported one-month-ahead coefficient signs. Section and table references are also inconsistent in the supplied extraction.

These issues affect interpretation, but they do not justify changing observed or reported signs in code. Result tables should retain specification labels, estimator names, sample counts, and provenance so that differences can be inspected directly.

The same principle applies to damaged equations. No formula is reconstructed when the supplied canonical record is incomplete or OCR-damaged. The implementation can document a prose-defined behavior or expose an explicit configuration choice, but it should not present an inferred formula as though it came from the paper.

Finally, all verification flags are disabled under the run policy. Static verification, semantic code verification, execution, test generation, and final quality review were not performed. The planned invariants—unique firm-month keys, same-firm target alignment, finite denominators, nondecreasing forecast windows, and separation of paper targets from computed outputs—are useful design requirements, not checks claimed to have passed.

Local data contracts and preprocessing decisions

Before estimating liquidity or return predictability, which observations should count as comparable trading records? The reproduction answers that question with a data contract: an explicit agreement about column names, types, keys, filters, missing values, and external inputs. Configuration keeps uncertain choices visible instead of burying them inside preprocessing functions.

The paper’s empirical sample is based on CRSP daily equity data, with a stated main period of 2020–2025 and a default exchange filter for CRSP exchange codes 1, 2, and 3. The pipeline then creates monthly firm observations. These are paper-supported design elements. By contrast, the exact application of CRSP adjustment factors, return conventions, delisting-return treatment, winsorization scope, and inference settings is not fully specified. Those choices must remain configurable.

Required local inputs

The normalized daily table has one row for one PERMNO on one trading date. PERMNO is the CRSP firm identifier, and the daily uniqueness key is the pair (PERMNO, date). The required source fields are:

• PERMNO: firm identifier.

• date: trading date.

• EXCHCD: CRSP exchange code.

• DlyPrc: daily price.

• DlyVol: daily share volume.

• DlyRet: daily stock return.

• ShrOut: shares outstanding.

• DisFacPr: CRSP price-adjustment field.

• DisFacShr: CRSP share-adjustment field.

DlyBid and DlyAsk may also be retained when available, but the core volume and lambda constructions do not require them.

The monthly inputs use a different key. Firm-level monthly returns and controls use (PERMNO, month), where month is normalized to a calendar-month timestamp. Aggregate risk-free data use month alone. The paper also identifies H.15 Treasury-bill data, WRDS book-to-market and factor data, and possible delisting and point-in-time membership information for particular analyses. These files are external requirements; they cannot be inferred from CRSP daily rows or fabricated by the loader.

The daily loader is local-only. Its public responsibility is to read a supported file, normalize types and dates, preserve source fields, sort chronologically, and reject duplicate firm-date observations. A focused excerpt shows the boundary check:

def validate_crsp_daily_columns(
    frame: pd.DataFrame, columns: DataColumnConfig
) -> None:
    """Validate the normalized CRSP daily schema and key invariants.

    The required schema contains PERMNO, date, EXCHCD, DlyPrc, DlyVol, DlyRet,
    ShrOut, DisFacPr, and DisFacShr under the names in ``columns``.  Bid and ask
    are deliberately optional because the paper identifies them as optional
    inputs and the core feature pipeline does not require them.
    """
    required = columns.required_daily_columns()
    missing = [name for name in required if name not in frame.columns]
    if missing:
        raise ValueError(f"CRSP daily frame is missing required columns: {missing}")

    if not isinstance(frame.index, pd.RangeIndex):
        # Index type is not part of the data contract; reset-indexed input is
        # normalized by the loader, while direct callers receive a clear error.
        if frame.index.has_duplicates:
            raise ValueError("CRSP daily frame index must not contain duplicates")

    if frame[columns.permno].isna().any():
        raise ValueError("PERMNO contains missing values")
    if frame[columns.date].isna().any():
        raise ValueError("date contains missing values")

    permno = pd.to_numeric(frame[columns.permno], errors="coerce")
    if permno.isna().any() or ((permno % 1) != 0).any():
        raise ValueError("PERMNO must contain integer-like values")
    if not pd.api.types.is_datetime64_any_dtype(frame[columns.date]):
        raise TypeError("date must be datetime-like after normalization")

    duplicate_keys = frame.duplicated(
        subset=[columns.permno, columns.date], keep=False
    )
    if bool(duplicate_keys.any()):
        raise ValueError("CRSP daily input contains duplicate PERMNO-date observations")

validate_crsp_daily_columns receives a pandas DataFrame and a DataColumnConfig. Its output is None when the schema is acceptable; malformed schemas raise an exception rather than silently dropping or repairing fields. The loader itself returns a table with one normalized row per firm-date observation. It does not establish that the source data have the same point-in-time or corporate-action treatment as the paper.

Configuration makes ambiguity explicit

DataColumnConfig maps source names to the fields used downstream. ReproductionConfig stores sample dates, local paths, filters, adjustment conventions, winsorization, covariance, forecasting, and portfolio choices. The following is a compact local configuration example copied from the generated tutorial:

from pathlib import Path

from liquidity_premium.config import ReproductionConfig

config = ReproductionConfig(
    sample_start="2020-01-01",
    sample_end="2025-12-31",
    daily_path=Path("data/crsp_daily.parquet"),
    monthly_returns_path=Path("data/monthly_returns.parquet"),
    risk_free_path=Path("data/risk_free_rates.csv"),
    auxiliary_monthly_path=Path("data/auxiliary_monthly.parquet"),
    output_dir=Path("outputs"),
)

The defaults additionally specify exchange codes (1, 2, 3), a minimum of 15 nonzero-volume days, the price_only adjustment convention, monthly winsorization, an uncentered lambda estimator, equal-weighted deciles, and no fixed Newey–West lag unless one is supplied. These are not all paper facts. The period, exchange codes, and 15-day rule are supported by the supplied paper description; the adjustment, winsorization, covariance, and portfolio defaults resolve underspecified choices in the implementation.

For example, a run can make those choices more visible:

config = ReproductionConfig(
    sample_start="2020-01-01",
    sample_end="2025-12-31",
    daily_path=Path("data/crsp_daily.parquet"),
    monthly_returns_path=Path("data/monthly_returns.parquet"),
    risk_free_path=Path("data/risk_free_rates.csv"),
    auxiliary_monthly_path=Path("data/auxiliary_monthly.parquet"),
    adjustment_convention="price_only",
    winsorization_scope="month",
    newey_west_lags=4,
    portfolio_weighting="equal",
)

The value 4 here is an implementation example, not a lag length specified by the paper. The paper does not provide an exact Newey–West lag. Similarly, price_only is a selected convention, not a uniquely established interpretation of DisFacPr and DisFacShr.

Filter order and the 15-day rule

The intended order is to load and normalize the daily data, restrict the sample period and exchanges, construct the calendar-month key, and apply the minimum-volume-day rule before aggregation. The paper’s default exchange filter retains codes 1, 2, and 3. The minimum-volume filter retains a firm-month when it contains at least 15 finite, strictly positive DlyVol observations.

The generated filter is deliberately a row filter: it returns the daily observations belonging to qualifying groups, leaving aggregation to a later module.

def filter_minimum_volume_days(
    frame: pd.DataFrame,
    minimum_days: int = 15,
) -> pd.DataFrame:
    """Retain firm-month groups with enough nonzero-volume trading days.

    A qualifying day has finite, strictly positive ``DlyVol``.  The count is
    computed independently for each ``PERMNO`` and calendar month, matching
    the paper's minimum-15-nonzero-volume-day filter.  This function is
    deliberately a row filter: aggregation and feature construction occur in
    the downstream aggregation module.  The original columns and row order
    are preserved for surviving observations.
    """
    _require_columns(frame, ("PERMNO", "date", "DlyVol"))
    if not isinstance(minimum_days, int) or isinstance(minimum_days, bool):
        raise TypeError("minimum_days must be an integer")
    if minimum_days < 1:
        raise ValueError("minimum_days must be positive")

    dates = _coerce_dates(frame["date"], "date")
    volume = pd.to_numeric(frame["DlyVol"], errors="coerce")
    valid_volume = volume.notna() & volume.gt(0) & volume.ne(float("inf")) & volume.ne(float("-inf"))

    month_key = dates.dt.to_period("M")
    group_keys = pd.DataFrame(
        {"PERMNO": frame["PERMNO"], "_month": month_key},
        index=frame.index,
    )
    counts = valid_volume.groupby([group_keys["PERMNO"], group_keys["_month"]], dropna=False).sum()
    row_counts = pd.MultiIndex.from_arrays(
        [group_keys["PERMNO"], group_keys["_month"]],
        names=["PERMNO", "_month"],
    )
    qualifying = counts.ge(minimum_days)
    mask = qualifying.reindex(row_counts, fill_value=False).to_numpy()
    return frame.loc[mask].copy()

The function accepts a daily DataFrame and returns a same-column DataFrame containing only rows from qualifying firm-month groups. It does not create a monthly row or calculate a return. A group with missing, infinite, or zero volume does not count that observation toward the threshold. The resulting aggregation must later enforce uniqueness of (PERMNO, month).

Adjustment fields and return conventions

The paper requires split-adjusted prices and names DisFacPr and DisFacShr, but the supplied extraction does not give an unambiguous formula for applying both fields. The generated compute_adjusted_price function therefore exposes three conventions: raw, price_only, and price_and_shares. It preserves the raw adjustment columns instead of overwriting them.

This distinction matters because adjusted prices feed later price changes, signed-flow directions, and potentially dollar-volume calculations. The selected convention can change derived variables, so it belongs in configuration and provenance.

validate_adjustment_factors checks that the adjustment fields exist and are finite and strictly positive under a factor-based convention. compute_adjusted_price returns a float-compatible series aligned with the input rows. Invalid adjusted prices raise an error rather than becoming silently usable values. These checks constrain the input but do not prove that a chosen convention matches CRSP documentation or the paper’s intended processing.

Monthly returns require a separate decision. load_monthly_returns normalizes the firm-month key and exposes a recognized return field as StockRet, but it does not automatically combine delisting returns or convert raw returns to excess returns. load_risk_free_rates normalizes aggregate monthly rates as risk_free_rate, but does not convert annualized yields to monthly returns. Those transformations depend on source units and conventions that the paper does not state consistently across all analyses.

This separation prevents a raw return, a delisting-adjusted return, and an excess return from being treated as interchangeable. Each model input should identify which convention it uses.

Monthly auxiliary data and duplicate protection

The monthly loaders normalize date-like values to month-start timestamps. Thus, dates within the same calendar month map to the same month key. Firm-month auxiliary data must not contain duplicate (PERMNO, month) rows, while aggregate risk-free data must not contain duplicate month rows. Duplicate keys are rejected rather than silently aggregated, because silent aggregation could conceal a data join error.

The loaders preserve auxiliary columns such as book-to-market or factor returns. They do not invent unavailable values. If WRDS factors, H.15 rates, point-in-time membership, or delisting returns are absent, the corresponding analysis must either be omitted or run with an explicitly documented reduced input set.

Provenance without fabricated evidence

A provenance record documents the configuration and local input paths. It does not claim that the files exist, that they are complete, or that they reproduce the paper’s reported coverage. The generated export function makes this boundary explicit:

def write_provenance(
    path: Path,
    config: ReproductionConfig,
    inputs: list[Path],
) -> None:
    """Write configuration and local input-path metadata for a reproduction run.

    The provenance record documents choices that affect reproducibility, including
    adjustment, winsorization, covariance, forecast, and portfolio conventions.
    It records paths only; it does not assert that external CRSP, H.15, WRDS, or
    delisting data are present, complete, or sufficient to reproduce paper results.
    """
    if not isinstance(config, ReproductionConfig):
        raise TypeError("config must be a ReproductionConfig")
    if not isinstance(inputs, list):
        raise TypeError("inputs must be a list of pathlib.Path values")
    if any(not isinstance(input_path, Path) for input_path in inputs):
        raise TypeError("each input must be a pathlib.Path")

    payload: dict[str, object] = {
        "schema_version": "1.0",
        "record_type": "liquidity_premium_reproduction_provenance",
        "paper_id": "2607.01377v1",
        "paper_title": "Liquidity Premium and Investment Horizons",
        "configuration": asdict(config),
        "input_paths": [str(input_path) for input_path in inputs],
        "notes": [
            "This record describes a local reproduction configuration and does not contain computed results.",
            "External data availability, preprocessing conventions, and unspecified paper choices affect reproducibility.",
            "Paper-reported targets are not represented as independently reproduced values.",
        ],
    }
    write_json(payload, path)

write_provenance receives a configuration and a list of local Path objects and writes metadata through write_json. The important invariant is evidentiary: a paper target, such as the reported firm or firm-month count, must remain a comparison target and must never be inserted into a computed report merely because it is known from the paper.

Worked example: one firm and two keys

Consider a conceptual daily input for one firm, PERMNO=10001, on several dates in January 2020. Each row has DlyPrc, DlyVol, DlyRet, ShrOut, DisFacPr, and DisFacShr, plus EXCHCD=1. After date normalization, all rows receive the same calendar month key, 2020-01-01. The daily key is therefore (10001, date), while the eventual aggregate key is (10001, 2020-01-01).

If one row has missing volume, it does not count toward the 15-day threshold. If a row has zero volume, it also does not count. If the firm has fewer than 15 finite, strictly positive-volume days in January, every January row is removed by filter_minimum_volume_days; no partial firm-month is created. With at least 15 qualifying days, the later aggregation can produce exactly one January row for PERMNO=10001.

The later pipeline assigns responsibilities as follows:

1. load_crsp_daily validates and sorts the source rows.

2. filter_exchange_codes removes observations outside codes 1, 2, and 3.

3. filter_sample_period applies inclusive sample-date bounds.

4. compute_adjusted_price creates an audit-preserving adjusted-price series under the configured convention.

5. Daily feature functions use the adjusted prices, volumes, and returns to create derived columns.

6. The aggregation module converts qualifying daily rows into one firm-month row.

7. Monthly loaders and alignment functions attach returns and controls without crossing firm boundaries.

This example explains keys, timing, and responsibilities only. It is not a CRSP result, and no generated code or numerical output is claimed to have been executed.

What a full reproduction still needs

A complete numerical reproduction depends on local CRSP data and, for relevant specifications, H.15, WRDS, factor, point-in-time, and delisting inputs. It also requires explicit choices for:

• application of DisFacPr and DisFacShr;

• raw, delisting-adjusted, or excess-return conventions;

• winsorization scope and timing;

• covariance and standard-error treatment;

• handling of missing calendar months and external controls;

• portfolio and risk-free-rate conventions.

The paper’s reported coverage and summary values remain paper claims until a run with suitable data produces comparable outputs. Under the authoritative run policy, execution, static verification, semantic code verification, and numerical validation were not performed. The implementation therefore documents a reproducible boundary and a set of explicit decisions, not a completed empirical match.

3. From daily prices and volume to firm-month features

How does a table of daily CRSP observations become the monthly predictors used in the paper? The pipeline first creates row-level quantities—adjusted price, price change, dollar volume, signed flow, and an Amihud ratio—and then compresses those rows into one record per PERMNO and calendar month.

Unsigned volume measures how many shares traded. Signed flow preserves a direction: the same volume contributes positively after an upward price change and negatively after a downward change. An unchanged price contributes zero under the implementation convention. The Amihud-style ratio instead measures absolute return per dollar traded, so a zero-dollar-volume observation has no valid ratio.

Daily inputs and shape flow

The normalized daily frame contains one row per security-date observation. The relevant columns are PERMNO, date, DlyPrc, DlyVol, and DlyRet; DisFacPr and DisFacShr provide the adjustment fields named by the paper. The daily feature functions preserve the input index and add derived columns without changing the number of daily rows.

The intended shape transition is:

• daily source table: (n_daily_rows, n_columns);

• daily feature table: (n_daily_rows, n_columns + derived_columns);

• firm-month table: one row for each retained (PERMNO, month) key.

make_month_key converts each parseable date to a pandas monthly Period. The aggregation key is therefore the pair PERMNO and month, not merely the calendar month. This prevents observations from different firms being combined.

Adjusted prices and daily dollar volume

The paper names CRSP adjustment fields but does not uniquely specify how DisFacPr and DisFacShr should be applied. compute_adjusted_price therefore exposes explicit conventions such as raw, price_only, and price_and_shares. The selected convention is an implementation decision and should be recorded in configuration and provenance; it is not a paper fact. Raw source columns remain available for auditability.

compute_daily_price_change calculates chronological differences within each PERMNO. A missing prior price produces a missing change. The daily dollar-volume function then prefers the derived adj_price column and multiplies it by DlyVol. Share volume must be finite and nonnegative. Missing inputs remain missing rather than being converted into artificial values.

A focused excerpt shows the dollar-volume boundary:

def compute_daily_dollar_volume(frame: pd.DataFrame) -> pd.Series:
    """Compute daily dollar volume as price times share volume.

    The function uses ``adj_price`` when it is already present, because the
    preprocessing pipeline constructs price-based features from the selected
    split-adjusted price.  Otherwise it falls back to ``DlyPrc`` and uses its
    absolute value, matching CRSP's convention that a negative price can mark
    a bid/ask-related observation.  This fallback is an implementation
    decision; the supplied paper does not fully specify the adjustment-factor
    convention.
    """
    if not isinstance(frame, pd.DataFrame):
        raise TypeError("frame must be a pandas DataFrame")

    price_column = (
        _ADJUSTED_PRICE_COLUMN
        if _ADJUSTED_PRICE_COLUMN in frame.columns
        else _DEFAULT_PRICE_COLUMN
    )
    if price_column not in frame.columns:
        raise KeyError(
            f"Missing price column: expected {price_column!r} or "
            f"{_ADJUSTED_PRICE_COLUMN!r}"
        )
    if _DEFAULT_VOLUME_COLUMN not in frame.columns:
        raise KeyError(f"Missing share-volume column: {_DEFAULT_VOLUME_COLUMN!r}")

    price = _numeric_series(frame[price_column], price_column).abs()
    volume = _numeric_series(frame[_DEFAULT_VOLUME_COLUMN], _DEFAULT_VOLUME_COLUMN)
    invalid_volume = volume.notna() & (~np.isfinite(volume) | (volume < 0))
    if bool(invalid_volume.any()):
        raise ValueError("Share volume must be finite and nonnegative when present")

    dollar_volume = (price * volume).astype("float64")
    dollar_volume.name = "dollar_volume"
    finite = dollar_volume.notna() & np.isfinite(dollar_volume)
    dollar_volume.loc[dollar_volume.notna() & ~finite] = np.nan
    return dollar_volume.reindex(frame.index)

The important implementation invariant is that the returned dollar_volume is aligned row-for-row with the input. It is nonnegative when present, but it can be missing when price or share volume is unavailable.

Signed order flow

The paper’s monthly signed-flow construction begins with daily share volume and the direction of the split-adjusted price change. Its purpose is to create an observable proxy for directional order flow before aggregation. The supplied canonical expression is:

Here, i identifies the firm and t identifies the month. τ indexes trading days within that month. Volume_iτ is daily share volume, while ΔP_iτ is the split-adjusted daily price change. The sign function maps a positive change to 1, a negative change to -1, and an unchanged change to 0 under this implementation convention. The result, signedflow_it, is one signed scalar for the firm-month.

The code separates this operation into a daily function and a monthly aggregation. The daily function implements the direction step:

def compute_daily_signed_flow(
    volume: pd.Series, price_change: pd.Series
) -> pd.Series:
    """Compute daily signed order flow from share volume and price direction.

    This implements Eq. 9: daily volume is multiplied by the sign of the
    split-adjusted daily price change.  An unchanged price has sign zero, so
    its signed flow is exactly zero.  Missing volume or price change remains
    missing rather than being treated as a trading direction.
    """
    volume_values = _numeric_series(volume, "volume")
    change_values = _numeric_series(price_change, "price_change")
    if not volume_values.index.equals(change_values.index):
        raise ValueError("volume and price_change must have identical indexes")

    invalid_volume = volume_values.notna() & (
        ~np.isfinite(volume_values) | (volume_values < 0)
    )
    if bool(invalid_volume.any()):
        raise ValueError("volume must be finite and nonnegative when present")

    # Eq. 9: signed flow uses volume times the sign of the price change.
    direction = np.sign(change_values)
    signed_flow = (volume_values * direction).astype("float64")
    signed_flow.name = "signed_flow"
    signed_flow.loc[change_values.isna() | volume_values.isna()] = np.nan
    finite = signed_flow.notna() & np.isfinite(signed_flow)
    if bool((signed_flow.notna() & ~finite).any()):
        raise ValueError("signed flow contains non-finite values")
    return signed_flow.reindex(volume.index)

The function accepts two one-dimensional, identically indexed series. It returns another one-dimensional series of the same length. A missing price change is not interpreted as a sell direction; the corresponding signed flow is missing. This preserves a distinction between no directional movement, represented by zero, and unavailable information, represented by missingness.

Monthly total volume and dispersion

The paper defines monthly unsigned share volume as follows:

sumvolume_it is the nonnegative total share volume for firm i in month t. In aggregate_monthly_volume, it is computed from valid daily DlyVol values and returned as the sumvolume field. The function also returns nonzero_volume_days, which records the activity count used by the minimum-activity filter.

The paper uses within-month volume standard deviation as a proxy for noise-trading variation. The canonical LaTeX for equation 8 is unavailable in the supplied extraction, so it is not displayed or reconstructed here. The implementation follows the paper’s prose description and uses the sample standard deviation, equivalent to pandas ddof=1, over valid daily volume observations. This is a prose-derived implementation detail, not a transcription of missing equation LaTeX.

The minimum-volume rule retains only firm-month groups with at least 15 strictly positive-volume trading days. The filter is applied at the group level before aggregation, so all daily rows from a nonqualifying group are removed. The 15-day rule normally supplies enough observations for a numeric sample standard deviation, although missing values are still handled explicitly.

The aggregation function then produces one record per firm-month:

# Eq. 7: monthly total unsigned share volume.
total_volume = float(valid_volume.sum())

# Eq. 8: sample standard deviation of daily volume; canonical LaTeX is
# unavailable, so the documented ddof=1 convention is used.
stdvolume = float(valid_volume.std(ddof=1)) if len(valid_volume) >= 2 else np.nan

# Eq. 9: monthly signed order flow is the sum of daily signed flow.
signedflow = float(signed_flow.sum(min_count=1)) if signed_flow.notna().any() else np.nan

The resulting columns are sumvolume, nonzero_volume_days, stdvolume, and signedflow, with amihud_lambda added when daily Amihud ratios are supplied. build_firm_month_features sorts the output and rejects duplicate PERMNO-month keys.

Amihud-style illiquidity

The daily Amihud quantity measures absolute return per dollar traded. The paper’s monthly estimator is the mean of valid daily ratios:

In this expression, r_iτ is the daily stock return, DollarVolume_iτ is daily dollar trading volume, and n is the number of valid daily observations used in the average. i and t retain their firm and month meanings. The estimate is nonnegative when valid.

The daily implementation excludes rows with missing or non-finite returns and rows whose dollar-volume denominator is zero. It never turns division by zero into infinity. The valid daily ratios are then averaged within aggregate_monthly_volume as amihud_lambda.

Equation 15 gives the same estimator under the Method A label:

The superscript A identifies Method A, the Amihud-style level estimator. The symbols have the same meanings as in the preceding equation: n is the valid-day count, r_iτ is daily return, and DollarVolume_iτ is a positive daily denominator. In code, both equation records map to compute_daily_amihud_ratio followed by the within-month mean. The two equation labels describe the same calculation rather than two different daily algorithms.

A focused denominator check from the generated implementation is:

valid = (
    returns.notna()
    & np.isfinite(returns)
    & dollars.notna()
    & np.isfinite(dollars)
    & (dollars > 0)
)
ratio = pd.Series(np.nan, index=returns.index, dtype="float64", name="amihud_ratio")
ratio.loc[valid] = (returns.loc[valid].abs() / dollars.loc[valid]).astype("float64")
if bool((ratio.notna() & ~np.isfinite(ratio)).any()):
    raise ValueError("Amihud ratios must be finite when present")
return ratio

Thus, a zero-dollar-volume day does not contribute to the monthly mean. This can make the number of valid Amihud days smaller than the number of days used for volume aggregation.

Worked example: three daily observations

Consider one firm with three already adjusted daily observations. Suppose the share volumes are 100, 200, and 300, and the corresponding price changes are positive, zero, and negative. The signed contributions are therefore +100, 0, and -300; the unchanged price contributes exactly zero. The unsigned total volume is 600.

If the three valid daily volumes are used for the sample dispersion, the calculation is the sample standard deviation of 100, 200, and 300, using denominator n - 1 rather than n. This example is conceptual: the paper’s minimum filter requires at least 15 positive-volume days for a retained firm-month, so three rows alone would not survive the actual aggregation filter.

For the Amihud calculation, suppose the daily returns are 0.01, 0.02, and -0.03, and the corresponding positive dollar volumes are 1,000, 2,000, and 0. The first two ratios are valid. The third is excluded because its denominator is zero; it does not become an infinite ratio and is not included in n. The monthly Amihud value is consequently the mean of the valid ratios only.

The same fields would then flow into later stages as follows: sumvolume, stdvolume, signedflow, and amihud_lambda become firm-month predictors; StockRet is merged from monthly data; and the alignment stage creates a same-firm next-month target. No numerical result from this illustrative example is a paper result or a verified execution outcome.

Orchestration and evidence boundary

build_daily_audit_table exposes the cleaned daily frame with derived fields for inspection. build_empirical_panel calls the daily stage, aggregates firm-month features, constructs controls, estimates lambda values, merges monthly returns, and aligns next-month returns. The generated pipeline does not access a network; CRSP and auxiliary files must already exist locally.

The implementation enforces key structural constraints such as nonnegative volume, finite valid Amihud ratios, at least 15 positive-volume days in retained groups, and unique PERMNO-month rows. However, the supplied verification records state that static verification and semantic code verification were skipped under the run policy. These constraints describe intended implementation behavior, not checks claimed to have passed. The adjustment convention, exact corporate-action treatment, return and delisting conventions, and winsorization policy remain configuration-dependent.

The theoretical Kyle relations and their empirical interpretation

How can a price-impact coefficient connect trading activity to prices without being confused with a directly observed quantity? The paper’s Kyle-style model provides the intuition: an informed trader responds to a fundamental-value deviation, noise traders add orders that are not separately observed, and a market maker adjusts the transaction price according to total signed order flow. The empirical pipeline then uses observed signed volume as a proxy for this latent flow; it does not observe the theoretical informed and noise orders individually.

This section therefore has two layers. The first is a scalar theoretical API in src/liquidity_premium/models/kyle.py. The second is the empirical estimation workflow described elsewhere, where daily signed flow and price changes are used to estimate firm-month price impact. The theoretical functions accept scalar values and return scalar values. They are not fitted to CRSP data by the generated module.

Informed demand responds to a value discrepancy

In the single-period model, v is the asset’s liquidation value, while p_0 is its initial or prior price. Their difference represents the fundamental-value discrepancy available to the informed trader. The positive scalar beta measures informed-trading intensity. The paper expresses informed demand as follows:

This is eq_1. The symbol x is the informed trader’s scalar market order. If v exceeds p_0, the order is positive under this convention; if v is below p_0, it is negative. The restriction beta > 0 means that the intensity scales the direction implied by the value discrepancy rather than reversing it.

The generated function informed_order(v, p0, beta) implements this relation. It validates that the inputs are finite scalars and that beta is strictly positive, then returns the scalar order quantity. The validation is an implementation safeguard corresponding to the model’s stated positivity restriction; it does not estimate beta or infer v from market data.

A focused excerpt from src/liquidity_premium/models/kyle.py shows the mapping:

def informed_order(v: float, p0: float, beta: float) -> float:
    """Compute informed demand from the supplied single-period Kyle relation.

    Parameters
    ----------
    v:
        Asset liquidation value.
    p0:
        Initial or prior asset price.
    beta:
        Positive informed-trading intensity.

    Returns
    -------
    float
        Informed order quantity ``x``.  The sign follows ``v - p0``.

    Notes
    -----
    Implements eq. 1: ``x = beta (v - p0)``.  This is a theoretical
    calculation, not an estimator of empirical order flow.
    """
    liquidation_value = _validate_finite_scalar(v, "v")
    initial_price = _validate_finite_scalar(p0, "p0")
    intensity = _validate_positive_scalar(beta, "beta")

    # Eq. 1: informed demand is proportional to the liquidation-value deviation.
    return intensity * (liquidation_value - initial_price)

The final comment is important for reproduction fidelity: the function computes x from supplied scalar inputs, but it does not produce the empirical signedflow feature.

Aggregate flow is latent in the model

The theoretical informed order is only one component of total order flow. Let u denote a noise-trader order and let y denote aggregate signed order flow. In the model, aggregate flow is formed by adding the informed and noise components. The helper aggregate_order_flow(informed, noise) performs that addition and preserves the signs of both scalar inputs.

This distinction matters when reading the empirical pipeline. The model’s y is latent: the data do not separately identify x and u. By contrast, empirical signedflow is constructed from daily share volume and the sign of the observed split-adjusted price change. It is a proxy motivated by the theory, not an observed measurement of y, x, or u.

The market maker maps flow into price

The paper’s market-maker rule says that the transaction price equals the prior price plus a flow-dependent price adjustment. The coefficient lambda is the Kyle price-impact coefficient: it measures the price displacement associated with one unit of aggregate signed flow. The paper writes:

This is eq_2. Here, p is the transaction price, p_0 is the initial or prior price, y is aggregate theoretical flow, and lambda is a positive scalar. Because lambda is positive, positive aggregate flow raises the transaction price relative to p_0, while negative aggregate flow lowers it.

The generated market_maker_price(p0, lam, aggregate_flow) function implements this pricing rule. It checks that p0, lam, and the aggregate flow are finite and that lam is strictly positive. The function is a theoretical pricing helper, not an empirical regression. In particular, passing an observed signed-flow value to it would be an API illustration, not proof that the observed value is the model’s latent y.

The relevant implementation excerpt is:

def market_maker_price(p0: float, lam: float, aggregate_flow: float) -> float:
    """Apply the linear market-maker pricing rule to aggregate flow.

    Parameters
    ----------
    p0:
        Initial or prior asset price.
    lam:
        Positive Kyle price-impact coefficient.
    aggregate_flow:
        Aggregate signed order flow ``y``.

    Returns
    -------
    float
        Transaction price ``p``.

    Notes
    -----
    Implements eq. 2: ``p = p0 + lambda y``.  The positive-lambda validation
    ensures that positive (negative) flow produces a positive (negative) price
    displacement relative to ``p0``.
    """
    initial_price = _validate_finite_scalar(p0, "p0")
    price_impact = _validate_positive_scalar(lam, "lam")
    flow = _validate_finite_scalar(aggregate_flow, "aggregate_flow")

    # Eq. 2: the market maker maps aggregate signed flow into price impact.
    return initial_price + price_impact * flow

The empirical within-month regression uses the same conceptual direction—price change related to signed flow—but estimates a firm-month coefficient from daily observations. That later coefficient is an empirical construction whose exact scale and interpretation depend on the data adjustments, regression convention, and valid observations.

Sequential price changes

The sequential extension indexes trading rounds by n. In round n, Delta y_n is the order-flow innovation and lambda_n is the price-impact coefficient for that round. The resulting price innovation is:

This is eq_5. The scalar Delta p_n is the price change in round n; lambda_n is a positive, round-specific price-impact coefficient; and Delta y_n is the round’s signed order-flow innovation. The equation does not specify a full dynamic path for lambda_n; it only states how a given round’s coefficient maps that round’s flow innovation into a price change.

The generated sequential_price_change(lam_n, delta_y_n) function implements this multiplication. It validates positive lam_n and finite delta_y_n, then returns a scalar. Its sign invariant follows directly from the positivity restriction: a positive flow innovation produces a positive price change, a negative innovation produces a negative price change, and a zero innovation produces zero change.

def sequential_price_change(lam_n: float, delta_y_n: float) -> float:
    """Compute a sequential-auction price innovation.

    Parameters
    ----------
    lam_n:
        Positive price-impact coefficient for round ``n``.
    delta_y_n:
        Order-flow innovation in round ``n``.

    Returns
    -------
    float
        Round-specific price change ``Delta p_n``.  Its sign equals the sign of
        ``delta_y_n`` unless the innovation is zero.

    Notes
    -----
    Implements eq. 5: ``Delta p_n = lambda_n Delta y_n``.  No dynamic path for
    ``lambda_n`` is inferred or simulated here.
    """
    round_price_impact = _validate_positive_scalar(lam_n, "lam_n")
    flow_innovation = _validate_finite_scalar(delta_y_n, "delta_y_n")

    # Eq. 5: each round's price innovation scales that round's flow innovation.
    return round_price_impact * flow_innovation

Worked API illustration

Consider a hypothetical asset with v = 101.0, p0 = 100.0, and beta = 2.0. The informed-order function represents a value deviation of one price unit scaled by an intensity of two, so the conceptual informed order is positive. Suppose a noise trader submits a negative order; aggregate_order_flow adds that noise order to the informed order. A positive lam then maps the resulting signed flow into a transaction-price displacement.

For a sequential illustration, set lam_n to a positive value and delta_y_n to either a positive or negative flow innovation. The returned price change follows the innovation’s sign. These are hand-worked API examples for understanding scalar inputs, scalar outputs, and sign behavior. They are not CRSP observations, paper calibrations, or executed numerical results.

The planned tests in tests/test_theoretical_kyle.py mirror these direct relations for eq_1, eq_2, and eq_5. Under the run policy, those tests were not executed or verified; their presence describes intended coverage rather than a passed check.

What is deliberately excluded

The paper extraction does not provide safe canonical formulas for every theoretical statement. eq_3, the closed-form equilibrium values for beta and lambda, is OCR-damaged and is not reconstructed. eq_4 is a qualitative sequential comparative-static claim whose conditioning and distributional assumptions are underspecified, so it is not simulated. eq_6, concerning the continuous-time path of lambda(t), also has damaged canonical LaTeX and is not implemented.

These exclusions are deliberate evidence boundaries. The generated module implements only the supplied scalar relations eq_1, eq_2, and eq_5; it does not fill gaps by importing a familiar Kyle formula or by choosing an unstated dynamic model. The paper’s broader model description and its formal single-period presentation are also not fully reconciled, and its statements about how noise-trading variance affects price impact are internally inconsistent. Those comparative statics should remain documented ambiguities rather than being resolved silently in code.

Finally, the theoretical lambda should not be treated as identical to either empirical firm-month estimator. The empirical pipeline estimates price impact or illiquidity from observed daily price, return, volume, and signed-flow proxies. The theoretical functions explain why such a coefficient is economically meaningful; they do not identify it from the supplied scalar inputs or replace the empirical estimator.

Estimating firm-month lambda

How can several daily observations become one monthly measure of price impact? For each PERMNO and calendar month, the reproduction computes two separate scalars. Method B estimates how daily price changes respond to signed daily flow. Method A computes an Amihud-style average of absolute return per dollar traded. They address related notions of illiquidity, but they are not interchangeable columns.

The daily inputs for one firm-month are one-dimensional vectors with shape (n_days,): price_change contains split-adjusted daily price changes, signed_flow contains signed daily volume, return_ contains daily returns, and dollar_volume contains daily dollar trading volume. The grouped function reduces each valid group to one row containing regression_lambda, amihud_lambda, and estimator-specific valid-day counts.

Method B: price change on signed flow

The paper’s within-month regression relates daily price changes to daily signed order flow. Its displayed specification has no intercept, so the default implementation is an uncentered one-predictor regression.

Here, \Delta P_{i\tau} is the daily split-adjusted price change for firm i on trading day \tau; OF_{i\tau} is that day’s signed-flow proxy; and \hat{\lambda}_{it} is the estimated scalar price-impact slope for firm i in month t. The residual \eta_{i\tau} captures the part of the daily price change not explained by the signed-flow regressor. The regression is performed separately within each PERMNO-month group, so its input has shape (n_valid_days, 1) for the design matrix and (n_valid_days,) for the response. The returned lambda is one scalar.

The default include_intercept=False follows the displayed equation. The generated function exposes include_intercept=True as an explicit alternative because the paper does not clearly resolve whether the intramonth estimator itself should include an intercept. This choice is separate from whether a later return regression includes an intercept.

The core implementation delegates the one-predictor calculation to fit_ols:

    try:
        result = fit_ols(
            flow.reshape(-1, 1),
            prices,
            include_intercept=bool(include_intercept),
        )
    except ValueError:
        # A group-level estimator should preserve missingness for groups that
        # cannot support the requested OLS specification rather than aborting
        # an otherwise valid firm-month batch.
        return None

    coefficient_index = 1 if include_intercept else 0
    estimate = float(result.coefficients[coefficient_index])
    return estimate if np.isfinite(estimate) else None

The flow.reshape(-1, 1) expression makes the design explicitly two-dimensional, while prices remains a one-dimensional response. With an intercept, coefficient index 1 is the flow slope because the intercept occupies index 0; without one, index 0 is the only coefficient. fit_ols validates finite values, row counts, rank, and residual degrees of freedom. At the group level, an unsupported specification becomes None, preserving estimator-specific missingness rather than stopping all firm-month processing.

A group also returns None when its signed-flow vector has no usable magnitude or when there are too few observations for the selected design. This is important for a no-intercept regression: a zero-flow group cannot identify a meaningful price-impact slope. The paper’s broader sample rule retains firm-months with at least 15 nonzero-volume trading days, but it does not state a separate minimum specifically for the intramonth regression. The upstream filter and these estimator-level degeneracy checks therefore serve different purposes.

Method A: Amihud-style illiquidity

The second estimator converts each valid daily observation into an absolute-return-per-dollar-volume ratio and averages those ratios within the firm-month. The paper records this construction as follows.

In this equation, \hat{\lambda}^{\mathrm{Amihud}}_{it} is the nonnegative Amihud-style estimate for firm i and month t. The integer n is the number of valid daily observations used in the mean. r_{i\tau} is the daily stock return, and \mathrm{DollarVolume}_{i\tau} is daily dollar trading volume. The absolute-value operator makes the numerator nonnegative, while the denominator must be strictly positive.

The paper also labels the same construction Method A:

Equations eq_13 and eq_15 therefore map to the same generated function, estimate_amihud_lambda. The function excludes rows with nonpositive dollar volume, so a zero denominator does not create an infinite ratio. Negative dollar-volume inputs are rejected as invalid. A valid result is nonnegative provided at least one daily observation remains.

The focused implementation is:

def estimate_amihud_lambda(
    return_: np.ndarray,
    dollar_volume: np.ndarray,
) -> float | None:
    """Estimate the within-firm-month Amihud-style illiquidity measure.

    The inputs are paired daily vectors with shape ``(n_days,)``.  Observations
    with non-finite returns or non-positive dollar volume are excluded, since
    the denominator in eq. 13 and eq. 15 must be strictly positive.  The
    resulting estimate is nonnegative whenever at least one valid observation
    remains.
    """
    returns = _as_float_vector(return_, "return_")
    dollars = _as_float_vector(dollar_volume, "dollar_volume")
    if returns.size != dollars.size:
        raise ValueError(
            "return_ and dollar_volume must have equal lengths: "
            f"{returns.size} != {dollars.size}."
        )
    if bool((dollars < 0).any()):
        raise ValueError("dollar_volume must be nonnegative.")

    valid = np.isfinite(returns) & np.isfinite(dollars) & (dollars > 0.0)
    if not bool(valid.any()):
        return None

    # Eq. 13 and Eq. 15: lambda_hat = mean(|r| / DollarVolume).
    ratios = np.abs(returns[valid]) / dollars[valid]
    if not np.isfinite(ratios).all():
        return None
    estimate = float(np.mean(ratios, dtype=np.float64))
    if estimate < 0.0 or not np.isfinite(estimate):
        return None
    return estimate

Notice the distinction between invalid input and unavailable estimation. A negative dollar volume raises an error because it violates the data contract. A group with no positive dollar-volume observations returns None, because no valid Amihud mean can be formed. The function uses float64 accumulation for the within-group mean and returns a scalar float when the calculation is defined.

Grouped firm-month computation

estimate_firm_month_lambdas applies both estimators independently to every (PERMNO, month) group. It requires price_change and signed_flow, plus a daily return column named DlyRet, return, or return_, and a dollar_volume column. It sorts by firm and month, converts the daily columns to numeric arrays, computes separate validity masks, and records the two estimates and their valid-day counts.

A compact excerpt shows the separate paths:

        regression_valid = np.isfinite(price_values) & np.isfinite(flow_values)
        amihud_valid = (
            np.isfinite(return_values)
            & np.isfinite(dollar_values)
            & (dollar_values > 0.0)
        )
        regression_lambda = (
            estimate_regression_lambda(
                price_values[regression_valid],
                flow_values[regression_valid],
                include_intercept=bool(include_intercept),
            )
            if bool(regression_valid.any())
            else None
        )
        amihud_lambda = (
            estimate_amihud_lambda(
                return_values[amihud_valid],
                dollar_values[amihud_valid],
            )
            if bool(amihud_valid.any())
            else None
        )

The two masks need not select the same days. For example, a daily price change and signed flow may be available while the daily return is missing, or a return may be available while dollar volume is zero. Consequently, regression_valid_days and amihud_valid_days are reported separately. The output contract is one row per firm-month key, with missingness retained independently for regression_lambda and amihud_lambda.

Lambda and the subsequent return target

Estimating lambda is only the first step. The current-month estimate must be aligned with the same firm’s subsequent calendar-month return. The paper writes the return relationship as:

Here, \mathrm{StockRet}_{i,t+1} is firm i’s return in the month after the estimate; \hat{\lambda}_{it} is the current-month lambda; \alpha_i and \beta_i are regression coefficients in the paper’s notation; and \varepsilon_{i,t+1} is the residual. This equation describes a later return regression, not the daily lambda estimator itself.

The supplied paper uses firm subscripts on the coefficients but also describes pooled observations and tables. The generated reproduction therefore treats pooled versus firm-specific estimation as an explicit modeling issue rather than assuming that the notation settles it. The alignment module uses an explicit PERMNO and month-plus-one join, so a missing calendar month does not silently become an adjacent observation.

    result["next_month"] = result[_MONTH] + pd.DateOffset(months=1)

    lookup = result[[_PERMNO, _MONTH, return_column]].rename(
        columns={_MONTH: "_target_month", return_column: "next_month_return"}
    )
    aligned = result.merge(
        lookup,
        left_on=[_PERMNO, "next_month"],
        right_on=[_PERMNO, "_target_month"],
        how="left",
        sort=False,
        validate="one_to_one",
    ).drop(columns=["_target_month"])

The join preserves the same firm identifier and requests exactly one calendar month later. The resulting next_month_return is then available to a later pooled regression; it is not used when computing the current-month lambda.

Worked example: vector contracts

Consider a synthetic firm-month with four daily observations. The intended inputs have these shapes:

• price_change: (4,)

• signed_flow: (4,)

• return_: (4,)

• dollar_volume: (4,)

A caller can request both estimators through the public functions:

import importlib
import numpy as np

lambda_module = importlib.import_module(
    "liquidity_premium.estimators.lambda"
)

price_change = np.array([0.10, -0.04, 0.08, 0.02], dtype=float)
signed_flow = np.array([100.0, -50.0, 75.0, 25.0], dtype=float)
daily_return = np.array([0.010, -0.004, 0.008, 0.002], dtype=float)
dollar_volume = np.array(
    [10_000.0, 8_000.0, 12_000.0, 9_000.0],
    dtype=float,
)

regression_lambda = lambda_module.estimate_regression_lambda(
    price_change,
    signed_flow,
    include_intercept=False,
)
amihud_lambda = lambda_module.estimate_amihud_lambda(
    daily_return,
    dollar_volume,
)

This excerpt documents the API and shape transition only; it was not executed under the run policy, so no numerical output is claimed. The first result is a scalar regression slope, and the second is a scalar mean ratio. If the same firm-month supplied a zero vector for signed_flow, the regression estimator would return None because the no-intercept slope is degenerate. The Amihud estimator could still be valid if its return and positive dollar-volume vectors remained usable.

Method B’s incomplete source equation

The paper describes Method B as the slope from regressing daily price changes on daily volume multiplied by the sign of the daily price change. The generated code maps that prose to the same signed-flow regression path used for eq_12. However, the canonical LaTeX for eq_16 is OCR-truncated. No missing formula, intercept convention, or additional term is reconstructed here. The implementation follows only the supplied prose and the complete displayed relationship in eq_12.

Finally, do not conflate two independent choices: include_intercept on estimate_regression_lambda controls the intramonth price-impact estimator, while include_intercept on the later lambda-return regression controls the return model. The paper’s estimator ambiguity is therefore isolated in the API rather than silently propagated to every downstream specification.

Contemporaneous, predictive, and lambda-return regressions

How should a monthly activity measure be connected to a stock return without accidentally using the wrong firm or the wrong month? The reproduction treats every regression row as a keyed observation: PERMNO identifies the firm, month identifies the predictor month, and the target is either that same row’s return or the same firm’s return in the immediately following calendar month.

This section covers three related specifications. The first asks whether monthly volume features are associated with contemporaneous returns. The second uses those features to forecast the next month’s return. The third replaces the volume features with one firm-month lambda estimate at a time. All three are pooled ordinary least squares (OLS) regressions: observations from eligible firms and months enter one design matrix. The implementation does not constrain coefficient signs.

The common panel contract

The panel functions expect one row per PERMNO-month key. The base predictor columns are:

• sumvolume: total unsigned share volume during the month;

• stdvolume: within-month sample standard deviation of daily volume;

• signedflow: monthly signed-flow proxy.

The target column is StockRet for the contemporaneous specification. For a one-month-ahead specification, the code creates next_month_return by matching the current firm and the next calendar month. Optional controls are appended by name. The paper discusses controls such as log market capitalization, book-to-market, momentum, and Amihud illiquidity, but unavailable external controls must remain missing rather than being fabricated.

The design-building function returns a predictor matrix with shape (n_used_observations, p), a target vector with shape (n_used_observations,), and labels preserving predictor order. The shared fit_ols function then optionally prepends an intercept column. With three base predictors and an intercept, the coefficient vector has shape (4,). The residual vector has length n_used_observations.

Equation 10: contemporaneous activity and returns

The contemporaneous specification asks whether firm-month activity and the return in that same firm-month are related. The paper records it as follows.

Here, StockRet_it is the monthly stock return for firm i in month t. The three predictors are total volume sumvolume_it, volume volatility stdvolume_it, and signed flow signedflow_it. alpha is the pooled regression intercept; beta_1, beta_2, and beta_3 are the corresponding slopes; and epsilon_it is the residual for that firm-month observation.

In the generated implementation, run_panel_return_regression selects this equation with horizon="contemporaneous". It delegates row construction to build_panel_dataset, creates the ordered predictor list, and passes the resulting arrays to build_panel_design and then fit_ols with an intercept.

A focused call is:

from liquidity_premium.regressions.panel import run_panel_return_regression

result = run_panel_return_regression(
    panel,
    horizon="contemporaneous",
    controls=None,
)

The no-control design has columns in this order:

sumvolume, stdvolume, signedflow

The returned labels add the specification context, including contemporaneous.no_controls.intercept. Rows with a missing target or missing required predictor are removed before fitting. The OLS primitive requires finite values, more observations than fitted parameters, and a full-rank design. These are numerical input conditions, not economic assumptions.

The paper does not fully specify fixed effects, weighting, clustering, or panel covariance treatment. The generated panel module therefore exposes a covariance argument but currently uses the shared classical homoskedastic OLS inference. That implementation choice must be recorded in result metadata and must not be presented as uniquely determined by the paper.

Equation 11: exact one-month-ahead alignment

The predictive version uses month-t activity to explain the same firm’s return in month t+1.

The notation changes only the target timing: StockRet_i,t+1 is the next calendar month’s return for the same firm. The predictors remain sumvolume_it, stdvolume_it, and signedflow_it, all measured in month t. alpha and the three beta coefficients retain their regression roles, while epsilon_i,t+1 is the forecast-regression residual.

The important implementation detail is that “next month” means an explicit calendar key, not merely the next available row for a firm. align_next_month_return normalizes month values, sorts by PERMNO and month, creates a month-plus-one key, and joins the return lookup on both PERMNO and target month. Thus, a January observation with no February row does not receive a March return as its target.

The core alignment operation is represented by this generated code excerpt:

result["next_month"] = result[_MONTH] + pd.DateOffset(months=1)

lookup = result[[_PERMNO, _MONTH, return_column]].rename(
    columns={_MONTH: "_target_month", return_column: "next_month_return"}
)
aligned = result.merge(
    lookup,
    left_on=[_PERMNO, "next_month"],
    right_on=[_PERMNO, "_target_month"],
    how="left",
    sort=False,
    validate="one_to_one",
).drop(columns=["_target_month"])

_PERMNO and _MONTH are the firm and calendar-month key columns. The lookup contains one return per firm-month. The one_to_one validation expresses the uniqueness contract, while the two-key merge prevents cross-firm leakage. Missing next-month observations remain missing and are later removed when build_panel_dataset prepares complete regression rows.

The predictive call changes only the horizon argument:

next_month_result = run_panel_return_regression(
    panel,
    horizon="next_month",
    controls=None,
)

With three base predictors and an intercept, this again produces four coefficients, but the retained observation count can be smaller because firms’ final available months have no following calendar-month target. The generated tests describe this behavior for synthetic panels, but those tests were not executed under the current run policy.

Adding controls without hiding missing data

Controls are appended after the three base predictors. For example, a fully populated controlled specification can be requested with:

controlled_result = run_panel_return_regression(
    panel,
    horizon="next_month",
    controls=[
        "log_size",
        "book_to_market",
        "momentum",
        "amihud_lambda",
    ],
)

If all four columns are available and numeric, the design contains seven economic predictors plus an intercept. The names must already exist in panel; the regression function raises an error for a missing requested column rather than inventing a substitute. This matters especially for book-to-market and factor-related inputs, which depend on external WRDS or factor files not supplied in the paper context.

The implementation keeps predictor units unstandardized. The paper does not specify a scaling convention, so raw economic units are retained for coefficient reporting. Every result should identify whether it uses no controls or controls and whether its return is raw, delisting-adjusted, or excess, because the supplied paper uses return conventions in different contexts without fully resolving them.

Equation 14: lambda as the sole predictor

The lambda-return specification tests whether a current firm-month illiquidity estimate predicts the same firm’s next-month return.

StockRet_i,t+1 is the next-month return for firm i. The predictor hat(lambda)_it is the lambda estimate formed from firm i’s daily observations in month t; it may be the regression-based estimator or the Amihud-style estimator. alpha_i and beta_i are the intercept and slope notation used in the paper, and epsilon_i,t+1 is the residual.

The generated implementation uses a pooled interpretation by default. Although the equation uses firm-specific subscripts on the coefficients, the supplied descriptions and reported pooled observation structure support fitting one regression over the eligible firm-month rows rather than silently fitting a separate regression per firm.

The lambda-return function recomputes same-firm next-month alignment through align_next_month_return, selects exactly one estimator column, removes rows with missing or non-finite estimator or target values, and calls fit_ols. A focused example is:

from liquidity_premium.regressions.lambda_return import (
    run_lambda_return_regression,
)

regression_lambda_with_intercept = run_lambda_return_regression(
    panel,
    estimator="regression_lambda",
    include_intercept=True,
)

amihud_lambda_uncentered = run_lambda_return_regression(
    panel,
    estimator="amihud_lambda",
    include_intercept=False,
)

The include_intercept argument belongs to this return regression. It is separate from the convention used earlier when estimating the firm-month regression lambda from daily price changes and signed flow. The generated default for that daily estimator is uncentered because the displayed price-impact equation omits an intercept; the return regression deliberately exposes both alternatives.

With an intercept, the lambda predictor matrix supplied to fit_ols has shape (n_used_observations, 1), and the fitted coefficient vector has shape (2,): intercept and lambda slope. Without an intercept, the coefficient vector has shape (1,) and contains only the lambda slope. The generated test specifies these structural dimensions without asserting any expected economic sign.

Comparing estimator and intercept specifications

The paper emphasizes that direct lambda-return relationships can be sensitive to both estimator construction and intercept treatment. run_lambda_specification_grid makes those comparisons explicit:

from liquidity_premium.regressions.lambda_return import (
    run_lambda_specification_grid,
)

lambda_grid = run_lambda_specification_grid(panel)

The grid contains four combinations:

regression_lambda, with intercept
regression_lambda, uncentered
amihud_lambda, with intercept
amihud_lambda, uncentered

Each row stores estimator and intercept labels, coefficient arrays, standard errors, t-statistics, R-squared, and observation count. The two lambda columns are never combined into one regression by this helper, so estimator differences remain visible.

A small timing example

Consider two firms, A and B, observed in January, February, and March. The January row for A receives A’s February return as its next-month target; the January row for B receives B’s February return. February rows similarly receive March returns. March rows have no target unless April is present. A January row for A must never receive B’s February return, and a January row for a firm with no February record must not receive that firm’s March return.

For the contemporaneous model, all six firm-month rows can be eligible if their current-month returns and predictors are present. For the next-month model, only rows with an exact same-firm calendar-next-month return are eligible. The resulting base design has three columns and shape (n_used_observations, 3) before the intercept is added. An uncentered lambda-return design has one column and shape (n_used_observations, 1).

This example explains key and shape behavior only. It is not a numerical result and has not been executed.

Interpreting signs and inference boundaries

The paper’s narrative sometimes suggests a positive predictive role for signed flow, while the supplied results report negative signed-flow coefficients in the one-month-ahead specifications. The reproduction must preserve those reported signs rather than flip them to match a hypothesis. A coefficient sign is an output of a particular data, return, alignment, winsorization, and inference specification; it is not an input restriction.

Similarly, classical OLS standard errors and t-statistics are available from the generated fit_ols implementation, but the paper does not uniquely specify panel clustering, fixed effects, weighting, or covariance treatment. The generated code was not executed, and local static verification and code semantic verification were skipped under the authoritative run policy. Consequently, this section documents the equation-to-code mapping and structural contracts, not verified numerical agreement with the paper’s tables.

Expanding-window forecasts and out-of-sample evaluation

How can a return forecast imitate an investor making predictions in real time? Start with an initial history, estimate a model using only that history, predict the next available month, then enlarge the history and repeat. This is the intuition behind an expanding-window forecast. Unlike a rolling window, which drops old observations, an expanding window retains all eligible historical observations as new information arrives.

The paper describes this procedure using the first 30 percent of each firm’s chronological observations as the initial training period. The source alternates between the words “rolling” and “expanding,” but its explicit procedural description supports the expanding interpretation used here. The generated implementation also makes the integer-rounding rule explicit: the initial length is the ceiling of 30 percent of the firm’s observations. Because the paper does not specify a rounding rule, this is an implementation decision rather than a uniquely paper-determined fact.

Inputs and timing contract

generate_expanding_forecasts() expects a firm-month pandas.DataFrame containing one row per PERMNO and calendar month. The required return column is StockRet. The function accepts a risk-free-rate column named rf, risk_free_rate, or r_f, and a lambda column named lambda, lambda_estimate, regression_lambda, or amihud_lambda.

The shape flow is:

firm-month panel:        (n_rows, p)
firm-local training:     (n_training_rows, p)
forecast records:        (n_oos_observations, q)
pooled evaluation sample: (n_oos_observations, 2 predictor/target fields)

For each firm, the implementation sorts rows by calendar month and treats adjacent calendar months as a valid forecast transition. If the next row is not exactly one month later, that forecast origin is skipped rather than treated as an artificial adjacent observation. Missing predictors, missing realized returns, and insufficient training observations are also skipped; the code does not fabricate values.

At forecast origin j, the current eligible observation is included in the training sample. The subsequent calendar month supplies the predictor values used for the forecast and the realized return used later for evaluation. This ordering is the central no-look-ahead rule: the target return must not enter the model that predicts it.

Equation 17: the recursive forecasting model

The paper’s recursive forecasting relation is:

Here, ActualReturn_t is the realized return at time t; a is the intercept; b_1 is the coefficient on the risk-free rate r_f,t; b_2 is the coefficient on the current firm-month lambda estimate hat(lambda)_it; and epsilon_t is the regression residual. In the generated implementation, the model is fit with an intercept and two predictors in the order risk-free rate, then lambda.

fit_recursive_forecast_model() in src/liquidity_premium/forecast/expanding.py receives a training DataFrame, removes rows missing any required variable, constructs a design matrix with shape (n_training_rows, 2), and calls fit_ols() from src/liquidity_premium/estimators/ols_core.py. With an intercept, fit_ols() adds a column of ones, so the fitted coefficient vector has shape (3,): intercept, risk-free-rate coefficient, and lambda coefficient.

A focused excerpt shows the equation-to-code mapping:

    x = complete[[risk_free_column, lambda_column]].to_numpy(dtype=np.float64)
    y = complete[_RETURN].to_numpy(dtype=np.float64)
    if complete.shape[0] <= x.shape[1] + 1:
        raise ValueError("training must contain more observations than fitted parameters.")

    # Eq. 17: ActualReturn_t = a + b_1 r_{f,t} + b_2 lambda_it + epsilon_t.
    result = fit_ols(x, y, include_intercept=True)
    result.labels = ("intercept", "risk_free_rate", "lambda")

The recursive model is estimated separately for each firm in the generated workflow. This follows the firm-level method card, although the paper’s notation and discussion do not fully resolve whether the forecasting relation should instead be pooled, firm-specific, or hierarchical. That ambiguity should remain visible in provenance metadata.

Building the expanding forecasts

The public function generate_expanding_forecasts(panel, initial_fraction=0.30) performs the chronological loop. Its algorithm is:

1. Normalize and sort the panel by PERMNO and month.

2. For each firm, compute ceil(0.30 * n_rows) as the initial window length.

3. Use the current origin and all earlier rows as the candidate training history.

4. Remove incomplete training rows and fit Equation 17 when enough observations remain.

5. Use the next calendar month’s risk-free rate and lambda as predictors.

6. Store the predicted return and the subsequent realized return.

7. Continue with a nondecreasing training endpoint.

The output records include the firm identifier, origin month, forecast month, actual and predicted returns, training start and end dates, training-observation count, estimator label, and frequency label. The intended output shape is one row per eligible firm-month forecast origin.

from liquidity_premium.forecast.expanding import generate_expanding_forecasts

forecasts = generate_expanding_forecasts(
    panel,
    initial_fraction=0.30,
)

The first forecast for a firm with n_rows observations uses row ceil(0.30 * n_rows) - 1 as its origin, because the current origin is included in training. The next row must be the immediately following calendar month. For example, with 10 ordered monthly rows, the ceiling-based initial length is 3. The first origin is therefore the third row, and the first target is the fourth month. This example explains the indexing convention only; it is synthetic and has not been executed.

The generated implementation validates several timing invariants through validate_forecast_timing(). It requires unique firm-forecast-month keys, a training endpoint earlier than the forecast month, a positive training-observation count, finite actual and predicted returns, and nondecreasing training endpoints within each firm.

from liquidity_premium.forecast.expanding import validate_forecast_timing

validate_forecast_timing(forecasts)

Calling this function is part of the intended workflow, not evidence that the workflow has been run. Under the authoritative run policy, no code execution, test execution, static verification, or semantic code verification was performed.

Equation 18: calibration of stored forecasts

Forecast generation and forecast evaluation answer different questions. Generation asks what prediction would have been available at each historical forecast origin. Calibration evaluates the relationship between those stored predictions and their realized outcomes. The paper specifies that pooled evaluation with:

In this equation, ActualReturn is the observed out-of-sample return, PredictedReturn is the previously stored forecast, alpha is the calibration intercept, beta measures the calibration slope, and epsilon is the evaluation residual. A calibration regression is not a replacement for the recursive forecasting step; it is applied only after forecasts have been generated.

evaluate_oos_predictions() in src/liquidity_premium/forecast/evaluation.py maps actual_return to the response and predicted_return to the single predictor. It requires an explicit boolean is_out_of_sample or out_of_sample column and filters to rows marked true before fitting Equation 18.

from liquidity_premium.forecast.evaluation import evaluate_oos_predictions

oos_evaluation = evaluate_oos_predictions(forecasts)

The core implementation is deliberately narrow:

    sample = _out_of_sample_rows(forecasts)
    x = sample[["predicted_return"]].to_numpy(dtype=float)
    y = sample["actual_return"].to_numpy(dtype=float)

    # Eq. 18: ActualReturn = alpha + beta * PredictedReturn + epsilon.
    result = fit_ols(x, y, include_intercept=True)

The resulting RegressionResult has two coefficient labels, intercept and predicted_return, and uses only the explicitly marked out-of-sample rows. It also checks that actual and predicted values are numeric and finite. If the marker is absent, the function raises an error instead of guessing which rows are out of sample.

An important interface issue should be recorded rather than hidden: the generated generate_expanding_forecasts() excerpt does not itself add an is_out_of_sample column, while evaluate_oos_predictions() requires one. A caller must therefore attach an explicit marker or adapt the output contract before evaluation. The supplied test fixture also uses lowercase names such as permno and training_end_month, whereas the generated forecast function emits PERMNO and training_end. These are unresolved generated-file interface mismatches, not results of a completed verification pass.

For separate estimator and frequency summaries, evaluate_by_method_and_frequency() groups the explicitly marked rows by estimator and frequency, then applies Equation 18 to each group:

from liquidity_premium.forecast.evaluation import (
    evaluate_by_method_and_frequency,
)

method_evaluations = evaluate_by_method_and_frequency(forecasts)

The paper does not specify a unique covariance or clustering treatment for this pooled evaluation. Consequently, inference metadata from the generated OLS result represents an implementation output and should not be presented as the paper’s uniquely mandated standard-error procedure.

Advanced detail: conventions that affect reproducibility

The paper’s risk-free-rate timing is not fully specified, and return conventions differ across parts of the empirical discussion. The workflow should record whether StockRet is a raw return, a delisting-adjusted return, or an excess return, and which month’s risk-free rate is used. These choices belong in configuration and provenance rather than being inferred silently.

The paper also leaves the exact pooled-versus-firm-specific interpretation of the recursive model unresolved. The generated implementation uses firm-local recursive estimation, with a separate model fit for each PERMNO. That choice preserves the firm-level expanding-window procedure in the method card, but it should be described as an implementation interpretation.

Conceptual worked example

Consider one firm with ordered month rows from January through October. A 30 percent initial fraction produces an initial length of three rows under the generated ceiling rule. January through March form the initial history, March is the first forecast origin, and April is the first target month. The model uses the information available through March, including March’s eligible return, risk-free rate, and lambda. It produces a prediction for April. After that forecast is stored, the training endpoint can move forward for the next origin.

The important objects are:

initial training rows: 3
first origin:          March
first target:          April
forecast record:       firm, origin month, target month,
                       actual return, predicted return,
                       training endpoint and metadata

If a target month is missing—for example, March is followed by May—the implementation does not silently treat May as the next month. It skips that transition because the forecast target is not the immediately subsequent calendar month. This preserves the stated one-month horizon and prevents a calendar gap from being mistaken for a valid forecast.

This example is conceptual and unexecuted. It demonstrates timing, shapes, and responsibilities rather than a numerical forecast or a reproduced paper result.

The resulting boundary is straightforward: recursive forecasts must be generated from information available at each origin, and Equation 18 must be fit only to stored out-of-sample pairs. The paper’s reported forecast and Fama–MacBeth values remain comparison targets; no numerical agreement is claimed here.

Fama–MacBeth inference and signal-sorted portfolios

How can we tell whether a return signal works across firms consistently, rather than only in one pooled regression? The paper uses two related procedures. First, a Fama–MacBeth regression fits a separate cross-sectional regression for each month. Second, a signal-sorted portfolio procedure ranks firms within each month and compares the highest-signal group with the lowest-signal group. Both procedures must use information available at portfolio-formation time; next-month returns are outcomes, not ranking inputs.

The generated implementation treats the firm-month panel as the central input. Each row identifies a firm with PERMNO and a calendar month, contains a current-information signal such as predicted_return or a lambda estimate, and contains a realized next-month return. The resulting monthly coefficient series and portfolio-return series are separate outputs. They should not be replaced by paper-reported values, especially because the supplied run policy disables execution and verification.

Monthly cross-sectional regression

Fama–MacBeth estimation separates the cross-sectional question from the time-series inference question. In a given month, the cross-sectional question is whether firms with different signals have different subsequent returns. The time-series question is whether the monthly slope estimates are consistently different from zero across the usable months.

The paper’s monthly cross-sectional specification is:

Here, r_{i,t+1} is firm i’s realized return in the month after the signal is formed. alpha_t is the intercept estimated separately for month t, and beta_t is that month’s cross-sectional slope. The signal hat(r)^pred_{i,t+1} is the model-implied predicted return associated with firm i; the supplied paper also discusses lambda-based signals, so the code accepts an explicitly selected signal column. The residual epsilon_{i,t+1} is the unexplained part of the firm’s subsequent return.

The function run_monthly_cross_section() in src/liquidity_premium/inference/fama_macbeth.py implements one month of this calculation. Its input is a pandas.DataFrame containing exactly one month’s observations, a numeric signal column, and a realized-return target. It removes rows with nonnumeric or missing signal or target values, requires at least three usable observations, and calls the shared fit_ols() primitive with an intercept. The returned coefficient vector has shape (2,): the intercept followed by the slope. The function labels these coefficients ("intercept", "slope") and records eq_19 in the result metadata.

A focused call looks like this:

from liquidity_premium.inference.fama_macbeth import run_monthly_cross_section

january = panel.loc[panel["month"] == pd.Timestamp("2020-01-01")]
monthly_result = run_monthly_cross_section(
    january,
    signal="predicted_return",
    target="next_month_return",
)

The important detail is month isolation: january is fitted independently of February or March. The function does not pool all firms and months into one cross-sectional coefficient. Its classical OLS fields describe that one month; the Fama–MacBeth aggregation happens afterward.

From monthly slopes to Fama–MacBeth inference

run_fama_macbeth() loops over the month groups in chronological order. For every usable group, it stores one intercept, one slope, the within-month R-squared, and the number of firms used. If no usable months remain, or if the requested HAC lag is incompatible with the number of months, it raises an error rather than silently producing a summary.

A complete call is:

from liquidity_premium.inference.fama_macbeth import run_fama_macbeth

fm_results = run_fama_macbeth(
    panel,
    signal="predicted_return",
    nw_lags=4,
)

The generated implementation recognizes the first available target among next_month_return, StockRet_next, and target. This makes the target-column contract explicit, but it does not determine which return convention the paper intended. The signal is always supplied by the caller because the paper extraction does not fully resolve whether portfolio and Fama–MacBeth rankings should use lambda, predicted return, or another model-implied value.

The resulting DataFrame has one row per usable month. Its monthly columns include intercept, slope, r_squared, and n_observations; summary fields record the mean coefficients, HAC standard errors, HAC t-statistics, selected signal, target, lag, and usable-month count. The mean slope is an average of monthly slopes, not an average over individual firm-level rows. This distinction is the defining aggregation step in the method.

Newey–West inference operates on monthly coefficients

Newey–West, also called HAC or heteroskedasticity-and-autocorrelation-consistent inference, adjusts uncertainty estimates when a time series may have changing variance or serial dependence. Here, the input is not the original firm-month panel. It is the chronological vector of monthly intercepts or slopes, with shape (n_usable_months,).

The helper newey_west_mean_inference() in src/liquidity_premium/inference/newey_west.py requires the lag explicitly:

from liquidity_premium.inference.newey_west import (
    newey_west_mean_inference,
)

slope_statistics = newey_west_mean_inference(
    monthly_slopes,
    lags=4,
)

The function validates a one-dimensional finite array and requires a nonnegative lag smaller than the number of observations. It applies Bartlett weights to autocovariances and returns a dictionary containing mean, standard_error, t_statistic, n_observations, and lags. The paper states that Newey–West adjustment is used but does not specify the lag length. Therefore, 4 is only an implementation example; it is not a paper-imposed value.

This ordering matters:

1. Fit one intercept-and-slope regression per month.

2. Sort those monthly estimates chronologically.

3. Average the monthly estimates.

4. Apply HAC inference to each coefficient time series.

Applying HAC directly to all firm-level rows would answer a different statistical question. The generated test fixture in tests/test_fama_macbeth_portfolios.py is designed to check month-local regression isolation and the propagation of the configured lag, but the test suite was not executed under the authoritative run policy.

Signal-sorted deciles without look-ahead

A portfolio sort translates a continuous signal into groups. For each month, assign_monthly_deciles() ranks firms from low to high using only the selected current-month signal. It assigns integer labels from 1 through 10, represented in the return table as D1 through D10. The next-month return is deliberately not read during assignment.

from liquidity_premium.portfolios.deciles import assign_monthly_deciles

ranked_panel = assign_monthly_deciles(
    panel,
    signal="predicted_return",
    n_deciles=10,
)

The generated function leaves missing or nonfinite signals unassigned. Ties use pandas’ first ranking policy, so tied observations are resolved by their input order. If a month contains fewer than ten eligible firms, the implementation retains the available firms in lower-numbered deciles and leaves unavailable higher deciles missing. These are explicit implementation decisions because the supplied paper does not specify tie handling or small-cross-section behavior.

The paper also does not uniquely specify portfolio weighting. The generated compute_decile_returns() function supports equal weighting and value weighting. Equal weighting computes the arithmetic mean of valid realized returns within each month and decile. Value weighting requires a positive, finite market_cap column.

from liquidity_premium.portfolios.deciles import compute_decile_returns

decile_returns = compute_decile_returns(
    ranked_panel,
    return_column="next_month_return",
    weighting="equal",
)

For value weighting, the corresponding call is:

value_weighted_deciles = compute_decile_returns(
    ranked_panel,
    return_column="next_month_return",
    weighting="value",
)

The output is a DataFrame indexed by month. Its normal shape is (n_months, 10), with columns D1 through D10. Months without a valid return for a particular decile retain a missing value rather than being silently removed. Portfolio formation still depends only on the current signal; return_column is used only after the decile labels exist.

The D10–D1 spread and Sharpe ratios

The long-short portfolio is formed by subtracting the lowest-signal portfolio from the highest-signal portfolio in the same month. The generated function performs this row-wise operation:

from liquidity_premium.portfolios.deciles import compute_long_short_returns

long_short = compute_long_short_returns(decile_returns)

compute_long_short_returns() requires unique monthly indexes and both D1 and D10 columns. Its output has shape (n_months,) and is named D10_minus_D1. It does not rerank firms, filter on returns, or average D10 and D1 over different months. The spread is a same-month subtraction, preserving the portfolio method’s timing invariant.

compute_sharpe_ratio() in src/liquidity_premium/portfolios/performance.py summarizes a monthly return series. The generated implementation uses the sample standard deviation and multiplies the monthly ratio by sqrt(12). If a risk-free series is supplied, it must have exactly the same monthly index and is subtracted without conversion. The paper does not uniquely specify annualization or the conversion of an H.15 yield into a monthly return, so these are implementation decisions rather than paper facts.

from liquidity_premium.portfolios.performance import compute_sharpe_ratio

sharpe = compute_sharpe_ratio(d10_returns)

summarize_portfolio_performance() collects the decile table, the D10–D1 series, and Sharpe statistics into a PortfolioResult. It records that weighting was determined upstream and that no risk-free series was used when none was supplied. No paper-reported Sharpe ratio is inserted into this object.

Worked example: conceptual month and coefficient series

Consider one conceptual month with a small number of firms whose current signals have already been computed. Suppose the signals are ordered from low to high. The portfolio procedure assigns the lowest eligible observations to D1 and the highest eligible observations to D10; with fewer than ten firms, not every decile can be populated. Their realized next-month returns are then averaged within the assigned groups.

For example, after assignment, the monthly return table might have the conceptual structure below. This is an explanatory shape, not an executed or paper-reproduced result:

month       D1      ...     D10
month_t     r_D1    ...     r_D10

The same-month spread is then r_D10 - r_D1. Importantly, the values in the D1 and D10 columns are outcomes observed after formation; they do not determine membership.

For Fama–MacBeth inference, imagine repeating the cross-sectional regression for months t, t+1, and t+2. The output is not one pooled slope but a chronological series:

monthly slopes: [beta_t, beta_t+1, beta_t+2]

newey_west_mean_inference() receives that one-dimensional series and a configured lag. The resulting mean and t-statistic summarize the time series of monthly relationships. Neither this conceptual example nor the code excerpts claim that numerical values were computed in the current run.

Evidence boundaries and unresolved choices

The supplied paper reports Fama–MacBeth results over a stated 58 months, including negative mean slopes and Newey–West t-statistics for the Amihud and regression lambda specifications. Those are paper claims and comparison targets, not independently reproduced values. The extraction also refers inconsistently to Table 9 and Table 10, so this section uses neutral method names rather than treating either table reference as definitive.

Several choices remain visible in the implementation because the paper does not settle them:

• the exact ranking signal may be lambda or predicted return;

• portfolio weighting may be equal or value weighted;

• tie handling and incomplete deciles are not specified;

• the Newey–West lag length is not stated;

• the exact usable-month range is unclear;

• Sharpe-ratio annualization and risk-free conversion are implementation conventions.

The generated files encode these choices as arguments or documented behavior rather than silently harmonizing them. Under the run policy, static verification, semantic code verification, test execution, and numerical reproduction were skipped. The code and examples therefore explain the intended responsibilities and invariants, but they do not establish that the implementation passed those checks or matched the paper’s reported numbers.

Reporting, provenance, limitations, and verification boundaries

How do you know whether a reproduction table contains a computed result, a paper target, or merely a planned output? Treat every reported value as part of an audit record. The record should identify the input sample, filtering and transformation conventions, model specification, observation count, and verification status. A number without that context is not yet evidence of reproduction.

Share Onepagecode

The reporting layer in this implementation is deliberately downstream. It consumes a cleaned panel and already-created result objects; it does not download data, fit missing models, or substitute values from the paper. Its central distinction is:

• a computed result is derived from the local inputs passed to the reporting function;

• a paper target is a value reported by the source paper and supplied only for comparison;

• provenance records the configuration and inputs that determine how a computed result was produced; and

• verification status states whether execution, tests, or independent checks actually occurred.

Under this run policy, generated code was not executed, statically checked, semantically verified, or subjected to final quality review. The reporting design therefore explains how a future run should label outputs; it does not claim that any table matches the paper.

Coverage is derived from the supplied frames

src/liquidity_premium/reporting/coverage.py provides summarize_coverage(daily, panel). Its inputs are two pandas.DataFrame objects: a daily frame containing a firm identifier and date, and a firm-month panel containing a firm identifier and month-like key. The function normalizes identifiers and timestamps, counts unique firms and distinct firm-month keys, and returns a table with metric, value, and source columns.

The function does not hard-code the paper’s reported 9,893 firms or 448,393 firm-month observations. It counts the rows and normalized keys in the frames it receives. This matters because coverage can change with exchange filters, date bounds, point-in-time membership, delisting treatment, missing-data rules, and adjustment conventions.

A focused excerpt shows the contract and the provenance label attached to computed rows:

rows = [
    _coverage_row("daily_unique_permnos", int(daily_keys["permno"].nunique()), "computed_from_daily_input"),
    _coverage_row("daily_unique_dates", int(daily_keys["date"].nunique()), "computed_from_daily_input"),
    _coverage_row("daily_min_date", daily_start, "computed_from_daily_input"),
    _coverage_row("daily_max_date", daily_end, "computed_from_daily_input"),
    _coverage_row("daily_firm_month_observations", int(daily_keys.assign(month=daily_keys["date"].dt.to_period("M")).drop_duplicates(["permno", "month"]).shape[0]), "computed_from_daily_input"),
    _coverage_row("panel_unique_permnos", int(panel_keys["permno"].nunique()), "computed_from_panel_input"),
    _coverage_row("panel_firm_month_observations", int(panel_keys.drop_duplicates(["permno", "month"]).shape[0]), "computed_from_panel_input"),
    _coverage_row("panel_min_month", panel_start, "computed_from_panel_input"),
    _coverage_row("panel_max_month", panel_end, "computed_from_panel_input"),
]

The output is suitable for a coverage table, but it is not a claim that the input universe is equivalent to the paper’s universe. summarize_coverage also fails explicitly when required identifier or time columns are missing, or when PERMNO or dates contain invalid values. Those failures protect the count from silently using an unintended column.

To compare a computed report with a paper value, use compare_coverage_to_paper(report, targets). The targets mapping is caller-supplied. The resulting comparison retains computed_value, paper_target, difference, and status, plus a note that no verification is claimed. A difference is a diagnostic, not proof that either implementation is correct.

Descriptive statistics retain sample meaning

src/liquidity_premium/reporting/descriptive.py contains summarize_variables(frame, variables, kurtosis_method). It accepts a panel and an explicit list of columns, excludes missing and non-finite values separately for each variable, and returns counts, means, sample standard deviations, kurtosis, extrema, and selected quantiles. The count is the number of finite numeric observations actually used for that variable, not automatically the total number of panel rows.

The helper build_table_2_to_4_summaries(panel, kurtosis_method="pandas") organizes available return, volume-feature, and lambda columns into paper-aligned groups. It uses only columns present in the local panel. Missing external variables are not fabricated, and an absent group may produce an empty summary.

Raw and winsorized samples must remain distinguishable. Winsorization clips extreme observations to configured percentile bounds; it does not remove rows. The paper calls for 1st and 99th percentile winsorization but does not fully specify its scope or timing. A report should therefore identify whether a statistic uses raw columns or explicitly named winsorized columns, and should record whether bounds were computed across all observations, by month, or under another configured scope.

The kurtosis convention is also configurable. The implementation accepts pandas, fisher, or pearson; the source material does not uniquely determine which convention should be used. Paper-reported descriptive values, such as the stated monthly-return mean and extreme maximum, remain comparison targets rather than independently reproduced statistics.

Regression tables separate estimates from paper targets

A RegressionResult represents an already-fitted result. format_regression_result(result) converts it into one row per uniquely labeled coefficient. Each row carries the computed coefficient, optional standard error and t-statistic, observation count, R-squared, covariance-method metadata, and specification metadata. build_regression_report(results) combines multiple RegressionResult objects while preserving their labels.

This design prevents a common reporting error: copying a paper coefficient into a local result column when the local model has not been run or has used a different sample. The annotation function instead adds separate comparison fields. The following excerpt is copied from src/liquidity_premium/reporting/regression_tables.py:

annotated = table.copy()
annotated["paper_target"] = np.nan
annotated["paper_target_source"] = pd.NA
unmatched: list[dict[str, Any]] = []

for specification, label, value, source in _target_entries(targets):
    mask = annotated["coefficient_label"].eq(label)
    if specification is not None:
        mask &= annotated["specification"].eq(specification)
    if not bool(mask.any()):
        unmatched.append(
            {
                "specification": specification,
                "coefficient_label": label,
                "paper_target": value,
            }
        )
        continue
    annotated.loc[mask, "paper_target"] = value
    annotated.loc[mask, "paper_target_source"] = (
        source or "paper-reported target; not independently verified"
    )

attach_paper_target_annotations(table, targets) matches targets by specification and coefficient label when those fields are supplied. Label-only targets can match every row with that label. Unmatched targets are stored in the table attributes rather than discarded. Crucially, the original coefficient column is not overwritten.

Observation counts must describe the rows used by that particular regression after missing-value filtering. They must not be copied from the broader panel or from a paper table. Similarly, covariance metadata should describe what the fitted result actually supplied. The reporting layer does not infer clustering, fixed effects, weighting, or standard-error treatment that the paper leaves unspecified.

Orchestration and local export

build_reproduction_report(panel, results, config) in src/liquidity_premium/pipeline/run_reporting.py assembles coverage, descriptive summaries, regression results, and already-structured forecast or portfolio tables. Its panel argument is the computed firm-month input; its results dictionary contains model outputs; and config supplies reporting settings and optional paper targets. It preserves table attributes such as paper identity, schema version, sample provenance, and the fact that paper targets are comparison-only.

The function can use a retained daily frame from results for daily coverage. If no daily frame is available, it falls back to the panel and marks that fallback in report metadata. This is transparent but weaker than reporting coverage from the actual cleaned daily input.

write_reproduction_report(report, output_dir) writes each report table as a local CSV and creates report_manifest.json. write_dataframe in src/liquidity_premium/reporting/export.py accepts .csv and .parquet paths; write_json accepts .json paths and converts common scientific-Python values into JSON-compatible representations. These writers create parent directories when writing, but the existence of an output file would only show that a writer was called. It would not establish that the underlying data or estimates are correct.

The provenance writer records configuration and input paths without asserting that external data are present or sufficient:

payload: dict[str, object] = {
    "schema_version": "1.0",
    "record_type": "liquidity_premium_reproduction_provenance",
    "paper_id": "2607.01377v1",
    "paper_title": "Liquidity Premium and Investment Horizons",
    "configuration": asdict(config),
    "input_paths": [str(input_path) for input_path in inputs],
    "notes": [
        "This record describes a local reproduction configuration and does not contain computed results.",
        "External data availability, preprocessing conventions, and unspecified paper choices affect reproducibility.",
        "Paper-reported targets are not represented as independently reproduced values.",
    ],
}

The command-line entry point scripts/run_reproduction.py loads a local JSON or YAML configuration, builds or loads a panel, calls run_empirical_reproduction, assembles the report, and writes local artifacts. It performs no network access. A typical planned invocation is:

python scripts/run_reproduction.py --config config/reproduction.yaml

This command is an execution example, not evidence that execution occurred in this tutorial.

Worked example: one report row, two kinds of evidence

Consider a conceptual regression table row with the following fields:

specification:             lambda_amihud_with_intercept
coefficient_label:        lambda
coefficient:               computed local estimate
paper_target:              paper-reported comparison value
n_observations:            rows used by this local regression
r_squared:                 local regression R-squared
sample_provenance:         input panel and missing-value policy
paper_target_source:       paper-reported target; not independently verified
verification_status:       not executed or verified

The coefficient and paper_target fields answer different questions. The first would describe a local fitted estimate if the model had been run; the second records what the paper reported. Keeping them in separate columns prevents a target from overwriting a computed result and makes disagreement visible without implying which value is authoritative.

The same principle applies to coverage and descriptive statistics. A computed row should identify its input frame and sample scope. A paper target should identify its source and remain explicitly unverified. A report with no computed model result should contain no invented coefficient merely because a paper table contains one.

What remains unresolved

A full numerical reproduction requires local CRSP daily data and, depending on the analysis, monthly returns, delisting returns, H.15 risk-free rates, WRDS book-to-market and factor data, and potentially point-in-time membership data. The paper extraction does not fully specify several conventions that can change every downstream table:

• how DisFacPr and DisFacShr are applied;

• how raw, delisting-adjusted, and excess returns are defined in each specification;

• how missing prices, unchanged prices, and zero dollar volume are handled;

• the scope and timing of winsorization;

• panel covariance, clustering, fixed effects, and weighting choices;

• whether recursive forecasts are pooled or estimated separately by firm;

• the Newey–West lag length;

• the portfolio signal, tie handling, weighting, risk-free alignment, and annualization; and

• the availability and point-in-time treatment of external datasets.

The source also contains OCR-damaged or incomplete equation records for the closed-form theoretical equilibrium, the continuous-time lambda statement, the volume standard-deviation expression, and Method B. Those formulas are not reconstructed here. The implementation can use the prose-supported sample standard deviation and Method B mapping, but the missing canonical LaTeX remains an evidence limitation.

Reported signs and narratives must also remain as reported. In particular, the paper’s narrative expectations for signed-flow predictability conflict with some negative one-month-ahead coefficients, and its theoretical discussion contains inconsistent statements about noise-trading variance and lambda. Reporting should preserve those distinctions rather than silently harmonize them.

Verification boundary

The planned verification strategy would inspect equation-to-function traceability, daily-row alignment, unique PERMNO-month keys, coefficient shapes, same-firm temporal alignment, forecast chronology, look-ahead leakage, monthly cross-sectional isolation, and separation of paper targets from computed values. It would also confirm that damaged equations are not reconstructed and that provenance accompanies exported tables.

Those checks were not performed here. Local static verification was disabled, code semantic verification was skipped, test generation and execution were disabled, and no final quality review occurred. Consequently, this tutorial can explain intended responsibilities and invariants but cannot claim that the generated files pass them. Semantic plausibility is not empirical validation: only a completed local run with appropriate data and independent checks could support numerical reproduction claims.

Use the button or URL below to download the source code.

Read more

The paper proposes a feature-wise compositional recurrent neural network approach for multivariate stock-price forecasting. The five OHLCV inputs are modeled separately with stacked LSTM, GRU, or SRU recurrent layers, regularized with dropout, fused by concatenation, optionally processed by additional recurrent layers and dense layers, and optimized using either Random Search (RS) or Grey Wolf Optimizer (GWO). The study evaluates 54 configurations on daily Hang Seng Index data from Yahoo Finance. The reported best result is an LSTM-GWO configuration, followed by GRU-GWO and SRU-GWO configurations.

Implementation Assumptions

• The five input channels are ordered Open, High, Low, Close, Volume.

• The 20-day and 40-day values are treated as input window lengths, not forecast horizons.

• Because the paper does not identify the target or horizon, the implementation exposes targetcolumn and forecasthorizon as configuration choices; the tutorial must label these as implementation decisions.

• Scaling defaults to training-only fitting over [0, 0.95] to reduce leakage, while documenting that the paper does not specify the fitting partition.

• The paper's opaque configuration labels such as LSTM-GWO (1-1-0-1) are retained as benchmark labels and are not decoded into architecture parameters.

• The training loss, gradient optimizer, search spaces, GWO budget, and metric formulas are configurable implementation decisions rather than asserted paper facts.

• SRU is represented behind a local interface because the paper does not specify an SRU variant or API.

• Yahoo Finance acquisition is optional for offline reproducibility; local OHLCV files and deterministic synthetic data are supported.

• No reported benchmark is treated as verified reproduction output.

• No canonical equations were supplied, so no equation-level LaTeX or equation_id mapping is asserted.

• Verification is limited to planned static and semantic checks; execution, generated tests, and local verification are disabled by policy.

Scope, evidence status, and reproduction target

What should happen when five market measurements describe the same trading day? The paper’s central idea is to avoid sending all of them immediately into one undifferentiated recurrent pathway. Instead, Open, High, Low, Close, and Volume each receive their own temporal pathway. The resulting representations are then joined before the final forecast. A useful intuition is to treat the five pathways as specialists: each studies one signal over time, and a prediction head combines their reports.

Here, OHLCV means the ordered features Open, High, Low, Close, and Volume. The generated model expects tensors in batch × time × feature layout. For a 20-day input window, for example, one batch has shape batch × 20 × 5; for the alternative window, the shape is batch × 40 × 5. The final dimension must retain the canonical OHLCV order.

What the paper establishes

The supplied paper context supports a feature-wise compositional recurrent method. Each OHLCV stream is processed independently by a stacked LSTM, GRU, or SRU family. Dropout is used for regularization, the five representations are fused through concatenation, and optional recurrent and dense layers produce the stock-price prediction. The paper also compares two outer hyperparameter strategies:

• Random Search (RS) samples candidate configurations stochastically.

• Grey Wolf Optimizer (GWO) maintains a population of candidates and updates them according to their relative objective values.

The paper reports 54 evaluated configurations and identifies an LSTM-GWO configuration as its best reported result. It also reports GRU-GWO and SRU-GWO benchmark configurations. These are useful comparison targets, but they do not fully describe how to construct the corresponding models.

The phrase “univariate encoding” in the extracted paper should be read carefully. The implementation target is not one single-variable forecasting model. It is feature-wise encoding: five separate one-channel sequences are modeled and then fused into a multivariate representation.

What remains underdetermined

An exact reproduction cannot be recovered from the supplied context alone. Several decisions needed by executable code are absent or ambiguous:

• The Yahoo Finance symbol, date range, adjustment policy, and missing-row treatment are not uniquely specified.

• The target column and forecast horizon are not identified. The paper names stock-price prediction, but does not establish that the target is Close or that the horizon is one day.

• The recurrent widths, exact layer counts, hidden-state extraction rule, dense widths, activations, and SRU implementation are missing.

• The training loss, gradient optimizer, learning-rate schedule, stopping rule, and checkpoint policy are unspecified.

• RS bounds, sampling distributions, trial counts, and validation details are absent.

• GWO population size, iteration budget, bounds, initialization, discrete-parameter encoding, and update specification are absent.

• Metric formulas and conventions, including percentage-error zero handling and evaluation scale, are not supplied.

The labels LSTM-GWO (1-1-0-1), GRU-GWO (2-1-1-1), and SRU-GWO (2-2-0-0) therefore remain opaque benchmark labels. They must not be decoded into presumed layer counts, dropout switches, or dense-layer structures.

How the generated package responds

The generated package treats the paper as a method specification plus a set of reported targets, not as a complete executable recipe. DataConfig exposes the target, horizon, window length, split, and scaling range. ModelConfig exposes the recurrent family and architecture choices. TrainingConfig, SearchConfig, and EvaluationConfig make training, search, and metric conventions explicit. validate_reproduction_config provides a common boundary for checking these records.

The model assembly is represented by CompositionalRNN and build_compositional_model. Their intended responsibilities are to create five independent recurrent encoders, concatenate their representations, apply the configured fusion treatment, and produce the configured output dimension. RS and GWO remain separate outer-search interfaces, but both are intended to evaluate candidates through a shared training and validation objective.

This separation is important. A derived implementation decision—such as using Close as the target, a one-step horizon, or a particular loss—can be changed when better evidence becomes available without changing the paper’s feature-wise architecture or the public configuration interfaces. It also prevents a convenient default from being mistaken for a fact reported by the paper.

Worked example: an explicit tutorial configuration

For a small paper-oriented setup, choose window_length=20, target_column='Close', and forecast_horizon=1 in DataConfig. These values are tutorial choices: the paper supports 20-day input windows, but it does not specify Close as the target or a one-step forecast horizon. A corresponding ModelConfig can select LSTM and an explicitly documented number of stream layers and units, while a TrainingConfig records the chosen loss, optimizer, batch size, epochs, and seed.

The resulting input contract is batch × 20 × 5, and the configured output dimension determines the prediction shape. That shape contract describes the scaffold; it does not demonstrate that the configuration reproduces the paper’s reported LSTM-GWO result.

Evidence status

This section and the generated package distinguish three kinds of statements:

1. Paper facts: feature-wise OHLCV processing, recurrent-family comparison, concatenation fusion, RS and GWO comparison, and the reported benchmark labels and values.

2. Derived explanations: the “five specialists” intuition and the interpretation of separate pathways as feature-wise rather than single-variable modeling.

3. Implementation decisions: target and horizon defaults, exact architecture settings, preprocessing policies, training choices, search budgets, and metric conventions.

The run policy disabled execution, local static verification, semantic code verification, generated test execution, tutorial verification, and final quality review. Consequently, this package should be described as a configurable reproduction scaffold, not as a verified implementation or a successful reproduction of the published results.

Data contract: HSI OHLCV acquisition and provenance

How can a forecasting model learn from market data if the source rows, feature order, and cleaning decisions are unclear? The data contract answers that question before any recurrent layer is built. It specifies which five measurements enter the model, preserves their chronological order, and records choices that the paper does not disclose.

What the paper specifies—and what it does not

The paper identifies daily Hang Seng Index history obtained through Yahoo Finance/YFinance and names the raw Open, High, Low, Close, and Volume fields as its OHLCV inputs. In the generated package, the canonical order is fixed as Open, High, Low, Close, Volume. A raw table therefore has shape num_days × 5: each row represents one observation and each column represents one input channel.

The paper does not provide a unique Yahoo Finance ticker, date range, download options, adjusted-price setting, missing-value policy, or duplicate-row policy. These are not minor details. Different tickers or date ranges change the observations, while adjusted and unadjusted prices can produce different price histories. The generated implementation consequently makes these choices explicit and stores them in AcquisitionMetadata rather than presenting a particular choice as recovered paper methodology.

The reproduction scope also stays deliberately narrow. It retains the five raw OHLCV fields and does not add technical indicators or other engineered predictors. Technical indicators are mentioned by the paper as possible future work, not as part of the evaluated method.

Canonical columns and provenance

CANONICAL_OHLCV_COLUMNS is the package-level contract used by acquisition and model-facing code:

# Canonical feature order required by the paper-oriented data and model APIs.
OHLCVColumns: TypeAlias = tuple[str, str, str, str, str]
CANONICAL_OHLCV_COLUMNS: OHLCVColumns = (
    "Open",
    "High",
    "Low",
    "Close",
    "Volume",
)

This excerpt is copied from src/compositional_rnn_stock/types.py. OHLCVColumns describes a five-name tuple, while CANONICAL_OHLCV_COLUMNS supplies the concrete order. Preserving this order matters because later model code interprets the final input axis as five channels. Reordering columns—for example, placing Volume before Close—would change the meaning of the model input without changing its shape.

The acquisition record captures the choices surrounding those columns:

@dataclass(frozen=True)
class AcquisitionMetadata:
    """Provenance and cleaning-policy record for an OHLCV acquisition.

    The paper does not specify the Yahoo Finance symbol, date range, adjustment
    mode, missing-value policy, or duplicate-row policy.  These fields preserve
    the choices made by a local reproduction instead of treating them as paper
    facts.
    """

    source: str
    symbol: str | None
    start: str | None
    end: str | None
    adjusted: bool | None
    frequency: str
    requested_columns: OHLCVColumns
    missing_value_policy: str
    duplicate_policy: str
    ordering_policy: str

This is an implementation record, not an additional claim about the paper. For a network acquisition, source, symbol, start, end, and adjusted identify the request. requested_columns records the five retained fields, and the remaining policy fields describe how the returned table was accepted. For an offline archive, some fields such as symbol may be None, but the artifact still records that the source was a local CSV.

Two acquisition paths

The generated adapter supports a network path through load_hsi_ohlcv_from_yfinance and an offline path through load_ohlcv_csv. The dispatcher requires exactly one source, preventing an ambiguous call that supplies both a symbol and a local file:

def acquire_hsi_ohlcv_data(
    *,
    symbol: str | None = None,
    local_path: Path | None = None,
    start: str | None = None,
    end: str | None = None,
    adjusted: bool = False,
) -> tuple[pd.DataFrame, AcquisitionMetadata]:
    """Dispatch to an explicit local-file or Yahoo Finance acquisition mode."""

    if (symbol is None) == (local_path is None):
        raise ValueError("provide exactly one of symbol or local_path")
    if local_path is not None:
        return load_ohlcv_csv(local_path, CANONICAL_OHLCV_COLUMNS)
    return load_hsi_ohlcv_from_yfinance(symbol=symbol, start=start, end=end, adjusted=adjusted)

This excerpt is copied from src/compositional_rnn_stock/data/acquisition.py. The function returns a pandas table plus its provenance record. In local mode, it delegates to load_ohlcv_csv; in Yahoo Finance mode, it delegates to load_hsi_ohlcv_from_yfinance. The optional yfinance import is isolated inside the network function, so an offline workflow does not need that package merely to load an archived file.

For a local archive, a typical call is:

frame, metadata = load_ohlcv_csv(Path("data/hsi_ohlcv.csv"), CANONICAL_OHLCV_COLUMNS)

The generated CSV loader looks for a conventional Date, date, Timestamp, or timestamp column and uses it as the index when present. It then selects and orders only the canonical OHLCV fields. An archive should therefore preserve its date column whenever the identity of trading days matters. If no recognized date column exists, the loader retains the file's row index; that is a documented limitation of the local input rather than evidence that dates were absent from the original paper dataset.

For Yahoo Finance, the caller must supply the symbol and may supply explicit date bounds and an adjustment choice. The paper does not identify these values, so a tutorial command should not silently substitute a supposedly canonical ticker or time range. The command-line entry point instead requires explicit source and target-related settings, while the acquisition function records the selected source parameters.

Validation and cleaning

Acquisition first validates the structural contract. validate_ohlcv_table requires a nonempty pandas table, all five columns, numeric finite values, an increasing index, and no duplicate index values. It does not infer missing observations or create rows. The relevant responsibility is summarized by its public interface:

def validate_ohlcv_table(frame: pd.DataFrame) -> None:
    """Validate a chronological table containing exactly the five OHLCV fields.

    Validation deliberately rejects missing values and duplicates rather than
    fabricating observations.  The paper leaves those policies unspecified, so
    callers must clean or otherwise resolve such records before acquisition
    results are accepted.
    """

The implementation's cleaning layer makes the missing and duplicate policies explicit:

cleaned = clean_ohlcv(frame, missing_policy="drop", duplicate_policy="reject")

This call is the focused usage pattern exposed by src/compositional_rnn_stock/data/cleaning.py. The `

Scaling, target definition, and sliding windows

How does an ordered table of daily market observations become training data without allowing future information to leak backward? The preprocessing pipeline answers this in three stages: scale the five OHLCV channels, construct overlapping time windows, and split the resulting supervised samples chronologically.

The paper states that the features are normalized to the interval [0, 0.95] and that the model uses 20-day and 40-day temporal windows. It does not supply a canonical scaling equation, identify the target column, or specify the forecast horizon. The generated implementation therefore makes those choices explicit instead of presenting them as recovered paper facts.

Scaling five channels independently

FeaturewiseMinMaxScaler treats Open, High, Low, Close, and Volume as five separate numeric channels. It stores one minimum and maximum for each channel, so the much larger numerical scale of Volume does not determine the scaling of price features. Its final feature axis must have length five, while arbitrary leading dimensions are allowed. Thus it can process both a raw table shaped num_days × 5 and model inputs shaped samples × window_length × 5.

The conventional feature-wise min-max mapping used by the generated scaler is an implementation decision because no canonical equation was supplied. The implementation also chooses to fit statistics on training observations only. This is a leakage-prevention policy: test observations are transformed using training metadata, rather than helping define that metadata. The paper states the target interval but does not state which partition fits the normalization statistics, so this choice must remain visible in provenance.

A focused part of the generated scaler shows the fitting contract:

scaler = FeaturewiseMinMaxScaler(feature_range=feature_range)
scaler.fit(training_values)
return scaler

Here, training_values is expected to contain only the earlier chronological observations selected by the caller. The scaler records feature_min_, feature_max_, and feature-specific scale metadata. Constant training features are mapped to the lower bound and restored to their fitted constant during inverse transformation, avoiding division by zero. Values outside the fitted extrema are not clipped by transform; the generated code documents this as an intentional choice so extrapolation remains visible.

The fit, transform, and inverse_transform methods preserve the input shape. That matters later when a prediction is converted back from normalized units to original price units. If evaluation is performed on original prices, the target column must be identified so that the corresponding feature scaling metadata can be applied.

Target and horizon are explicit choices

The paper describes stock-price prediction and lists OHLCV predictors, but it does not say whether the target is Close, adjusted Close, another price field, or a multivariate output. It also does not specify how far into the future the target lies. The generated windowing API therefore requires both target_index and forecast_horizon.

window_length describes how many past observations enter one model input. For paper-oriented runs, it is treated as either 20 or 40 trading days. forecast_horizon is different: it describes the offset from the end of the input window to the target observation. The 20-day and 40-day values should not be interpreted as prediction horizons merely because they are temporal quantities.

For a start position s, the generated implementation uses the input rows from s through the end of the selected window, then takes the target at the configured future position. With window_length=3 and forecast_horizon=1, the first input contains rows 0, 1, and 2, and its target is row 3. A horizon of 2 would instead select row 4 for that same first input.

The key implementation fragment is copied from make_sliding_windows:

sample_count = day_count - window_length - forecast_horizon + 1
if sample_count <= 0:
    raise ValueError(
        "insufficient observations for the requested window and horizon: "
        f"num_days={day_count}, window_length={window_length}, "
        f"forecast_horizon={forecast_horizon}"
    )

inputs = np.empty((sample_count, window_length, _FEATURE_COUNT), dtype=np.float64)
targets = np.empty(sample_count, dtype=np.float64)

target_timestamps = None if timestamp_array is None else np.empty(sample_count, dtype=timestamp_array.dtype)

for start in range(sample_count):
    end = start + window_length
    target_position = end + forecast_horizon - 1
    inputs[start] = array[start:end]
    targets[start] = array[target_position, target_index]
    if target_timestamps is not None:
        target_timestamps[start] = timestamp_array[target_position]

The resulting WindowedDataset records inputs with shape samples × window_length × 5 and scalar targets with shape samples. It also records the selected target column, horizon, and optional target timestamps. The five channels remain in canonical Open, High, Low, Close, Volume order; window construction does not reorder or mix them.

The following focused example is the same alignment pattern used by the generated contract tests. It uses synthetic arrays only and does not represent Hang Seng Index data:

dataset = make_sliding_windows(
    values,
    target_index=3,
    window_length=3,
    forecast_horizon=2,
    timestamps=timestamps,
)

assert dataset.inputs.shape == (4, 3, 5)
assert dataset.targets.shape == (4,)
np.testing.assert_array_equal(dataset.inputs[0], values[0:3])
np.testing.assert_array_equal(dataset.targets, values[4:8, 3])

This example uses target index 3, which corresponds to Close in the canonical order. That index is an example configuration choice, not evidence that the paper definitively forecasts Close.

Chronological splitting

After windows and targets have been aligned, chronological_split divides complete supervised samples without shuffling. Its boundary is the floor of sample_count * train_fraction. The first partition contains earlier target timestamps, and the second contains later ones. The function rejects a split that would leave either partition empty.

The generated test expresses the central ordering invariant:

train, test = chronological_split(dataset, train_fraction=0.6)

expected_train_count = int(dataset.sample_count * 0.6)
assert train.sample_count == expected_train_count
assert test.sample_count == dataset.sample_count - expected_train_count
assert train.sample_count > 0
assert test.sample_count > 0
assert train.timestamps is not None
assert test.timestamps is not None
assert int(train.timestamps[-1]) < int(test.timestamps[0])

These are contract tests supplied in the generated repository, but they were not executed under the authoritative run policy. They describe intended behavior rather than reporting a completed verification.

The paper reports an 80%/20% chronological partition and gives sample counts of 4,678 training samples and 1,170 test samples. Those counts are useful reproduction targets, but they do not uniquely reveal the raw date range, the number of overlapping windows, or whether the 20-day and 40-day datasets were constructed separately.

How the prepared-data pipeline orders its work

prepare_data records the selected target, horizon, window length, split ratio, scaling range, and cleaning policies. Its relevant flow is:

observation_split = _observation_split_index(
    raw_values.shape[0], data_config.split_ratio
)

scaler = fit_scaler_on_training_observations(
    raw_values=raw_values,
    split_index=observation_split,
    feature_range=data_config.scaling_range,
)
scaled_values = scaler.transform(raw_values)
dataset = make_sliding_windows(
    values=scaled_values,
    target_index=_target_index(data_config.target_column),
    window_length=data_config.window_length,
    forecast_horizon=data_config.forecast_horizon,
    timestamps=timestamps,
)
train_dataset, test_dataset = chronological_split(
    dataset,
    train_fraction=data_config.split_ratio,
)

Notice the distinction between the observation boundary used to fit the scaler and the later sample split. The generated pipeline first determines an observation-level boundary, fits the scaler on the leading observations, transforms the ordered series, constructs windows, and then partitions the completed samples. This is the package's documented leakage-avoidance design, not a uniquely specified procedure from the paper. Because overlapping windows can span an observation boundary, a stronger reproduction should confirm the intended split order from the original experiment details.

The pipeline also records scaler_fit_policy as training_observations_only, along with the target column, horizon, observation counts, window sample counts, and scaling range. These records make it possible to replace an assumption later without changing the public interfaces.

Returning to original units

Normalized inputs are useful for model training, but reported price errors may need to be calculated in original units. FeaturewiseMinMaxScaler.inverse_transform restores all five channels, while a target-specific evaluation adapter can select the configured target feature. This distinction is important because the paper does not specify whether its metrics were calculated on normalized values or inverse-transformed prices.

The generated offline demonstration follows the same policy on synthetic data: it fits the scaler on an earlier chronological portion, transforms the complete ordered sequence, and constructs 20-day windows. The demonstration is useful for checking shape contracts conceptually, but it is not HSI data and cannot establish the paper's reported performance.

In summary, the preprocessing contract is precise about shapes and ordering but deliberately honest about unresolved semantics. The implementation preserves five feature channels, treats 20 and 40 as input lengths, aligns each window with an explicitly configured future target, and prevents test observations from fitting the scaler. The target column, forecast horizon, split order, and normalization convention remain choices that must be documented for any reproduction run.

The feature-wise compositional recurrent model

How can five related market signals contribute to one forecast without being mixed too early? The paper’s answer is feature-wise composition: Open, High, Low, Close, and Volume each travel through an independent recurrent pathway. Their learned summaries are then placed side by side and passed to a prediction head. This is not five separate forecasting models. The final forecast is multivariate because the five representations are fused before prediction.

The paper supports this overall structure, but it does not uniquely specify the recurrent widths, exact layer counts, representation extraction rule, dense-layer widths, or activation functions. The generated package therefore implements these details through ModelConfig. The code is a configurable reproduction scaffold, not a verified reconstruction of every hidden architecture choice.

Tensor flow through five independent streams

The public model input, x, has shape batch × time_steps × 5. The final dimension follows the canonical order Open, High, Low, Close, Volume. A 20-day run therefore uses batch × 20 × 5; a 40-day run uses batch × 40 × 5.

FeatureWiseEncoder splits that final dimension into exactly five tensors. Each slice has shape batch × time_steps × 1, so one recurrent stack receives one feature channel rather than the complete OHLCV vector. The five encoders are separate module instances, which means their parameters are not shared.

The following excerpt shows the stream-level contract. Notice that the recurrent stack is configured with return_sequences=False; this local choice selects the final time-step representation for fusion. The paper does not say whether the final hidden state, a complete sequence, or another aggregation was used.

class FeatureStreamEncoder(nn.Module):
    """Encode one OHLCV channel with its own recurrent stack.

    The input contract is ``(batch, time_steps, 1)`` and the output contract is
    ``(batch, hidden_size)``.  A separate instance is created for every OHLCV
    channel, so parameters are not shared between feature streams.
    """

    def __init__(
        self,
        recurrent_family: RecurrentFamily | str,
        hidden_size: int,
        layers: int,
        dropout_rate: float = 0.0,
        feature_name: str | None = None,
    ) -> None:
        super().__init__()
        if isinstance(hidden_size, bool) or not isinstance(hidden_size, int) or hidden_size <= 0:
            raise ValueError(f"hidden_size must be a positive integer; received {hidden_size!r}")
        if isinstance(layers, bool) or not isinstance(layers, int) or layers <= 0:
            raise ValueError(f"layers must be a positive integer; received {layers!r}")
        if feature_name is not None and feature_name not in CANONICAL_OHLCV_COLUMNS:
            raise ValueError(
                f"feature_name must be one of {CANONICAL_OHLCV_COLUMNS!r}; received {feature_name!r}"
            )

        self.feature_name = feature_name
        self.input_size = 1
        self.hidden_size = hidden_size
        self.layer_count = layers
        self.recurrent_family = RecurrentFamily.coerce(recurrent_family)
        self.recurrent = build_recurrent_stack(
            family=self.recurrent_family,
            input_size=self.input_size,
            hidden_size=hidden_size,
            layers=layers,
            return_sequences=False,
        )
        self.dropout = DualDropout(stream_rate=dropout_rate, fusion_rate=0.0)

The important invariant is input_size=1: every stream receives one feature channel. After recurrent processing, each stream produces a fixed-width tensor with shape batch × hidden_size. Stream dropout is then applied. Dropout is a training-time regularizer: it masks values while the model is training and is disabled when the module is in evaluation mode.

FeatureWiseEncoder.forward preserves the channel order with a one-element split along the final axis:

        # Each slice retains its one-channel axis: (batch, time_steps, 1).
        feature_slices = torch.split(x, split_size_or_sections=1, dim=-1)
        if len(feature_slices) != 5:
            raise RuntimeError(f"expected five feature slices, received {len(feature_slices)}")

        encoded_streams = tuple(
            encoder(feature_slice)
            for encoder, feature_slice in zip(self.encoders, feature_slices)
        )

This code does not perform early averaging, summation, or mixing. The five encoded outputs remain separate until the fusion module receives them.

One interface for LSTM, GRU, and SRU

A recurrent layer processes a sequence while maintaining a learned state that carries information across time steps. LSTM and GRU are the two standard recurrent families exposed by the generated wrapper. The paper also evaluates SRU, but the supplied paper context does not identify an SRU variant or Python API.

RecurrentFamily provides a common selector, and build_recurrent_stack returns a RecurrentStack with the same input and output contract for each family. For a stream, the input is batch × time_steps × 1. With final-time-step reduction, the output is batch × hidden_size.

class RecurrentFamily(str, Enum):
    """Supported recurrent-family selectors.

    ``SRU`` is included because it is one of the families evaluated by the
    paper.  The paper does not identify an SRU variant or Python API, so the
    local adapter below is an explicit implementation choice rather than a
    claim of exact SRU reproduction.
    """

    LSTM = "LSTM"
    GRU = "GRU"
    SRU = "SRU"

For LSTM and GRU, the wrapper constructs batch-first PyTorch stacks. The local SRUAdapter is different: it is a dependency-free compatibility adapter supplied by the generated implementation. Its documentation explicitly says that it is not a canonical SRU implementation. Consequently, selecting SRU makes the interface available for experimentation, but it must not be described as proof of paper-equivalent SRU behavior.

The wrapper also validates rank, feature width, positive time steps, and floating-point input. Invalid inputs fail explicitly rather than being silently reshaped. This matters because changing batch × time_steps × features to another ordering would change the meaning of the recurrent computation.

Dropout and concatenation fusion

The paper describes dual dropout: one stage after feature-specific recurrent processing and another stage around feature fusion. The first stage is clear enough to place after each stream encoder. The second stage is not: “around fusion” does not establish whether it is before concatenation, after concatenation, or both. DualDropout and ConcatenationFusion expose this as fusion_placement.

Concatenation means placing the representations end to end along the final dimension. If each of the five streams has width hidden_size, the fused representation has width 5 × hidden_size. The operation preserves the batch dimension and does not combine channels by summation.

        # The paper's stated fusion operation is concatenation along the
        # representation dimension; stream representations are never summed.
        fused = torch.cat(processed_streams, dim=-1)

        if self.fusion_placement in {"after_concatenation", "both"}:
            fused = self.fusion_dropout.apply_fusion(fused)
        return fused

validate_stream_shapes requires exactly five floating-point tensors, matching leading dimensions, a positive representation width, the same dtype, and the same device. A mismatch such as one stream having a different batch size is rejected before concatenation. The default generated placement applies fusion dropout after concatenation, but that default is an implementation decision rather than a recovered paper setting.

Optional post-fusion processing and prediction

After fusion, PostFusionHead can either send the fused vector directly to dense layers or apply additional recurrent processing first. The paper allows post-fusion recurrent layers but does not specify how a fused vector should be treated as a sequence. The generated head makes one explicit choice: it adds a single time step, giving a tensor shaped batch × 1 × fused_features, and then uses the final representation from that one-step recurrent stack.

Dense layers then produce the configured target output. The generated implementation uses ReLU between configurable dense layers, followed by a final linear layer. Both the widths and the activation choice are local implementation decisions. The output shape is batch × output_dimension, where output_dimension must agree with the separately configured target definition.

Assembling the complete model

CompositionalRNN connects the stages in order: validate the input, encode five streams, concatenate them, apply the optional post-fusion head, and validate the prediction shape. Its central forward path is deliberately explicit:

        # Feature-wise recurrent encoding: five tensors of shape
        # (batch, stream_units), preserving the canonical OHLCV order.
        encoded_streams = self.feature_encoder(x)
        if len(encoded_streams) != self.input_feature_count:
            raise RuntimeError(
                f"expected {self.input_feature_count} encoded streams, "
                f"received {len(encoded_streams)}"
            )

        # Paper architecture stage: fuse representations by concatenation.
        fused = self.fusion(encoded_streams)
        expected_fused_shape = (x.shape[0], self.fused_representation_size)
        if tuple(fused.shape) != expected_fused_shape:
            raise RuntimeError(
                "fusion returned an unexpected shape: "
                f"received {tuple(fused.shape)}, expected {expected_fused_shape}"
            )

        # Optional post-fusion recurrence and dense prediction are delegated to
        # the head; its recurrent representation is a configurable choice.
        prediction = self.post_fusion_head(fused)

The model boundary requires exactly five channels and a positive batch and time dimension. It returns batch × output_dimension; an unexpected output shape raises an error. These checks enforce the paper-supported architecture without pretending that unspecified widths or layer counts have been recovered.

Focused configuration example

The following excerpt is adapted exactly from the generated model-contract test. It demonstrates a small local configuration with one recurrent layer, eight stream units, one dense layer, and one output. Those values are demonstration settings, not the paper’s reported best architecture. The 20 in the input shape represents the paper-oriented window choice, while the target semantics remain configured elsewhere.

def _make_model_config(
    family: RecurrentFamily | str = RecurrentFamily.LSTM,
    *,
    stream_dropout: float = 0.0,
    fusion_dropout: float = 0.0,
) -> ModelConfig:
    """Build a small deterministic configuration for contract tests.

    These tests verify structural invariants of the implementation rather than
    the paper's unresolved architecture details or opaque benchmark labels.
    """
    return ModelConfig(
        recurrent_family=family,
        stream_layers=1,
        stream_units=8,
        stream_dropout=stream_dropout,
        fusion_dropout=fusion_dropout,
        fusion_placement="after_concatenation",
        post_fusion_layers=0,
        post_fusion_units=8,
        dense_layers=(8,),
        output_dimension=1,
    )

A caller would construct the model and prepare an input with shape batch × 20 × 5 as follows:

model = build_compositional_model(_make_model_config())
inputs = _make_inputs(batch_size=4, time_steps=20)

predictions = model(inputs)

The intended contract is predictions with shape 4 × 1 for this configuration. This excerpt was not executed under the run policy, so it demonstrates the interface and expected shapes only; it does not establish runtime correctness or paper-level reproduction.

The opaque labels in the reported results—LSTM-GWO (1-1-0-1), GRU-GWO (2-1-1-1), and SRU-GWO (2-2-0-0)—are deliberately not translated into ModelConfig fields. The supplied paper context does not define what those four positions mean. Keeping them as benchmark labels prevents an attractive but unsupported architecture claim.

The generated test plan reinforces structural invariants such as five independent streams, additive fusion width, common family dispatch, and training-only dropout. Those tests were not executed: local static verification, semantic verification, and code execution were disabled or skipped by policy.

Training one candidate without hiding missing specifications

What does it mean to train one candidate model in this reproduction? The candidate receives batches of chronological windows, produces predictions, computes a selected scalar loss, and updates its trainable parameters with gradients. After each epoch, a separate validation pass records loss with dropout disabled and without changing the parameters. This inner training loop is distinct from Random Search and Grey Wolf Optimizer (GWO), which choose different candidate configurations outside the loop.

Paper facts and implementation choices

The paper says that recurrent models are trained and compared using validation performance, and it names learning rate, batch size, and training epochs among the optimized hyperparameters. However, the supplied paper context does not specify the training loss, base gradient optimizer, initialization, learning-rate schedule, random seed, batch-shuffling policy, early-stopping rule, or checkpoint-selection rule. It also does not clearly define how the reported 80%/20% partition relates to the validation loss discussed in the method.

The generated code makes these gaps visible. TrainingConfig supplies the local choices, while ForecastLoss supports explicit mse, mae, and huber objectives. These are available implementation options, not claims about which loss reproduced the paper. Likewise, make_optimizer supports explicit gradient optimizers for a candidate; this inner optimizer should not be confused with RS or GWO, which operate at the outer configuration-search level.

The generated pipeline uses the prepared test partition as the validation loader for a one-candidate run. Its source documentation explicitly calls this an implementation decision because the paper does not define a separate validation construction. Consequently, this scaffold should not be described as recovering the paper’s exact validation protocol.

Validating predictions and targets

A forecasting loss is meaningful only when each prediction is paired with its corresponding target. validate_loss_inputs requires PyTorch tensors with floating-point dtypes, identical shapes, at least one element, and finite values. A mismatch, empty batch, integer tensor, or non-finite value raises an error rather than allowing silent broadcasting or an invalid objective.

The public loss interface is ForecastLoss. Its output is a scalar tensor, so it can participate in backpropagation. The generated implementation uses mean reduction for each supported local objective. The following excerpt shows the explicit dispatch; it is copied from src/compositional_rnn_stock/losses.py.

        if self.name == "mse":
            loss = F.mse_loss(predictions, targets, reduction="mean")
        elif self.name == "mae":
            loss = F.l1_loss(predictions, targets, reduction="mean")
        else:
            loss = F.smooth_l1_loss(
                predictions,
                targets,
                beta=self.huber_delta,
                reduction="mean",
            )

The inputs have the model’s configured target shape. In the scalar-target pipeline, the model returns batch × 1, while window datasets expose targets as batch. The training loop performs only this unambiguous single-output reshape; other mismatches are rejected. That safeguard matters because accidental broadcasting could produce a finite-looking loss with incorrect semantics.

One epoch: update in training mode, evaluate in validation mode

During training, train_one_candidate puts the model in training mode, reads a batch shaped batch × time_steps × 5, computes predictions, aligns the targets, backpropagates the scalar loss, and calls the selected optimizer. The five-channel input contract is checked before the model is used. A training loader with no batches or non-finite values is rejected.

The core update sequence is shown below. This excerpt is copied from src/compositional_rnn_stock/training/loops.py.

        for batch in train_loader:
            inputs, targets = _prepare_batch(batch, device, "training")
            optimizer.zero_grad(set_to_none=True)
            predictions = model(inputs)
            targets = _align_targets(predictions, targets)
            loss = loss_fn(predictions, targets)
            loss.backward()
            optimizer.step()

After the training batches, evaluate_loss switches the model to evaluation mode and wraps inference in torch.no_grad(). Evaluation mode is important for the compositional model because dropout must be disabled when validation predictions are measured. The function restores the model’s previous training state afterward, computes a value-weighted mean over the validation targets, and rejects an empty or non-finite result.

    was_training = model.training
    model.eval()
    total_loss = 0.0
    total_values = 0
    try:
        with torch.no_grad():
            for batch in loader:
                inputs, targets = _prepare_batch(batch, device, "validation")
                predictions = model(inputs)
                targets = _align_targets(predictions, targets)
                loss = loss_fn(predictions, targets)
                value_count = int(targets.numel())
                total_loss += float(loss.detach().cpu().item()) * value_count
                total_values += value_count
    finally:
        model.train(was_training)

This separation is the main training invariant: validation loss is observed, not optimized directly. The validation loader is never passed to backward() or optimizer.step() by this workflow. The paper’s exact split and model-selection policy remain unresolved, so the generated code retains the final state after the configured epochs rather than silently inventing a best-checkpoint rule.

Optimizer and reproducibility settings

make_optimizer constructs the inner gradient optimizer from the configured name and learning rate. The generated implementation accepts adam, adamw, and sgd in this function. It validates that the learning rate is positive and finite, that the model has trainable parameters, and that the optimizer name is supported. The paper does not identify which of these, if any, was used.

    if normalized == "adam":
        return Adam(parameters, lr=learning_rate)
    if normalized == "adamw":
        return AdamW(parameters, lr=learning_rate)
    return SGD(parameters, lr=learning_rate)

set_global_seed seeds supported local Python, NumPy, and PyTorch generators. SeedContext additionally snapshots and restores available random states and can request deterministic PyTorch algorithms. These facilities improve the comparability of local experiments, but the paper does not report its seed or deterministic-backend settings. A seed therefore belongs in provenance, not in a claim that the paper used the same setting.

Histories and provenance records

Each completed epoch becomes an EpochRecord containing its epoch number, training loss, and validation loss. TrainingHistory preserves insertion order and rejects non-increasing epoch numbers. Its best_epoch helper can identify the earliest record with the smallest available validation loss, but the training loop does not automatically restore that epoch’s weights. Selecting and restoring a best checkpoint would be an additional explicit implementation decision.

The outer record is TrainingRun. It links the trained CompositionalRNN, its TrainingHistory, the ModelConfig, the TrainingConfig, and a prepared-data identifier. That identifier records acquisition and preprocessing metadata, window length, forecast horizon, target column, and train/test sample counts. This prevents a local loss curve from becoming detached from choices such as Close versus another target, a 20-day versus 40-day window, or a particular scaling policy.

Worked example: defining a local candidate run

The following configuration fragment is copied from scripts/train_model.py. It selects Close and a one-step horizon, but those are tutorial-level implementation choices because the paper does not identify the target column or forecast horizon.

    data_config = DataConfig(
        symbol=args.symbol,
        local_path=args.local_path,
        start=args.start,
        end=args.end,
        target_column=args.target_column,
        forecast_horizon=args.forecast_horizon,
        window_length=args.window_length,
        split_ratio=args.split_ratio,
        scaling_range=(0.0, 0.95),
    )

A focused call path inside train_one_candidate then creates the selected loss and optimizer and trains for the configured number of epochs. This excerpt is copied from src/compositional_rnn_stock/training/loops.py.

    loss_fn = _loss_for_config(config)
    optimizer = make_optimizer(model, config)
    device = _model_device(model)
    history = TrainingHistory()

At the pipeline level, run_training validates the prepared data and configurations, builds chronological loaders, constructs the model, checks that loader inputs have shape batch × time_steps × 5, and delegates to train_one_candidate.

    trained_model, history = train_one_candidate(
        model=model,
        train_loader=train_loader,
        validation_loader=validation_loader,
        config=training_config,
    )

This call defines one local experiment. It does not establish that the loss, optimizer, architecture settings, validation protocol, or final model state match the paper. It also does not execute here: the authoritative run policy disabled code execution and verification.

Why this boundary matters

The gradient loop answers, “How do we fit one selected candidate?” RS and GWO answer a different question, “Which candidate settings should we try next?” Keeping those responsibilities separate allows both search methods to use the same loss, loaders, validation objective, and provenance schema. It also makes missing paper specifications inspectable instead of burying them in defaults.

For this scaffold, the honest output of training is a locally configured model, an epoch-aligned history, and provenance describing the choices. No training result, convergence behavior, or successful reproduction of the paper’s reported benchmarks is claimed. Verification was skipped under policy, so the code excerpts describe the generated interfaces and intended control flow rather than a checked execution outcome.

Random Search and Grey Wolf Optimizer

How should a reproduction choose among many possible recurrent-network recipes? Random Search (RS) samples independent candidates, while Grey Wolf Optimizer (GWO) maintains a population of candidates and moves them using the best-ranked candidates as guides. In both cases, a candidate is more than a model family: it can include recurrent-layer count, recurrent units, dropout rates, learning rate, batch size, and training epochs.

The paper names both RS and GWO and reports that GWO performed better across the LSTM, GRU, and SRU families. However, it does not specify the search bounds, probability distributions, number of trials, wolf population, iteration count, initialization, update settings, invalid-candidate policy, or randomization controls. The generated package therefore treats these as explicit implementation configuration. No displayed setting in this section should be read as the paper's missing experimental setting.

One objective shared by both methods

The outer optimizer does not train a model directly. Instead, it proposes a typed CandidateConfig. The shared candidate objective builds the corresponding compositional model, creates or receives chronological training and validation loaders, trains the candidate, and records its best validation loss. Validation loss is used here as a local implementation choice because the paper discusses validation performance but does not define the loss or the exact selection rule.

CandidateResult stores the candidate configuration, objective, metrics, provenance, success status, and any failure information. SearchResult stores the selected result and the complete search history. This separation matters: a failed candidate remains visible in the history rather than being silently treated as the best candidate, and the held-out test set is not needed for hyperparameter selection.

The objective's public protocol is deliberately small. Its candidate has type CandidateConfig, its data bundle must provide training and validation loaders, and its seed is recorded for reproducibility. The generated implementation explicitly records that test data were not used for selection:

provenance: dict[str, Any] = {
    "seed": seed,
    "objective": "minimum_validation_loss",
    "selection_basis": "lowest validation loss recorded during training",
    "test_data_used_for_selection": False,
}

This excerpt comes from src/compositional_rnn_stock/search/objectives.py. The objective returns a CandidateResult after train_one_candidate supplies a TrainingHistory; the history must contain a validation loss from which the lowest recorded value can be selected. The code does not claim that this objective is the paper's exact loss or checkpoint policy.

The same selection helper serves both optimizers. Its minimize argument makes the direction explicit, which is important because validation loss is normally minimized, whereas some metrics are maximized:

def select_best_candidate(
    results: Sequence[CandidateResult],
    minimize: bool,
) -> CandidateResult:
    """Select a successful candidate using an explicit objective direction."""
    if not isinstance(minimize, bool):
        raise TypeError(f"minimize must be boolean; received {type(minimize).__name__}")
    if not isinstance(results, Sequence):
        raise TypeError("results must be a sequence of CandidateResult instances")

    validated: list[CandidateResult] = []
    for index, result in enumerate(results):
        if not isinstance(result, CandidateResult):
            raise TypeError(
                f"results[{index}] must be a CandidateResult; "
                f"received {type(result).__name__}"
            )
        if result.success:
            if result.objective is None:
                raise ValueError(
                    f"successful results[{index}] must contain an objective"
                )
            validated.append(result)

    if not validated:
        failure_count = sum(1 for result in results if not result.success)
        raise ValueError(
            "cannot select a best candidate: no successful candidate results "
            f"were available ({failure_count} recorded failures)"
        )

    if minimize:
        return min(validated, key=lambda result: float(result.objective))
    return max(validated, key=lambda result: float(result.objective))

The function accepts a sequence of results, rejects malformed successful records, ignores unsuccessful records for selection, and raises an error when no successful candidate exists. That failure behavior is a derived implementation safeguard, not a reported paper procedure.

Typed search spaces

A search space must represent different kinds of values. Recurrent-layer counts, units, batch sizes, and epochs are integer-valued. Dropout rates and learning rates are continuous. A model family is categorical when one search spans LSTM, GRU, and SRU. ParameterDomain gives each parameter one of these meanings and validates its bounds or allowed values.

The following excerpt is copied from src/compositional_rnn_stock/search/space.py and shows the domain constructors exposed by the generated package:

    @classmethod
    def continuous(cls, lower: float, upper: float) -> "ParameterDomain":
        return cls("continuous", lower=lower, upper=upper)

    @classmethod
    def integer(cls, lower: int, upper: int) -> "ParameterDomain":
        return cls("integer", lower=lower, upper=upper)

    @classmethod
    def categorical(cls, values: Sequence[Any]) -> "ParameterDomain":
        return cls("categorical", values=tuple(values))

    @classmethod
    def discrete(cls, values: Sequence[Any]) -> "ParameterDomain":
        return cls("discrete", values=tuple(values))

continuous and integer domains use inclusive lower and upper bounds. categorical and discrete domains hold explicit values. The distinction between categorical and discrete values is useful when documenting intent, although both are represented by positions when a vector-based optimizer needs numeric coordinates.

SearchSpace keeps the named domains in a stable order and exposes encoded bounds. CandidateConfig then groups the decoded values into ModelConfig, TrainingConfig, and any explicitly declared extra parameters. candidate_from_mapping validates a named mapping before constructing that typed record. The paper's list of optimized hyperparameters tells us what should be represented, but not the actual ranges; those ranges must be supplied by the caller.

Random Search: independent proposals

RS draws one value from every domain for each trial. The generated random_search function accepts a SearchSpace, a callable CandidateObjective, and a search configuration containing a seed, trial count, data bundle, and objective direction. Each trial receives a derived candidate seed, and each result is appended to the history, including failures.

The core loop is implemented as follows in src/compositional_rnn_stock/search/random_search.py:

    history: list[CandidateResult] = []
    for trial_index in range(trial_count):
        candidate = sample_random_candidate(space, rng)
        candidate_seed = seed + trial_index
        try:
            result = objective(candidate, data_bundle, candidate_seed)
            if not isinstance(result, CandidateResult):
                raise TypeError(
                    "objective must return CandidateResult; "
                    f"received {type(result).__name__}"
                )
        except Exception as exc:
            result = CandidateResult(
                configuration=candidate,
                objective=None,
                metrics={},
                provenance={
                    "method": "random_search",
                    "trial_index": trial_index,
                    "seed": candidate_seed,
                    "failure_type": type(exc).__name__,
                },
                success=False,
                error=f"{type(exc).__name__}: {exc}",
            )
        else:
            result.provenance.update(
                {
                    "method": "random_search",
                    "trial_index": trial_index,
                    "sampler_seed": seed,
                    "candidate_seed": candidate_seed,
                }
            )
        history.append(result)

Notice the two seeds recorded in successful results: the sampler seed identifies the sequence of sampled candidates, while the candidate seed identifies the training run. This is a useful provenance design for comparing methods locally. It does not establish that the paper used the same seeding scheme.

A tutorial experiment might declare three trials for a quick demonstration, but that would be a tutorial budget only. The paper's statement that 54 configurations were evaluated must not be reverse-engineered into a presumed RS trial count or into a presumed division between model families and optimizers.

GWO: population-guided proposals

GWO works with a matrix of wolf positions. A position is a numeric vector whose coordinates correspond to the ordered parameters in SearchSpace. The best three successful candidates are called alpha, beta, and delta in the generated implementation. Their encoded vectors guide a new population. After each update, positions are clipped to the declared bounds and decoded back into valid typed candidates.

The generated update function accepts positions with shape population × encoded_dimension. Each leader vector has shape encoded_dimension. The update is intentionally documented as a local GWO-style choice because no canonical GWO equations or settings were supplied in the paper:

def update_wolf_positions(
    positions: Any,
    alpha: Any,
    beta: Any,
    delta: Any,
    iteration: int,
    total_iterations: int,
    rng: random.Random,
) -> np.ndarray:
    """Return one local GWO-style update for every wolf.

    ``positions`` has shape ``(population, encoded_dimension)`` and each leader
    has shape ``(encoded_dimension,)``.  The implementation uses the usual
    linearly decreasing exploration coefficient and independent random draws.
    These update rules are implementation decisions, not equations supplied by
    the paper.
    """
    current = np.asarray(positions, dtype=float)
    leader_arrays = [np.asarray(value, dtype=float) for value in (alpha, beta, delta)]
    if current.ndim != 2:
        raise ValueError("positions must have shape (population, encoded_dimension)")
    if current.shape[0] < 1:
        raise ValueError("positions must contain at least one wolf")

The remainder of the function validates leader shapes and iteration bounds, generates independent random coefficients, averages the three leader-guided proposals, and returns an array with the same population-by-dimension shape. The linearly decreasing exploration coefficient, random update details, and alpha/beta/delta ranking are implementation decisions rather than recovered paper settings.

Decoding mixed parameters safely

A continuous vector cannot directly be used as a batch size or a model-family name. CandidateCodec bridges that gap. It encodes typed values into a stable vector ordering and decodes them by clipping to the domain, rounding integer-like coordinates, selecting categorical positions, and constructing validated ModelConfig and TrainingConfig records.

The public decoding contract is concise:

    def decode(self, vector: Sequence[float] | np.ndarray) -> CandidateConfig:
        array = np.asarray(vector, dtype=float)
        if array.ndim != 1 or array.shape[0] != self.dimension:
            raise ValueError(
                f"encoded candidate must have shape ({self.dimension},); received {array.shape}"
            )
        if not np.all(np.isfinite(array)):
            raise ValueError("encoded candidate must contain only finite values")
        values = {
            name: self.space.domains[name].decode_value(float(array[index]))
            for index, name in enumerate(self.space.parameter_names)
        }
        return self._build_candidate(self.space.validate_candidate(values))

For example, a vector coordinate intended for batch sizes (8, 16, 32) is clipped to the valid index range and rounded to one of those positions. That prevents GWO from producing an invalid fractional batch size. It also means that several nearby continuous positions can decode to the same typed candidate; this is an expected consequence of using a continuous optimizer for mixed parameters.

Worked configuration example

The following excerpt is a small configuration-only example based on the generated search-contract test. It demonstrates integer, continuous, and discrete domains without asserting that these bounds came from the paper:

PARAMETER_DOMAINS = {
    "stream_layers": ParameterDomain.integer(1, 3),
    "units": ParameterDomain.integer(8, 32),
    "dropout": ParameterDomain.continuous(0.0, 0.5),
    "learning_rate": ParameterDomain.continuous(1.0e-4, 1.0e-2),
    "batch_size": ParameterDomain.discrete((8, 16, 32)),
    "epochs": ParameterDomain.integer(1, 4),
}

space = SearchSpace(domains=PARAMETER_DOMAINS)
codec = CandidateCodec(space)

Here, stream_layers, units, batch_size, and epochs are discrete choices, while dropout and learning_rate vary continuously. codec can translate a validated CandidateConfig into a numeric vector for GWO and decode a clipped vector back into a typed candidate. The displayed ranges are deliberately modest tutorial settings; they are not paper-reported bounds.

In a complete local run, the same space and shared objective can be passed to RS or GWO. The pipeline functions run_random_search and run_gwo_search bind both methods to a chronological training/validation split created from the prepared training partition. Their documented contract excludes PreparedData.test from selection, so the held-out test data are reserved for evaluation after a candidate has been chosen.

Practical limits of the search reproduction

A fair local comparison requires RS and GWO to use the same feature order, scaling policy, candidate-training routine, validation partition, objective, and failure policy. Only the proposal mechanism should differ. The generated pipeline records the validation fraction as a local choice and shares the candidate objective, but this does not recover the paper's undisclosed validation protocol.

The paper's GWO superiority claim remains a reported paper result, not a result established by this scaffold. In particular, the generated GWO update, population size, iteration count, bounds, initialization, and discrete decoding cannot be labeled as the paper's implementation. The four-part labels in LSTM-GWO (1-1-0-1), GRU-GWO (2-1-1-1), and SRU-GWO (2-2-0-0) also remain opaque; the search code does not decode them.

The related contract tests are intended to check local invariants such as candidate decoding, seeded RS sampling, GWO shape and bound handling, and the common objective interface. They were not executed in this run. Static verification and semantic code verification were also skipped, so the generated files should be reviewed before relying on them operationally. No search, training run, or reproduction result is claimed here.

Held-out evaluation, metrics, and residual analysis

How do we know whether a forecast is useful? First, each prediction must be paired with the correct held-out observation. Then both arrays must be evaluated under the same scale and metric conventions. The evaluation layer therefore aligns predictions and targets, optionally restores original price units, computes the seven metrics named by the paper, and summarizes residual behavior.

This section distinguishes three kinds of statements. The paper names the metrics and discusses error distributions, but it does not provide canonical formulas, denominator rules, residual signs, or plotting conventions. The generated implementation supplies explicit conventional choices for those missing details. The resulting values would be local evaluation outputs—not verified reproductions of the paper's reported results.

From model output to aligned evaluation arrays

The generated predict_dataset function runs inference in loader order. It switches the model to evaluation mode so dropout is disabled, uses no-gradient inference, collects scalar predictions and targets batch by batch, and restores the model's previous training-mode flag afterward. Its scalar-target contract accepts arrays shaped samples or samples × 1. It does not reorder or silently truncate samples.

The key alignment operation is exposed separately as align_predictions_and_targets. It rejects unequal sample counts and preserves the order emitted by the loader. This is why held-out loaders should normally use shuffle=False: a shuffled loader can still produce pairs within a batch, but it no longer represents the original chronological output order.

The following excerpt is copied from predict_dataset. Notice the two independent checks: the model output and the batch targets are normalized to the scalar-target representation, and their batch counts must agree before either is appended.

                outputs = model(inputs)
                if not isinstance(outputs, Tensor):
                    raise TypeError(
                        f"model output for batch {batch_index} must be a tensor"
                    )
                predictions, _ = _normalise_scalar_targets(outputs, "predictions")
                batch_targets, _ = _normalise_scalar_targets(targets, "targets")
                if predictions.shape[0] != batch_targets.shape[0]:
                    raise ValueError(
                        f"batch {batch_index} prediction and target counts differ: "
                        f"{predictions.shape[0]} != {batch_targets.shape[0]}"
                    )
                prediction_parts.append(predictions)
                target_parts.append(batch_targets)

The function returns predictions in loader order, while the targets are retained internally for alignment validation. The model input remains the package-wide batch × time × 5 contract; this evaluation helper specifically supports the configured scalar-output case. A multivariate target would require an explicit extension rather than an implicit interpretation.

Choosing the evaluation scale

The paper states that inputs are scaled to [0, 0.95], but it does not say whether its reported metrics were computed in normalized space or after inverse transformation to price units. The generated pipeline exposes both choices through EvaluationConfig.

In normalized evaluation, actual_targets and predictions remain in scaled space. RMSE then has normalized units. In original-unit evaluation, inverse_transform_target embeds each scalar target into a five-feature row, applies FeaturewiseMinMaxScaler.inverse_transform, and extracts the configured target column. RMSE can then be interpreted in price units. MAPE, RMSPE, and PBIAS remain percentage-valued under the local conventions, while agreement metrics remain unitless.

The target column itself is also unresolved by the paper. The generated pipeline requires a configured choice such as Close; it does not assume that the paper's phrase “stock price” uniquely means Close or adjusted Close. The forecast horizon is likewise recorded separately from the input window_length.

The following excerpt is copied from inverse_transform_target and shows why the scaler needs a target index even though the prediction contains only one value per sample.

    embedded = np.zeros((flat_values.shape[0], scaler.n_features), dtype=np.float64)
    embedded[:, target_index] = flat_values
    restored = scaler.inverse_transform(embedded)[:, target_index]

    if len(original_shape) == 1:
        return restored
    return restored.reshape(original_shape)

This operation preserves the prediction shape while using the scaler's five feature-specific statistics. It is an implementation mapping, not a formula recovered from the paper. The scaler must already be fitted, and its fitting policy is recorded as training-only in the generated preprocessing pipeline to reduce leakage.

The seven named metrics

The paper names R², RMSE, MAPE, RMSPE, PBIAS, Willmott Index, and NSE. Because no canonical equation records were supplied, the generated metrics.py module documents and implements conventional local definitions. The important practical point is consistency: every candidate must use the same scale, alignment rule, sign convention, and zero-denominator policy.

• R² summarizes explained variation relative to an actual-value mean baseline. The generated function stores the conventional result on the unit-interval scale by default. A display convention can multiply it by 100, which supports the paper's percentage-style reporting. Thus a paper display of 99.2427% is distinct from the underlying unit-interval representation 0.992427.

• RMSE measures the typical magnitude of prediction error in the units supplied to the function. It is therefore scale-sensitive: normalized inputs yield normalized-unit RMSE, while inverse-transformed prices yield price-unit RMSE.

• MAPE averages absolute relative errors and returns a percentage. It cannot divide by zero actual values, so EvaluationConfig selects whether such observations raise an error, are ignored, or receive a defined local treatment.

• RMSPE is the root-mean-square version of relative error and uses the same zero-actual policy.

• PBIAS summarizes signed aggregate bias. The generated default residual direction is actual_minus_predicted, so positive PBIAS indicates underprediction under that convention.

• Willmott Index measures agreement using the generated module's selected standard convention. Its denominator can be undefined for degenerate data, in which case the local implementation returns NaN.

• NSE, or Nash–Sutcliffe Efficiency, compares squared forecast error with variation around the actual-series mean. Like R², it is undefined when the actual series has no variation.

The module's report assembler keeps these choices together. The following excerpt is copied from compute_metric_report; it shows that one configuration controls the conventions used by the individual metric functions.

    _validate_zero_policy(str(zero_policy))
    return {
        "r_squared": r_squared(actual_vector, predicted_vector, str(r2_convention)),
        "rmse": rmse(actual_vector, predicted_vector),
        "mape": mape(actual_vector, predicted_vector, str(zero_policy)),
        "rmspe": rmspe(actual_vector, predicted_vector, str(zero_policy)),
        "pbias": pbias(actual_vector, predicted_vector, str(pbias_convention)),
        "willmott_index": willmott_index(
            actual_vector,
            predicted_vector,
            str(willmott_convention),
        ),
        "nash_sutcliffe_efficiency": nash_sutcliffe_efficiency(actual_vector, predicted_vector),
    }

Before reaching this block, _aligned_arrays converts supported array-like inputs to finite one-dimensional vectors and rejects mismatched shapes. This failure behavior matters: a metric calculated from mispaired samples can look numerically plausible while describing the wrong forecast errors.

A small illustrative interface example

The following excerpt is adapted directly from the generated metric test's synthetic setup. It demonstrates the intended interface with short local arrays; it is not HSI data, and no numerical result from this example should be compared with the paper's benchmarks. The test uses EvaluationConfig() to select the generated defaults rather than claiming that those defaults were specified by the paper.

    actual = np.array([100.0, 105.0, 110.0, 115.0], dtype=float)
    predicted = np.array([99.0, 106.0, 109.0, 116.0], dtype=float)

    # Defaults are deliberately supplied by EvaluationConfig rather than
    # inferred from the paper, whose metric conventions are unspecified.
    report = compute_metric_report(actual, predicted, EvaluationConfig())

A corresponding residual summary can be created from the same aligned arrays. In the generated implementation, compute_residuals(actual, predicted) returns actual - predicted with the same shape, and summarize_residuals flattens the result before calculating its count, mean, population standard deviation, standardized skewness, and excess kurtosis. Fewer than three observations, or zero variance, leave skewness undefined; fewer than four observations, or zero variance, leave excess kurtosis undefined. These estimator and insufficiency rules are implementation decisions because the paper only mentions mean, standard deviation, skewness, and kurtosis without defining their conventions.

Residuals and visual analysis

Residuals make systematic behavior easier to inspect than a single score. Under the generated actual_minus_predicted convention, positive residuals indicate underprediction and negative residuals indicate overprediction. compare_error_distributions creates a stable table for named residual groups, preserving empty groups with an explicit count and undefined summary values rather than inventing observations.

The visualization adapters implement the analysis types mentioned by the paper:

• plot_error_boxplots shows grouped residual spread and central tendency.

• plot_error_violins shows distribution shape.

• plot_metric_comparison gives each metric its own panel so percentage and unitless measures are not placed on one misleading axis.

• plot_pbias_radar displays signed PBIAS values using a locally selected radial convention.

• plot_taylor_summary uses a locally selected correlation-angle and standard-deviation representation.

These functions operate on local predictions and reports. They do not contain the paper's plotted points, styling, axes, or source predictions. Therefore they can support analogous analysis, but they cannot claim exact reproduction of the paper's figures.

Evaluate only after model selection

The evaluation pipeline enforces the intended experiment order conceptually: train candidates and compare them using the configured validation objective, select a candidate, and only then read the chronological held-out partition for final metrics. evaluate_training_run evaluates a retained trained model. evaluate_search_result requires that a selected trained model be retained; it does not silently retrain from an incomplete search record.

The resulting EvaluationRun stores predictions, aligned actual targets, residuals, the metric report, and provenance. Provenance includes the evaluation scale, target column, forecast horizon, window length, metric conventions, residual sign, held-out sample count, and preprocessing metadata. This makes a future comparison auditable: a difference in RMSE can be investigated as a possible scale, target, or convention difference rather than treated as unexplained model behavior.

The paper's seven metrics and its discussion of residual distributions are therefore represented, but not overclaimed. The formulas and conventions are local implementation choices, the target and evaluation scale remain configurable, and no execution or verification occurred in this run. Any future local report must be labeled as a computed result and kept separate from the paper's reported benchmark records.

Reported LSTM-GWO, GRU-GWO, and SRU-GWO benchmarks

How should you use the paper’s reported numbers when the original experiment cannot yet be reconstructed exactly? Treat them as reference records, not as expected outputs that automatically validate a new run. The generated package stores the reported values separately from locally computed predictions and metrics, so a later experiment can be compared with the paper without confusing the two.

What the paper reports

The paper identifies its strongest reported configuration as LSTM-GWO (1-1-0-1). The four-part label is opaque: the supplied paper context does not define whether its fields represent layer counts, dropout settings, dense layers, or another encoding. The label must therefore remain a benchmark identifier rather than being translated into ModelConfig values.

The reported LSTM-GWO metrics are:

• R²: 99.2427%

• RMSE: 339.3902

• MAPE: 1.1721%

• RMSPE: 1.6221%

• PBIAS: 0.0523

• Willmott Index: 0.9981

• NSE: 0.9924

The paper also reports the following best labels and associated values:

• GRU-GWO `(2-1-1-1)`: R² 99.2322%, RMSE 341.7225, MAPE 1.1821%, and PBIAS −0.1357.

• SRU-GWO `(2-2-0-0)`: R² 99.2009%, RMSE 348.6384, and MAPE 1.2080%.

The remaining GRU and SRU metric associations are not clear in the supplied extraction. They are intentionally left unavailable rather than inferred from nearby text or reconstructed tables. Likewise, the configuration labels remain undecoded.

These values also use mixed display conventions. R² is shown by the paper as a percentage, whereas Willmott Index and NSE are shown on a unit-interval scale. A local metrics implementation may store R² internally as 0.992427 and display it as 99.2427%, but that conversion must be recorded explicitly before comparing values.

How the generated code preserves the evidence

PaperBenchmark is an immutable record containing the recurrent family, optimizer, opaque configuration label, metric values, units, and provenance. Its metric schema always contains the seven named metrics, but unavailable values are represented by None. The following excerpt is copied from reported_benchmarks in src/compositional_rnn_stock/evaluation/benchmarks.py:

def reported_benchmarks() -> tuple[PaperBenchmark, ...]:
    """Return the three benchmark records supplied in the paper context.

    Missing GRU and SRU associations remain ``None`` rather than being inferred from
    other reported values. R² is stored in the paper's percentage display convention.
    """
    return (
        PaperBenchmark(
            family="LSTM",
            optimizer="GWO",
            configuration_label="1-1-0-1",
            metrics=_benchmark_metrics(
                r_squared=99.2427,
                rmse=339.3902,
                mape=1.1721,
                rmspe=1.6221,
                pbias=0.0523,
                willmott_index=0.9981,
                nash_sutcliffe_efficiency=0.9924,
            ),
            metric_units=dict(_METRIC_UNITS),
        ),

The function returns records marked as supplied paper targets. It does not train a model, generate predictions, or assert that any local candidate achieved these values. The LSTM record is complete, while the later GRU and SRU records retain None for unavailable associations.

The generated contract test makes this distinction explicit when it retrieves the records:

benchmarks = reported_benchmarks()

assert [(item.family, item.optimizer, item.configuration_label) for item in benchmarks] == [
    ("LSTM", "GWO", "1-1-0-1"),
    ("GRU", "GWO", "2-1-1-1"),
    ("SRU", "GWO", "2-2-0-0"),
]

This is a planned test file, not evidence of an executed check. Under the authoritative run policy, code execution and verification were disabled.

Comparing a future local report

The compare_to_report function accepts a local metric mapping and one PaperBenchmark. It produces rows only when both sides have a value. Differences are expressed as local minus reported in the benchmark’s displayed units. Thus a missing GRU RMSPE value does not become a guessed comparison row, and a local unit-interval R² can be converted to the paper’s percentage convention when the local report has not declared another unit.

A comparison is meaningful only when the upstream experiment is also aligned. The provenance should identify the target column, forecast horizon, scaling range and fitting policy, window length, chronological split, recurrent-family configuration, seed, training settings, search settings, evaluation scale, and metric conventions. Without those fields, a numerical difference cannot reveal whether the models or datasets were actually comparable.

The evaluation pipeline places these comparisons alongside a local EvaluationRun, whose predictions, aligned targets, residuals, metric report, and provenance are kept together. This connects benchmark comparison to the feature-wise compositional model: the local prediction must first come from the five independent OHLCV streams, concatenation fusion, and the selected recurrent and dense head. A benchmark difference cannot repair an architecture or target choice that the paper leaves unspecified.

Reported evidence versus local conclusions

The paper states that GWO outperformed Random Search across the evaluated LSTM, GRU, and SRU families. That statement is reported evidence from the paper. It is not a result established by this unexecuted scaffold. A local RS-versus-GWO comparison would require the same preprocessing, validation objective, candidate space, and declared search budget for both methods, followed by actual execution and evaluation.

The safe interpretation is therefore:

1. reported_benchmarks() preserves the supplied reference records.

2. compare_to_report provides a labeled comparison mechanism for future local outputs.

3. Missing metrics and opaque configuration labels remain unresolved.

4. No local metric is claimed to match the paper.

The benchmark records are useful precisely because they retain their provenance and limitations. They provide targets for a future, better-specified reproduction rather than proof that the current configurable implementation reproduces the published experiment.

Offline synthetic demonstration and artifact provenance

How can you learn the repository’s data and model interfaces without downloading market data? Use the offline demonstration. It creates clearly labeled synthetic OHLCV rows, applies the same five-channel scaling and windowing contracts, builds a configurable compositional model, and inspects the expected tensor shapes. This is a plumbing demonstration only: the synthetic series is not Hang Seng Index data and cannot validate forecasting performance.

Why use an offline demonstration?

The paper describes daily Hang Seng Index history obtained from Yahoo Finance, but the supplied context does not identify a unique ticker, date range, adjustment policy, or download configuration. Yahoo Finance data can also change as providers revise historical records. An offline example avoids those dependencies while making every local choice visible.

The generated example uses a deterministic pseudo-random generator when given a selected seed. It creates five columns in the canonical order Open, High, Low, Close, Volume. The values are synthetic prices and volumes; they are not intended to reproduce the empirical distribution of the HSI.

The example’s generator is defined in examples/offline_synthetic_demo.py:

def make_synthetic_ohlcv(num_days: int, seed: int) -> pd.DataFrame:
    """Create deterministic, clearly labeled synthetic OHLCV observations.

    The returned table has shape ``(num_days, 5)`` and canonical column order:
    Open, High, Low, Close, Volume. Values are generated only for this
    demonstration; they are not intended to model the empirical HSI series.
    """
    if isinstance(num_days, bool) or not isinstance(num_days, int) or num_days <= 0:
        raise ValueError("num_days must be a positive integer")
    if isinstance(seed, bool) or not isinstance(seed, int) or seed < 0:
        raise ValueError("seed must be a non-negative integer")

    rng = np.random.default_rng(seed)
    daily_returns = rng.normal(loc=0.0004, scale=0.012, size=num_days)
    close = 18_000.0 * np.exp(np.cumsum(daily_returns))
    open_noise = rng.normal(loc=0.0, scale=35.0, size=num_days)
    open_price = close + open_noise
    intraday_spread = np.abs(rng.normal(loc=55.0, scale=18.0, size=num_days))
    high = np.maximum(open_price, close) + intraday_spread
    low = np.minimum(open_price, close) - intraday_spread
    volume = np.maximum(
        1.0,
        1_000_000.0 + rng.normal(loc=0.0, scale=120_000.0, size=num_days),
    )

    values = np.column_stack((open_price, high, low, close, volume))
    return pd.DataFrame(
        values,
        index=pd.date_range("2020-01-01", periods=num_days, freq="D"),
        columns=list(CANONICAL_OHLCV_COLUMNS),
    )

The function accepts a positive number of rows and a non-negative integer seed. Its output is a table with shape num_days × 5. The validation errors are useful failure cases: a boolean is not accepted as an integer, and invalid row counts or seeds are rejected rather than silently corrected.

Scaling an earlier chronological portion

The paper states the intended feature range [0, 0.95], but it does not specify exactly which observations fit the scaling statistics. The demonstration chooses an earlier chronological portion for fitting. This is a leakage-prevention decision made by the implementation, not a recovered paper detail. The remaining observations are transformed with the already fitted scaler.

The relevant portion of main is:

data_config = DataConfig(
    target_column="Close",
    forecast_horizon=1,
    window_length=20,
    split_ratio=0.8,
    scaling_range=(0.0, 0.95),
)
raw_frame = make_synthetic_ohlcv(num_days=96, seed=7)
raw_values = raw_frame.loc[:, list(CANONICAL_OHLCV_COLUMNS)].to_numpy(
    dtype=np.float64
)

# Fit on an earlier chronological portion to demonstrate a leakage-avoiding
# choice. The paper states the [0, 0.95] range but does not specify fit scope.
scaler_split = int(raw_values.shape[0] * data_config.split_ratio)
scaler = FeaturewiseMinMaxScaler(feature_range=data_config.scaling_range)
scaler.fit(raw_values[:scaler_split])
scaled_values = scaler.transform(raw_values)

Here, target_column='Close', forecast_horizon=1, and window_length=20 are tutorial configuration choices. The paper does not identify the target column or forecast horizon; it only describes 20-day and 40-day input windows. Similarly, the demonstration’s 96 synthetic rows and seed 7 are not paper data.

FeaturewiseMinMaxScaler requires a final feature axis of length five. It stores separate scaling metadata for each OHLCV channel, so Volume does not determine the scaling of the price columns. Its inverse_transform method is available when later evaluation should return predictions to original price units. The example does not calculate or present forecast metrics.

Building windows and inspecting model shapes

After scaling, the example selects the configured target column and constructs overlapping supervised windows:

target_index = list(CANONICAL_OHLCV_COLUMNS).index(data_config.target_column)
windowed = make_sliding_windows(
    scaled_values,
    target_index=target_index,
    window_length=data_config.window_length,
    forecast_horizon=data_config.forecast_horizon,
    timestamps=raw_frame.index.to_numpy(),
)

# The model follows the paper's feature-wise structure: five independent
# recurrent streams, concatenation fusion, and a one-value prediction head.
model_config = ModelConfig(
    recurrent_family="LSTM",
    stream_layers=1,
    stream_units=16,
    stream_dropout=0.0,
    fusion_dropout=0.0,
    use_post_fusion_recurrent=False,
    dense_layers=(16,),
    output_dimension=1,
)
model = build_compositional_model(model_config)
model.eval()

batch_size = min(4, windowed.sample_count)
batch_inputs = torch.as_tensor(windowed.inputs[:batch_size], dtype=torch.float32)
with torch.no_grad():
    predictions = model(batch_inputs)

make_sliding_windows returns inputs with shape samples × window_length × 5 and scalar targets with shape samples. With this tutorial configuration, each input contains 20 time steps and five channels. The target is the selected Close value one step after the input window according to the implementation’s explicit alignment convention.

build_compositional_model then constructs the feature-wise recurrent model. Each batch input has shape batch × 20 × 5; the model separates the five channels, encodes them independently, concatenates their representations, and returns a prediction with shape batch × output_dimension. The model configuration shown here is an educational choice. It does not decode the paper’s opaque benchmark label LSTM-GWO (1-1-0-1), nor does it claim to recover the paper’s exact layer widths or training recipe.

The source example prints these contracts:

print("Synthetic demonstration only; no paper result is being reproduced.")
print(f"raw OHLCV shape: {raw_values.shape}")
print(f"scaled OHLCV range: {data_config.scaling_range}")
print(
    "window input shape (samples, time_steps, features): "
    f"{windowed.inputs.shape}"
)
print(f"window target shape: {windowed.targets.shape}")
print(
    "model batch input shape (batch, time_steps, features): "
    f"{tuple(batch_inputs.shape)}"
)
print(f"model prediction shape (batch, output_dimension): {tuple(predictions.shape)}")

These statements document intended dimensions; this run did not execute the example. In particular, no output numbers, predictions, or paper-comparison result should be inferred from the excerpt.

Recording provenance and arrays locally

A demonstration becomes more useful when its assumptions travel with its outputs. Artifact provenance should identify at least the data source, cleaning policy, scaling range and fit scope, window length, target column, forecast horizon, model family, architecture settings, seed, and evaluation conventions. For real HSI work, the archived raw file and the Yahoo Finance query parameters should also be retained.

The generated artifact module provides local persistence through save_json_record, save_array, and load_json_record. The README illustrates the JSON-record call as follows:

save_json_record(report, Path('artifacts/report.json'))

A JSON record is appropriate for serializable configuration, acquisition metadata, histories, benchmark labels, and metric reports. Numeric predictions and residuals should be stored with save_array, which preserves their numeric shape and dtype. load_json_record reads a previously saved top-level JSON object. The artifact implementation uses local filesystem operations and rejects implicit overwrites, helping prevent an old experiment from being silently replaced.

For example, a future evaluation record could contain a local run identifier, window_length, target_column, forecast_horizon, scaler settings, model configuration, seed, metric conventions, and a separate reference to prediction and residual arrays. It should distinguish computed local values from the paper’s reported benchmark records. The generated interfaces provide the storage operations, while the exact record contents remain an experiment-provenance responsibility.

What this demonstration does not establish

The synthetic path confirms the intended software contracts conceptually: five ordered channels can be scaled, transformed into windows, and supplied to a feature-wise compositional model. It does not establish that the model forecasts the HSI accurately, that the chosen target and horizon match the paper, or that the reported LSTM-GWO, GRU-GWO, or SRU-GWO values can be reproduced.

No example execution, artifact creation, test execution, or verification occurred under the run policy. A stronger reproduction would archive the original Yahoo Finance data query and downloaded rows, supply the missing architecture and optimization details, run the configured experiment, and then evaluate predictions under explicitly documented metric conventions. Until then, the offline example is a reproducibility aid—not evidence of paper-level performance.

Verification status, limitations, and responsible reproduction claims

How can a carefully organized implementation be useful without overstating what it proves? The answer is to separate invariants, planned checks, and verified results. The generated package records the paper’s intended data flow, model structure, search interfaces, evaluation conventions, and provenance. However, this run did not execute code or perform verification. Its correct status is therefore verification_skipped: a configurable reproduction scaffold with explicit gaps, not a validated reproduction of the published experiment.

What the implementation is designed to preserve

Several structural requirements are clear enough to review independently of the paper’s missing experimental details:

• The input feature order is exactly Open, High, Low, Close, Volume.

• Model inputs use batch × time × 5 ordering.

• Preprocessing constructs chronological windows and keeps training samples before later samples.

• The scaler targets the stated interval [0, 0.95]. Fitting it on earlier training observations only is a leakage-prevention implementation choice; the paper does not specify its fitting partition.

• The model creates exactly five independent feature streams.

• Stream representations are fused by concatenation, rather than by early summation or mixing.

• Dropout is intended for training mode and should be inactive during evaluation.

• Predictions and targets must remain sample-aligned before metric calculation.

• Search candidates must decode to valid typed values for layer counts, units, dropout rates, learning rates, batch sizes, and epochs.

• RS and GWO should use the same candidate objective, preprocessing, and validation protocol when compared locally.

• Benchmark labels such as LSTM-GWO (1-1-0-1) remain opaque identifiers and are not translated into architecture settings.

These are implementation contracts and interpretation safeguards. They are not evidence that a particular model reached the paper’s reported metrics.

Planned contract tests are not completed checks

The repository contains generated test files intended to make these invariants reviewable later. For data handling, test_scaler_range_and_inverse_round_trip, test_window_target_alignment, and test_chronological_split_has_no_reordering address scaling behavior, window-to-target indexing, and chronological partitioning. The model tests include test_model_accepts_five_feature_input, test_fusion_dimension_is_additive, test_recurrent_family_dispatch_is_consistent, and test_dropout_differs_only_in_training_mode.

The remaining planned tests cover local metric conventions, candidate encoding, search bounds, and provenance. Examples include test_rmse_zero_for_equal_arrays, test_percentage_metric_zero_policy, test_candidate_codec_validates_discrete_parameters, test_gwo_positions_stay_in_bounds_after_decoding, test_reported_benchmarks_preserve_supplied_values, and test_incomplete_metrics_remain_missing.

These names describe intended checks, not completed checks. Under the authoritative run policy, test generation and execution were disabled. No test result, passing assertion, successful import, or model output should be inferred from the presence of these files.

Static and semantic verification status

Static verification and semantic verification address different questions. Static review would inspect syntax, imports, public interfaces, tensor-shape contracts, and configuration propagation without running the experiment. Semantic review would assess whether the implementation’s behavior matches the intended method, including feature independence, concatenation, dropout mode changes, search-objective consistency, and metric conventions.

Both forms of verification were skipped here. Local static verification reports the status verification_skipped, and semantic code verification was also skipped. Code execution, test execution, tutorial-section verification, and final quality review were disabled by policy. Consequently, this section must not claim that the generated code is correct, that the tests pass, or that any local result matches the paper.

Why exact reproduction remains underdetermined

The supplied paper context does not uniquely determine several decisions that materially affect results:

• The Yahoo Finance ticker or symbol, date range, download options, adjustment policy, missing-value handling, and duplicate-row handling.

• The forecast target and horizon. The paper names “stock price” and lists OHLCV inputs, but does not identify whether the target is Close, adjusted Close, another price, or a multivariate output.

• The exact interpretation of the 20-day and 40-day windows beyond treating them as input lengths.

• The normalization formula, fitting partition, clipping behavior, and inverse-transformation procedure.

• Recurrent layer counts, widths, activations, hidden-state extraction, dense-layer structure, and exact placement of both dropout stages.

• The SRU variant and Python implementation or API.

• The training loss, gradient optimizer, initialization, learning-rate schedule, seed, shuffling policy, stopping rule, and checkpoint-selection rule.

• RS search domains, distributions, trial budget, and validation protocol.

• GWO population size, initialization, bounds, iteration count, update schedule, objective, and encoding of discrete parameters.

• Metric formulas, percentage conventions, zero-denominator handling, bias signs, and the scale used for evaluation.

The paper’s reported total of 54 configurations does not resolve these omissions. Nor can the opaque four-part labels in the LSTM-GWO, GRU-GWO, and SRU-GWO records be safely decoded from the supplied evidence.

A responsible review checklist

Before treating a future local run as evidence, review the following items and record each one in the experiment provenance:

1. Data shape: confirm that the prepared inputs use batch × time × 5 and the canonical OHLCV order.

2. Chronology: confirm that the split and window-target alignment preserve temporal order and do not use future observations during fitting.

3. Scaling: record the [0, 0.95] range, the fitting partition, constant-feature policy, and any inverse transformation.

4. Target semantics: record the target column and forecast horizon instead of assuming that the paper specifies them.

5. Model structure: record the five independent encoders, recurrent family, representation choice, dropout placement, fusion operation, post-fusion layers, and output dimension.

6. Training: record the loss, gradient optimizer, learning rate, batch size, epochs, seed, validation protocol, and checkpoint policy.

7. Search: record the typed domains, objective direction, trial or wolf budget, bounds, discrete decoding, and failure handling.

8. Evaluation: record whether metrics use normalized or original units, every metric convention, percentage display rules, and zero handling.

9. Benchmarks: preserve reported values separately from local values, keep missing GRU and SRU associations missing, and leave configuration labels opaque.

10. Verification: record whether code and tests were actually executed and reviewed; do not substitute planned test names for evidence.

This checklist is a review target, not a statement that the items were checked in this run.

What a stronger reproduction would require

A stronger claim would require the original data query or an archived copy of the downloaded OHLCV data, the complete architecture and configuration tables, the exact target and forecast horizon, and the full training and validation protocol. It would also require the RS and GWO search spaces and budgets, the selected SRU implementation, canonical metric definitions, and the original prediction or error outputs needed to compare visual analyses.

After those details were supplied, the implementation could use its provenance records to replace local choices without changing the overall public interfaces. The resulting experiment would still need to be executed, checked, and compared under a documented protocol before it could be described as a reproduction.

Attention mechanisms, hybrid architectures, and technical indicators are mentioned as future work in the paper. They are outside this reproduction workflow and should not be added to close the evidence gaps.

The appropriate conclusion at this stage is deliberately modest: the package makes the paper-to-code assumptions visible and organizes the intended pipeline, but neither the generated scaffold nor the supplied benchmark constants establish successful or exact reproduction.

Use the button or URL below to download the source code.

Read more

The paper develops and evaluates separate GRU recurrent neural-network models for forecasting weekly adjusted closing prices of Apple Inc. (AAPL) and Microsoft Corporation (MSFT). Each series contains 263 observations from Yahoo Finance. The workflow consists of exploratory analysis, missing-value and outlier checks, decomposition, ADF stationarity testing, autocorrelation analysis, three-period sliding-window transformation, training-only MinMax normalization, chronological splitting, Hyperband-based hyperparameter tuning, GRU training with Adam, early stopping and checkpointing, inverse transformation, and validation forecasting evaluation with MSE, MAE, MAPE, and forecast accuracy. The selected AAPL model has three GRU layers with 240, 16, and 16 units; the selected MSFT model has two GRU layers with 224 and 48 units. Reported MAPE values are 3.37% for AAPL and 2.55% for MSFT.

Implementation Assumptions

• Use appendix records as the primary local data source when the exact Yahoo Finance retrieval and weekly aggregation procedure cannot be established.

• Represent each raw asset series as 263 chronological observations and each windowed dataset as 260 samples with shape (260, 3, 1).

• Use chronological sample partitions of 156 training, 52 testing, and 52 validation samples.

• Keep AAPL and MSFT data, scalers, models, histories, checkpoints, and metrics separate.

• Treat exploratory decomposition, ADF testing, autocorrelation, histograms, and boxplots as diagnostics only; they do not modify forecasting inputs.

• Use one explicit, documented MinMaxScaler convention for reproducibility, while exposing the convention because the paper does not specify whether raw training prices or flattened window values were used.

• Use the Table 1 configurations as the final reproduction targets: AAPL widths (240, 16, 16), MSFT widths (224, 48), zero dropout, learning rate 0.001, MSE, batch size 8, maximum 100 epochs, and patience 7.

• Keep the testing subset distinct from validation until an explicit evaluation-subset option is selected.

• Do not claim exact paper metric reproduction because source versions, tuner space, random seed, callback details, and evaluation protocol are incomplete.

• No code execution, tests, static verification, semantic verification, tutorial verification, or final review is claimed under the supplied run policy.

Scope, Paper Facts, and Reproducibility Limits

What exactly is being reproduced? The paper forecasts the next weekly adjusted closing price for two stocks: Apple, identified as AAPL, and Microsoft, identified as MSFT. Each stock is treated as its own univariate time series. In other words, the AAPL model does not receive MSFT prices, and neither model uses technical indicators, sentiment, volume, macroeconomic variables, or other explanatory features.

The model sees a short ordered history of adjusted prices and produces one next-price forecast. Adjusted closing prices, rather than ordinary closing prices, are the paper's retained price field because they account for effects such as dividends and stock splits. This distinction matters: substituting ordinary closing prices would change the input data and would no longer be the same experiment.

Three evidence categories

It helps to separate what is known from the paper, what follows mathematically from the data dimensions, and what the implementation must choose.

Paper facts include 263 weekly adjusted closing-price observations for each asset, separate GRU models, a three-period sliding window, chronological partitioning, training-only MinMax normalization, Hyperband-based tuning, Adam optimization, early stopping, checkpointing, and evaluation with MSE, MAE, MAPE, and forecast accuracy. The final reported AAPL architecture has GRU widths of 240, 16, and 16. The final MSFT architecture has widths of 224 and 48. Both configurations report zero dropout, a learning rate of 0.001, MSE loss, batch size 8, a maximum of 100 epochs, and patience 7.

Derived dimensional correction. Windowing creates supervised examples before splitting. Three preceding prices are needed to predict one following price. Therefore, 263 raw observations produce 260 supervised samples. The paper-consistent chronological partition is consequently 156 training samples, 52 testing samples, and 52 validation samples. Each recurrent input has shape (n, 3, 1): n examples, three time steps, and one price feature. Each target is standardized in the generated API as shape (n, 1), one scalar target per example.

Implementation decisions. Several details are absent from the supplied paper text. The generated project uses local appendix-style CSV files instead of inventing a Yahoo Finance download and weekly-resampling procedure. It exposes the choice between the testing and validation subsets rather than silently combining them. It also records a scaler convention, diagnostic parameters, Keras layer details, callback behavior, and tuner search ranges as implementation choices rather than paper facts.

No canonical equation records were supplied. This section therefore explains the tensor contracts, algorithm responsibilities, and data flow in prose and code. It does not reconstruct GRU gate equations, scaling formulas, loss formulas, or metric formulas.

The end-to-end reproduction boundary

The package is organized as a sequence of isolated per-asset stages:

1. Load and validate one local adjusted-price series.

2. Run exploratory diagnostics without modifying the forecasting data.

3. Convert 263 prices into 260 three-step supervised windows.

4. Split those windows chronologically into 156, 52, and 52 samples.

5. Fit normalization using training data only and transform the later subsets.

6. Build the selected AAPL or MSFT GRU architecture.

7. Optionally run asset-specific Hyperband tuning, or instantiate the reported final configuration directly.

8. Train with Adam, validation-loss monitoring, early stopping, and checkpointing.

9. Select either the testing or validation subset explicitly.

10. Inverse-transform forecasts and targets, calculate metrics, and compare them with the paper's Table 3 references.

The central orchestration function is run_asset_pipeline. Its inputs are an asset identifier, a local data directory, an output directory, an evaluation_subset value of either "validation" or "test", and a Boolean tune flag. Its output is an AssetRunResult containing the raw PriceSeries, diagnostic records, the WindowedDataset, scaled DatasetSplits, scaler, model, training result, and forecast result. This retained-artifact design makes the intermediate boundaries inspectable instead of reducing the workflow to a single opaque training call.

The pipeline rejects unsupported assets, missing data directories, invalid evaluation-subset names, and invalid tuning flags. The lower-level validation stage is responsible for checking the expected 263 rows, chronology, missingness, finite numeric values, and positive adjusted prices. A failure at these boundaries should stop the reproduction rather than silently repair or reinterpret the input.

Here is the public configuration interface that captures the paper's final model settings and reference values:

from stock_forecasting.config import (
    get_default_split_config,
    get_final_model_config,
    get_reference_metrics,
)

config = get_final_model_config("AAPL")
split = get_default_split_config()
reference = get_reference_metrics("AAPL")

get_final_model_config returns an immutable AssetModelConfig. For AAPL, its units are (240, 16, 16); for MSFT, they are (224, 48). It also carries the reported dropout, learning rate, loss name, batch size, epoch limit, and patience. get_default_split_config returns the derived 156/52/52 split and rejects configurations that do not total 260 samples. get_reference_metrics returns Table 3 values for comparison only; it does not calculate a forecast and does not claim that those values were achieved.

Worked example: trace one AAPL run

Assume that data/raw/AAPL.csv contains the normalized local input with columns date and adjusted_close, one asset per file, and 263 chronological rows. A complete AAPL pipeline call is:

from pathlib import Path

from stock_forecasting.pipeline import run_asset_pipeline

result = run_asset_pipeline(
    asset="AAPL",
    data_dir=Path("data/raw"),
    output_dir=Path("artifacts"),
    evaluation_subset="validation",
    tune=False,
)

This call expresses several important decisions. asset="AAPL" keeps the model univariate and isolated. evaluation_subset="validation" follows the paper's statement that validation forecasting was reported, but it does not consume or merge the separate testing subset. tune=False selects the reported final Table 1 configuration directly rather than pretending that the incomplete original Hyperband metadata can be recovered.

The resulting AssetRunResult retains the main stages:

raw_prices = result.series.prices
window_inputs = result.windowed.inputs
scaled_training_inputs = result.splits.train_inputs
trained_model = result.model
forecast = result.forecast
comparison = result.diagnostics["reference_comparison"]

The expected raw price array has length 263. The window input array has shape (260, 3, 1). The scaled training input array has 156 samples, while the testing and validation arrays each have 52. forecast contains date-aligned actuals, predictions, and metrics after inverse transformation. comparison contains computed, reference, and difference fields for the four reported metrics; it is a descriptive comparison, not a verification result.

The lower-level pipeline ordering is visible in the generated implementation. This excerpt shows the key dimensional transitions and the explicit scaler choice:

    # load_validate_price_series: local appendix records are the offline input.
    series = load_asset_records(data_dir, canonical_asset)
    quality = validate_price_series(series)
    diagnostics = _run_diagnostics(series)
    diagnostics["quality"] = quality

    # create_windowed_supervised_dataset: 263 raw observations become 260 samples.
    windowed = create_windowed_supervised_dataset(series, window_length=3)

    # chronological_split_and_minmax_scale: 156/52/52 is applied after windowing.
    split_config: SplitConfig = get_default_split_config()
    raw_splits = split_windowed_dataset(windowed, split_config)
    scaled_splits, scaler = _scale_splits(
        raw_splits,
        convention=ScalerConvention.RAW_TRAINING_PRICES,
    )

Notice that diagnostics are computed before modeling but are not passed into the GRU as features. Also notice that the split occurs after windowing and that the scaler is fitted through _scale_splits using training data. The generated pipeline records ScalerConvention.RAW_TRAINING_PRICES as its selected decision, while the paper itself leaves open whether the original implementation fitted on raw training prices or on flattened training windows and targets.

After preprocessing, the pipeline chooses the model configuration, builds the model, trains it, and evaluates the explicitly selected subset:

    model_config = _select_model_config(
        canonical_asset,
        scaled_splits,
        tune=tune,
        output_dir=output_dir,
    )
    diagnostics["model_config"] = model_config
    diagnostics["tuning_requested"] = tune

    model = build_asset_gru_model(model_config, input_shape=(3, 1))
    checkpoint_path = output_dir / canonical_asset / "best.weights.h5"
    checkpoint_path.parent.mkdir(parents=True, exist_ok=True)
    training = train_gru_with_callbacks(
        model=model,
        splits=scaled_splits,
        config=model_config,
        checkpoint_path=checkpoint_path,
    )

    evaluation_inputs, evaluation_targets, evaluation_dates = select_evaluation_subset(
        scaled_splits,
        normalized_subset,
    )
    forecast = evaluate_forecasts(
        model=training.model,
        inputs=evaluation_inputs,
        targets=evaluation_targets,
        dates=evaluation_dates,
        scaler=scaler,
    )

The model consumes scaled inputs with shape (batch, 3, 1) and returns one normalized scalar per sample. evaluate_forecasts converts predictions and targets back to price units using the training-fitted scaler before producing the reported metrics. The evaluation_dates are the dates of the target observations, so plotted forecasts correspond to the prices they are intended to predict.

For convenience, run_both_assets repeats this workflow for AAPL and MSFT independently. It returns a dictionary keyed by ticker and uses validation evaluation by default. Calling run_asset_pipeline directly is the appropriate way to select the testing subset for a particular asset.

Command-line entry point

The generated script exposes the unresolved evaluation choice and optional tuning choice explicitly:

python scripts/run_reproduction.py \
  --data-dir data/raw \
  --output-dir artifacts \
  --evaluation-subset validation

To keep the distinct testing subset as the held-out evaluation target, use:

python scripts/run_reproduction.py \
  --data-dir data/raw \
  --output-dir artifacts \
  --evaluation-subset test

Adding --tune requests the optional asset-specific Hyperband path. The non-tuning path is the clearer reproduction target when the goal is to instantiate the paper's reported Table 1 architectures. The exact original tuner search space, random seed, number of Hyperband iterations, and stopping criteria are unavailable, so a locally selected configuration cannot be described as an exact recovery of the paper's search.

What the reference metrics do—and do not—mean

The paper reports the following Table 3 reference values: AAPL MSE 55.80, MAE 6.11, MAPE 3.37%, and forecast accuracy 96.63%; MSFT MSE 144.44, MAE 9.41, MAPE 2.55%, and forecast accuracy 97.45%. The generated compare_to_reference function reports differences between a local run and these values. It does not turn the comparison into a pass/fail claim unless a caller supplies a tolerance, and even then the result describes the selected run rather than proving scientific equivalence.

These values may differ because the exact source-data version, weekly aggregation procedure, scaler fitting convention, random initialization, unspecified Keras defaults, callback behavior, Hyperband search space, and evaluation-subset protocol are incomplete. The lower MSFT MAPE reference also illustrates why relative and absolute errors should not be conflated: a higher price level can produce larger absolute errors while still producing a lower percentage error.

Reproducibility and verification limits

This is a reproduction scaffold, not a claim that the original experiment has been exactly rerun. The supplied run policy disabled code execution, test generation, local static verification, semantic code verification, tutorial verification, and final quality review. The local verification record therefore reports a skipped status, not a passing result, and semantic code verification was also skipped. No commands, training runs, tests, generated metrics, or reference matches are claimed here.

The paper itself also limits interpretation. It uses only two technology stocks and 263 weekly observations per asset. A price-history-only GRU may smooth abrupt movements and can be affected by distribution shift. The paper does not supply rolling-origin evaluation, confidence intervals, statistical significance testing, or broad benchmark comparisons. Consequently, forecasts should be treated as decision-support information for analysis and risk management, not as an autonomous trading system.

The practical scope is therefore precise: preserve the two isolated adjusted-price series, reproduce the 260-sample dimensional interpretation, keep the 156/52/52 chronology and training-only scaling boundary, expose unresolved protocol choices, and label Table 1 and Table 3 as paper references. Any numerical conclusion should wait until execution and the relevant verification stages are actually enabled.

Local Adjusted-Price Inputs and Diagnostic Analysis

Before training a forecasting model, ask a simpler question: do we have the right observations in the right order? For this reproduction, each asset is an independent weekly sequence of adjusted closing prices. The diagnostic stage checks that sequence, summarizes it, and records useful time-series behavior without changing the data later supplied to the GRU.

The paper uses 263 observations for each of AAPL and MSFT. Its retained price field is the adjusted closing price, not the ordinary close. The implementation keeps dates alongside prices so that later windows and forecasts can remain aligned with their target dates.

What is fixed and what is chosen

The paper establishes several input facts: the two assets are modeled separately, each series contains 263 observations, and no missing values were reported. It also describes an upward trend, non-stationarity, strong autocorrelation, and no visible outliers.

Other details are not fully specified. The exact Yahoo Finance retrieval query, weekly aggregation rule, calendar handling, and source-data version are unavailable. Similarly, the paper does not state the decomposition method or period, the ADF regression and lag settings, or the autocorrelation lag range and interval construction. The generated code makes these choices explicit rather than presenting them as recovered paper settings.

Most importantly, diagnostics are descriptive. They do not remove outliers, replace values, difference the series for forecasting, or add decomposition components as GRU features.

Load local appendix-style records

The reproduction uses local CSV files instead of silently downloading from Yahoo Finance. This is an implementation decision that makes the input inspectable and avoids inventing an unspecified retrieval and resampling procedure. The expected layout is:

data/raw/
├── AAPL.csv
└── MSFT.csv

The loader accepts date and adjusted-price column aliases, but the normalization script writes the stable date and adjusted_close schema. A focused excerpt from src/stock_forecasting/io/appendix_loader.py shows the public loading contract:

from pathlib import Path

from stock_forecasting.data_types import PriceSeries
from stock_forecasting.exceptions import DataValidationError


def load_appendix_csv(path: Path, asset: str) -> PriceSeries:
    """Load one local appendix-style CSV as a validated price series.

    The local appendix file is deliberately used instead of downloading from Yahoo
    Finance: the paper identifies Yahoo Finance as provenance but does not specify
    the exact retrieval, calendar, or weekly aggregation procedure.  The loader
    therefore accepts already prepared records and preserves the adjusted-close
    field as the sole forecasting variable.
    """

load_appendix_csv reads one file and returns a PriceSeries containing dates, prices, and the canonical asset name. It checks optional asset labels so a file intended for AAPL cannot quietly contain MSFT rows. It sorts records chronologically and converts prices to floating-point values. It does not remove rows identified visually as unusual.

For the standard directory layout, use load_asset_records:

from pathlib import Path

from stock_forecasting.io.appendix_loader import load_asset_records

series = load_asset_records(Path("data/raw"), "AAPL")
print(series.asset)
print(series.dates.shape)
print(series.prices.shape)

The generated UnavailableYahooSource is deliberately not a network client. Its responsibility is to make an unsupported source choice fail clearly rather than download a different dataset:

class UnavailableYahooSource:
    """Explicit offline adapter for the paper's Yahoo Finance provenance."""

    def get_series(self, asset: str) -> PriceSeries:
        """Reject external retrieval and direct the caller to local data."""
        if not isinstance(asset, str) or asset not in _ALLOWED_ASSETS:
            raise DataValidationError(
                "asset must be either 'AAPL' or 'MSFT'; "
                f"received {asset!r}"
            )
        raise DataValidationError(
            f"No offline Yahoo Finance adapter is configured for {asset}. "
            "Provide the paper's local appendix-style records and load them "
            "with load_appendix_csv or load_asset_records."
        )

This failure is intentional. The paper names Yahoo Finance as provenance, but the supplied specification does not identify the exact query or weekly aggregation operation.

Validate the 263-observation contract

Loading a CSV is not the same as validating an experiment input. validate_price_series checks the paper's structural assumptions and returns a quality summary. Its default expected count is 263, although callers can use another count for isolated fixtures.

from stock_forecasting.data_validation import validate_price_series

quality = validate_price_series(series)
print(quality)

The validation boundary requires all of the following:

• dates and prices each contain 263 entries;

• dates are present, unique, and strictly increasing;

• prices are one-dimensional, finite, and strictly positive;

• the retained field is adjusted closing price;

• no observations are removed as outliers.

The lower-level functions make the invariants visible. check_chronological_dates rejects missing, duplicate, or non-increasing dates. check_adjusted_prices rejects nonnumeric, non-finite, multidimensional, or non-positive values. summarize_quality reports counts, extrema, endpoints, chronology, and the outlier policy without modifying the series.

A concise end-to-end loading example is:

from pathlib import Path

from stock_forecasting.data_validation import validate_price_series
from stock_forecasting.io.appendix_loader import load_asset_records

DATA_DIR = Path("data/raw")
aapl = load_asset_records(DATA_DIR, "AAPL")
quality = validate_price_series(aapl)

print(aapl.asset)       # AAPL
print(aapl.dates.shape) # Expected shape: (263,)
print(aapl.prices.shape) # Expected shape: (263,)
print(quality)

The corresponding MSFT call should be made independently. Do not concatenate the two PriceSeries objects: asset isolation is a modeling requirement, not merely an organizational preference.

Prepare files without downloading data

When source records are supplied in an appendix-style directory, scripts/prepare_local_data.py validates and normalizes them into the expected raw-data layout. It requires local files and writes only date and adjusted_close.

python scripts/prepare_local_data.py \
    --input-dir appendix \
    --output-dir data/raw

prepare_asset_file loads one asset, invokes validate_price_series with expected_count=263, and writes a normalized CSV. It does not fill missing values, infer missing weeks, or correct outliers. Consequently, a short or malformed input fails before it can enter the modeling pipeline.

Descriptive statistics and reference comparisons

The paper's descriptive table reports count, mean, standard deviation, minimum, quartiles, median, and maximum for each 263-observation series. The generated compute_descriptive_statistics function returns those fields as a pandas Series:

from stock_forecasting.eda.descriptive import (
    compare_with_table_2,
    compute_descriptive_statistics,
)

stats = compute_descriptive_statistics(aapl)
print(stats)

# Generated statistic minus the paper's Table 2 reference value.
differences = compare_with_table_2(stats, aapl.asset)
print(differences)

The stored Table 2 values are paper references, not assertions that an unverified local file matches the paper's source exactly. Differences may reflect source-data versions, weekly aggregation, or numeric details. The helper reports signed differences and does not label them as a pass or fail.

Plot the distribution and chronology

The plotting functions in src/stock_forecasting/eda/plots.py return Matplotlib axes and read the source arrays without mutating them. The histogram describes the price distribution, the boxplot supports visual outlier screening, and the chronological plot reveals trend and fluctuations.

import matplotlib.pyplot as plt

from stock_forecasting.eda.plots import (
    plot_boxplot,
    plot_price_distribution,
    plot_price_series,
    save_figure,
)

figure, axes = plt.subplots(3, 1, figsize=(10, 12), constrained_layout=True)
plot_price_distribution(aapl, ax=axes[0])
plot_boxplot(aapl, ax=axes[1])
plot_price_series(aapl, ax=axes[2])
save_figure(figure, Path("artifacts/diagnostics/AAPL_overview.png"))
plt.close(figure)

A boxplot point is evidence for inspection only. The paper says that no visible outliers were found, but it gives no formal threshold or correction rule. Therefore, this implementation retains every validated observation even if a different local file produces a visually unusual point.

Decomposition: diagnostic components only

The paper asks for decomposition into trend, seasonal, and residual components, but it does not specify how. The generated wrapper requires a DecompositionConfig, making the method and period visible:

from stock_forecasting.eda.decomposition import (
    DecompositionConfig,
    decompose_price_series,
)

# Explicit implementation decision for this diagnostic run.
config = DecompositionConfig(method="additive", period=52)
components = decompose_price_series(aapl, config)

for name, component in components.items():
    print(name, component.shape, component.index.equals(aapl.dates))

Here, additive decomposition and a 52-observation period are implementation decisions for a weekly series. They are not parameters recovered from the paper. The returned dictionary contains trend, seasonal, and residual pandas Series, each aligned to the original 263 dates. Boundary component values may be missing because the decomposition needs neighboring observations; they are retained rather than imputed.

The components are not passed into the GRU. The forecasting input remains the single adjusted-price feature. This separation prevents an exploratory choice from silently changing the proposed model.

ADF stationarity diagnostic

The paper reports raw-price non-stationarity and gives approximate ADF p-values of 0.4594 for AAPL and 0.9229 for MSFT. Those values depend on test settings, and the supplied paper text does not specify the deterministic regression term or lag-selection policy. The generated run_adf_test therefore requires both:

from stock_forecasting.eda.stationarity import interpret_adf, run_adf_test

# Explicit implementation choices, not recovered paper settings.
aapl_adf = run_adf_test(aapl, regression="c", autolag="AIC")
print(aapl_adf)
print(interpret_adf(aapl_adf, alpha=0.05))

ADFReport contains the test statistic, p-value, and critical values. interpret_adf uses the paper's stated significance threshold of 0.05: a p-value below that threshold is reported as rejection of the unit-root null, while a p-value at or above it is reported as failure to reject.

Failure to reject is a diagnostic conclusion under one test specification, not proof about every future path. Likewise, the presence of non-stationarity does not mean that a GRU mathematically removes non-stationarity or remains valid under distribution shift. The ADF call does not difference or otherwise alter the price series used later.

Repeat the explicitly configured diagnostic for MSFT:

msft = load_asset_records(Path("data/raw"), "MSFT")
msft_adf = run_adf_test(msft, regression="c", autolag="AIC")
print(interpret_adf(msft_adf, alpha=0.05))

Autocorrelation diagnostic

Autocorrelation measures how values in the ordered series relate to earlier values at selected lags. The paper describes strong autocorrelation but does not specify the lag range or confidence-interval construction. The generated implementation therefore accepts an inclusive max_lag and returns only lag-aligned values:

from stock_forecasting.eda.autocorrelation import compute_autocorrelation

aapl_acf = compute_autocorrelation(aapl, max_lag=40)
print(aapl_acf.lags.shape)   # (41,)
print(aapl_acf.values.shape) # (41,)
print(aapl_acf.values[0])     # 1.0 by contract

Using lags 0 through 40 is an implementation decision. AutocorrelationReport requires contiguous lags beginning at zero and equal-length lags and values arrays. The report does not include confidence intervals because no interval method was supplied.

The diagnostic can be plotted without adding those values to the forecasting features:

import matplotlib.pyplot as plt

figure, axis = plt.subplots(figsize=(10, 4))
axis.stem(aapl_acf.lags, aapl_acf.values)
axis.set_title("AAPL adjusted-price autocorrelation")
axis.set_xlabel("Lag")
axis.set_ylabel("Autocorrelation")
save_figure(figure, Path("artifacts/diagnostics/AAPL_acf.png"))
plt.close(figure)

Run both assets from the command line

The diagnostic script applies the same explicit parameter choices independently to AAPL and MSFT and writes JSON reports, decomposition CSV files, and figures:

python scripts/run_diagnostics.py \
    --data-dir data/raw \
    --output-dir artifacts/diagnostics

Its defaults record an additive decomposition with period 52, an ADF constant term with AIC autolag, significance threshold 0.05, and autocorrelation through lag 40. Because several of these values are not paper facts, the script stores them as implementation metadata in each asset's diagnostic report.

What this stage establishes

After this stage, the intended state is a pair of independently validated PriceSeries objects and non-mutating diagnostic artifacts. The qualitative expectations from the paper are an upward trend, non-stationary raw prices, substantial autocorrelation, and no visible outliers. Those are interpretive reference points, not execution results claimed here.

No code execution, tests, static verification, semantic code verification, tutorial verification, or final quality review was performed under the supplied run policy. The next stage can therefore explain the planned transformation without implying that these diagnostics have already produced verified outputs.

The following section turns each unchanged 263-value sequence into supervised examples: three preceding adjusted prices become one input sequence, and the following price becomes its target. That transformation remains separate from the diagnostic plots and tests described here.

From a Price Sequence to Supervised Windows

How can a plain chronological price list become input that a recurrent neural network can learn from? The paper uses a three-period sliding window: take three consecutive adjusted prices, use them as one short sequence, and assign the following adjusted price as the target. Then move forward by one observation and repeat.

This transformation is performed separately for each asset and before any train, test, or validation split. AAPL and MSFT therefore produce independent WindowedDataset objects; their prices are never combined into one multivariate input.

The input and target contract

The generated code represents one raw asset as a PriceSeries. Its dates and prices fields are aligned one-to-one, with 263 chronological observations and positive adjusted prices. The create_windowed_supervised_dataset function then returns a WindowedDataset with three coordinated fields:

• inputs: recurrent input windows with shape (260, 3, 1);

• targets: one scalar next-price target per window, with shape (260, 1); and

• target_dates: the dates associated with those target prices, with length 260.

The three dimensions of inputs mean (samples, time steps, features). There are 260 samples, each sequence contains 3 time steps, and each time step contains 1 feature: the adjusted price. The singleton feature dimension is important because Keras recurrent layers expect a feature axis even for a univariate series.

The paper supplies the three-period window and 263 raw observations. The count of supervised samples follows from the implementation: the first three observations form an input but do not yet have a preceding window of their own, so each subsequent raw observation can serve as one target. Consequently, 263 raw observations produce 260 samples.

Building the windows

The public function accepts a validated PriceSeries and defaults to window_length=3:

from stock_forecasting.supervised.windowing import create_windowed_supervised_dataset

windowed = create_windowed_supervised_dataset(series, window_length=3)

print(windowed.inputs.shape)       # (260, 3, 1)
print(windowed.targets.shape)      # (260, 1)
print(windowed.target_dates.shape) # (260,)

Inside create_windowed_supervised_dataset, the implementation constructs each input from a consecutive slice and adds the feature axis. The following excerpt is the central transformation copied from src/stock_forecasting/supervised/windowing.py:

    # Three preceding adjusted prices predict the next adjusted price.
    inputs = np.stack(
        [prices[index : index + int(window_length)] for index in range(sample_count)],
        axis=0,
    )[..., np.newaxis]
    targets = prices[int(window_length) :].reshape(-1, 1)
    target_dates = dates[int(window_length) :]

For sample index 0, the input slice contains raw prices at positions 0, 1, and 2. The target slice begins at position 3, so its first value is the price immediately after that input. The [..., np.newaxis] operation changes a two-dimensional collection of windows into the rank-three recurrent shape (samples, 3, 1). The target reshape standardizes one scalar per sample as (samples, 1), matching the intended scalar model output.

A worked example with inspectable values

A monotonic synthetic series makes the boundary behavior easy to see without depending on external data. This is an illustration only, not a replacement for the paper's AAPL or MSFT records.

import numpy as np
import pandas as pd

from stock_forecasting.data_types import PriceSeries
from stock_forecasting.supervised.windowing import (
    create_windowed_supervised_dataset,
)

_RAW_COUNT = 263

raw_dates = pd.date_range("2020-01-03", periods=_RAW_COUNT, freq="W-FRI")
raw_prices = np.arange(1, _RAW_COUNT + 1, dtype=np.float64)

series = PriceSeries(
    dates=raw_dates,
    prices=raw_prices,
    asset="AAPL",
)

windowed = create_windowed_supervised_dataset(series)

The first sample contains prices 1, 2, and 3. Its target is price 4, and its target date is the fourth date in the raw series:

np.testing.assert_allclose(
    windowed.inputs[0, :, 0],
    np.array([1.0, 2.0, 3.0]),
)
np.testing.assert_allclose(windowed.targets[0], np.array([4.0]))
assert windowed.target_dates[0] == raw_dates[3]

The final sample contains the last three prices before the final raw observation. Its target is 263, and its date is the final raw date:

np.testing.assert_allclose(
    windowed.inputs[-1, :, 0],
    np.array([260.0, 261.0, 262.0]),
)
np.testing.assert_allclose(windowed.targets[-1], np.array([263.0]))
assert windowed.target_dates[-1] == raw_dates[-1]

This boundary example makes the no-future-leakage invariant visible: the target never appears inside its own input window. It also explains why target dates, rather than the final input dates, are retained. A forecast plotted against target_dates is compared with the observation it was intended to predict.

Structural validation and failure cases

WindowedDataset validates the recurrent and target shapes, matching sample counts, finite values, positive prices, and chronological target dates. The companion validate_windowed_dataset function checks the expected relationship between source length, window length, and sample count. It rejects a non-integer or non-three window length in this reproduction, because the generated container deliberately implements the paper-specific (n, 3, 1) contract.

The function also rejects common data problems before a model sees them. It raises a domain-specific DataValidationError when the input is not a PriceSeries, when the series does not contain exactly 263 prices and dates, when values are non-finite or non-positive, or when the constructed result violates the expected shapes. These checks protect both temporal alignment and the downstream Keras input contract.

The supplied test file expresses the important boundary checks with deterministic prices:

def test_recurrent_shape_is_three_by_one() -> None:
    """Verify the recurrent input and scalar-target tensor contracts."""
    dataset = create_windowed_supervised_dataset(_synthetic_price_series())

    assert dataset.inputs.shape == (_EXPECTED_SAMPLE_COUNT, _WINDOW_LENGTH, 1)
    assert dataset.targets.shape == (_EXPECTED_SAMPLE_COUNT, 1)
    assert dataset.target_dates.shape == (_EXPECTED_SAMPLE_COUNT,)
    assert dataset.inputs.dtype == np.float64
    assert dataset.targets.dtype == np.float64

The other generated tests check that the sample count is exactly 260 and that the first and last windows use the correct preceding prices, targets, and dates. Those tests are planned verification artifacts; under the current run policy, they were not executed and no passing result is claimed.

Why windowing comes before splitting

The paper-consistent dimensional interpretation is to construct all 260 one-step samples first and then partition those samples chronologically into 156 training, 52 testing, and 52 validation samples. Splitting the raw 263 observations first would describe a different boundary procedure and could change which windows are available at each partition boundary.

After windowing, each sample already has an input, a target, and a target date. The later splitting stage can therefore slice all three arrays together while preserving order. This keeps the temporal direction clear: no future target is included in an earlier sample, and no random shuffle is needed to create the supervised dataset.

Advanced detail: target shape conventions

Some forecasting libraries store scalar targets as a one-dimensional array with shape (n,). That representation can be valid, but this implementation intentionally standardizes targets as (n, 1). The choice makes the target contract match a model whose final output is one value per sample and helps prevent accidental broadcasting during training and evaluation. It is an implementation convention, not an additional feature or a change to the paper's one-step forecasting task.

No display equation is included here: the supplied paper context contains no canonical equation records. The index-level procedure, array shapes, and code slices above provide the supported mathematical meaning without reconstructing an equation from memory.

Chronological Splitting and Training-Only Scaling

How do we prevent a forecasting experiment from learning about the future before evaluation? The answer is procedural: create all supervised windows first, place them in chronological order, divide those windows into contiguous partitions, and learn scaling parameters from the training partition only. Later testing and validation values may be transformed with that state, but they must not determine it.

This section uses the paper-consistent dimensional interpretation. Each asset begins with 263 weekly adjusted closing prices. A three-period one-step window produces 260 supervised samples. Those samples are then divided into 156 training samples, 52 testing samples, and 52 validation samples. AAPL and MSFT go through this process independently.

What the paper specifies—and what it leaves open

The paper specifies chronological partitioning and training-only MinMax normalization. This protects the temporal direction of the experiment: earlier samples are used before later samples. It does not specify whether the scaler was fitted to the raw one-dimensional training-price series, to flattened training windows, or to training inputs and targets together.

That missing detail matters because a MinMax scaler stores data-dependent bounds. Different fitting populations can produce different normalized values, training losses, and eventually price-unit metrics. The implementation therefore exposes two named ScalerConvention choices. This is an implementation decision, not a recovered detail from the paper.

The paper also keeps testing and validation conceptually distinct, although its reported metrics are described as validation evaluation. Because the exact use of the testing partition is unclear, the reproduction retains both partitions and requires the caller to choose explicitly later.

Partition the windowed samples

split_windowed_dataset receives a WindowedDataset and a SplitConfig. The dataset contains recurrent inputs, scalar targets, and target dates. The configuration contains the three requested partition sizes. The function slices all three fields at the same boundaries and returns a DatasetSplits object with fields such as train_inputs, test_inputs, and validation_inputs.

The important algorithm is short and deliberately does not shuffle:

train_end = config.train_size
test_end = train_end + config.test_size

train_inputs = dataset.inputs[:train_end]
train_targets = dataset.targets[:train_end]
train_dates = dataset.target_dates[:train_end]

test_inputs = dataset.inputs[train_end:test_end]
test_targets = dataset.targets[train_end:test_end]
test_dates = dataset.target_dates[train_end:test_end]

validation_inputs = dataset.inputs[test_end:]
validation_targets = dataset.targets[test_end:]
validation_dates = dataset.target_dates[test_end:]

With the default configuration, train_end is 156 and test_end is 208. Therefore the training indices are 0 through 155, testing indices are 156 through 207, and validation indices are 208 through 259. Every windowed sample is consumed once.

The configuration is supplied by get_default_split_config:

def get_default_split_config() -> SplitConfig:
    """Return the paper-consistent chronological 156/52/52 split.

    The split is a derived dimensional correction: 263 raw observations minus
    a three-period window produce 260 supervised samples.
    """
    return SplitConfig(train_size=156, test_size=52, validation_size=52)

The SplitConfig constructor rejects sizes that do not sum to 260. That check encodes the derived dimensional contract rather than treating 263 raw observations as 263 supervised examples.

Worked example: inspect the three boundaries

The following example uses the monotonic synthetic WindowedDataset introduced in the windowing discussion. It is useful because the sample indices are easy to inspect; it is not AAPL or MSFT data.

split_config = get_default_split_config()
splits = split_windowed_dataset(windowed, split_config)

assert splits.train_inputs.shape == (156, 3, 1)
assert splits.train_targets.shape == (156, 1)
assert len(splits.train_dates) == 156

assert splits.test_inputs.shape == (52, 3, 1)
assert splits.test_targets.shape == (52, 1)
assert len(splits.test_dates) == 52

assert splits.validation_inputs.shape == (52, 3, 1)
assert splits.validation_targets.shape == (52, 1)
assert len(splits.validation_dates) == 52

Notice the shape invariants. Each input remains rank three: sample count, sequence length 3, and one feature. Each target remains a rank-two scalar batch with shape (n, 1). Dates have one entry per target, not one entry per input timestep.

validate_split_boundaries checks these shapes and also checks finite positive values, chronological order within each partition, and non-overlap between adjacent partitions. This function is a boundary guard: it does not repair an incorrectly ordered dataset or infer a missing split.

Random splitting would undermine the intended experiment. It could place later price regimes in the training partition and earlier regimes in validation, making the held-out data no longer a clean later period. The generated splitter therefore performs contiguous slicing and never calls a random shuffling operation.

Fit the scaler after splitting

Scaling occurs after the chronological split so that the fitting function receives training arrays only. The generated API is:

scaler = fit_training_scaler(
    train_inputs=splits.train_inputs,
    train_targets=splits.train_targets,
    convention=ScalerConvention.RAW_TRAINING_PRICES,
)

fit_training_scaler returns a one-feature scikit-learn MinMaxScaler. It validates that inputs have shape (n, 3, 1) and that targets have shape (n,) or (n, 1). In this reproduction, the selected convention is recorded through the enum value, so experiment metadata can state which policy was requested.

There is an important implementation limitation to understand precisely. The current helper does not receive the original raw training series separately. Consequently, its RAW_TRAINING_PRICES branch fits from the price values available in the training inputs and targets. The FLATTENED_TRAINING_WINDOWS branch currently uses the same supplied training values after flattening. The two names preserve the unresolved protocol choice in the public API, but they do not yet implement two different fitting populations. This should not be mistaken for evidence that the paper used either convention.

The fitting function deliberately receives no testing or validation arrays. Applying a fitted scaler later is allowed; refitting it later is not.

Transform while preserving shapes

Use the same scaler for every partition belonging to the same asset:

scaled_train_inputs = transform_windows(scaler, splits.train_inputs)
scaled_test_inputs = transform_windows(scaler, splits.test_inputs)
scaled_validation_inputs = transform_windows(scaler, splits.validation_inputs)

scaled_train_targets = transform_targets(scaler, splits.train_targets)
scaled_test_targets = transform_targets(scaler, splits.test_targets)
scaled_validation_targets = transform_targets(scaler, splits.validation_targets)

assert scaled_train_inputs.shape == (156, 3, 1)
assert scaled_test_inputs.shape == (52, 3, 1)
assert scaled_validation_inputs.shape == (52, 3, 1)
assert scaled_train_targets.shape == (156, 1)
assert scaled_test_targets.shape == (52, 1)
assert scaled_validation_targets.shape == (52, 1)

transform_windows flattens the final one-feature view temporarily for the scaler, then reshapes the result back to (n, 3, 1). transform_targets similarly preserves the target's scalar-batch convention. Neither function refits the scaler, and neither changes sample order or date arrays.

AAPL and MSFT need separate scaler instances. Their prices have different levels and distributions, and the paper models the assets in isolation. Reusing one scaler across both series would introduce cross-asset information and would not match the stated univariate design.

Inverse transformation before evaluation

The GRU is trained on normalized values, so its predictions are initially in normalized target units. Before calculating price-unit metrics, convert both predictions and actual targets back with the same training-fitted scaler:

restored_training_targets = inverse_targets(scaler, scaled_train_targets)

The inverse operation preserves the input rank. For a held-out validation batch, an array with shape (52, 1) returns to the same shape and represents adjusted-price values again. The evaluation stage then flattens or otherwise validates the arrays according to its metric contract, aligns them with the 52 validation target dates, and computes metrics in original units.

This ordering is essential. Evaluating normalized values would produce errors whose units and scale depend on preprocessing rather than on the stock price itself. The intended sequence is therefore: fit on training values, transform all model inputs and targets, predict in normalized space, inverse-transform predictions and actuals, then calculate price-unit metrics.

Planned tests and verification status

The generated test file contains focused checks for the split and scaler invariants, including the expected 156/52/52 shapes, date contiguity, unchanged scaler state after transforming validation values, shape preservation, and inverse transformation. These tests are planned artifacts, not evidence of completed verification. Under the supplied run policy, code execution, testing, local static verification, semantic code verification, and tutorial verification were disabled. No test result is claimed here.

Reproduction checklist

For this preprocessing stage, a reviewable run should record the following:

1. The input asset and its 263 chronological adjusted prices.

2. The three-period windowing step that produced 260 samples.

3. The SplitConfig values 156, 52, and 52.

4. The fact that testing remains separate from validation.

5. The selected ScalerConvention value.

6. The scaler's training-only fitting population.

7. The recurrent and target shapes before and after transformation.

8. The scaler instance retained for inverse transformation.

9. The explicit held-out subset selected for later evaluation.

The result is leakage-aware preprocessing, but not a guarantee of accurate forecasting. Scaling choices can affect the numerical outcome even when the GRU architecture and training settings remain unchanged. Exact reproduction still depends on unresolved source-data, preprocessing, and evaluation details.

Asset-Specific GRU Architectures

How does the implementation turn a three-price sequence into one next-price forecast? A gated recurrent unit, or GRU, reads the input steps in order and maintains an internal representation of the sequence. Here, each sample contains three prior adjusted prices for one asset, so the model receives a tensor with shape (batch, 3, 1) and returns one normalized scalar for every sample, with shape (batch, 1).

The paper uses separate univariate models. The AAPL model never receives MSFT values, and the MSFT model never receives AAPL values. This isolation is important because the paper evaluates asset-specific architectures rather than a shared multivariate network.

Table 1 configurations versus implementation choices

The reported final configurations are paper facts. AAPL uses three GRU layers with widths (240, 16, 16), while MSFT uses two layers with widths (224, 48). Both configurations use dropout 0.0, learning rate 0.001, MSE loss, batch size 8, a maximum of 100 epochs, and patience 7.

The generated AssetModelConfig stores these values together with the asset identifier:

@dataclass(frozen=True)
class AssetModelConfig:
    """Immutable configuration for one isolated asset GRU model.

    ``units`` contains the number of units in each stacked GRU layer.  The
    paper-selected values are AAPL ``(240, 16, 16)`` and MSFT ``(224, 48)``.
    Activation functions, initializers, return-sequence behavior, and the
    output-layer details are not specified by the supplied paper text and are
    intentionally not represented as paper facts here.
    """

    asset: str
    units: tuple[int, ...]
    dropout: float
    learning_rate: float
    loss_name: str
    batch_size: int
    max_epochs: int
    patience: int

The validation in AssetModelConfig rejects unsupported assets, empty or non-positive unit tuples, invalid dropout rates, non-positive learning rates, unsupported losses, and invalid training limits. These checks protect the configuration boundary; they do not add new model behavior described by the paper.

The factory returns the reported settings directly. It does not pretend to recover the original Hyperband search space or random seed:

def get_final_model_config(asset: str) -> AssetModelConfig:
    """Return the supplied Table 1 final configuration for ``asset``.

    The layer widths, zero dropout, learning rate, MSE loss, batch size,
    epoch limit, and patience are paper facts.  They are returned directly
    rather than inferred from an unavailable Hyperband search space.
    """
    if asset == "AAPL":
        units = (240, 16, 16)
    elif asset == "MSFT":
        units = (224, 48)
    else:
        raise ValueError(f"unsupported asset {asset!r}; expected AAPL or MSFT")

    return AssetModelConfig(
        asset=asset,
        units=units,
        dropout=0.0,
        learning_rate=0.001,
        loss_name="mse",
        batch_size=8,
        max_epochs=100,
        patience=7,
    )

Connecting recurrent layers

A stacked GRU passes a sequence from each intermediate recurrent layer to the next one. In the generated builder, every GRU except the last uses return_sequences=True. The terminal GRU returns its final representation, which is passed to Dense(1) to produce one scalar output.

These settings are implementation decisions, not additional paper facts. The supplied paper text does not specify return-sequence behavior, activation functions, recurrent activations, initializers, or the terminal output-layer type. The generated implementation makes those choices explicit so that the model has a definite tensor contract:

for index, width in enumerate(units):
    is_terminal = index == len(units) - 1
    representation = keras.layers.GRU(
        width,
        activation="tanh",
        recurrent_activation="sigmoid",
        dropout=float(dropout),
        return_sequences=not is_terminal,
        name=f"gru_{index + 1}",
    )(representation)

outputs = keras.layers.Dense(1, name="next_adjusted_close")(representation)
return keras.Model(inputs=inputs, outputs=outputs, name="stacked_gru_forecaster")

The helper validate_gru_architecture checks that the unit tuple is non-empty and contains positive integers, that dropout lies in the interval [0, 1), and that the input shape is exactly (3, 1). Its failure cases are deliberate: a different sequence length or feature count would no longer represent the paper's three prior prices and one univariate feature.

No GRU gate equations are displayed here. The supplied context contains no canonical equation records, so reconstructing equations from general GRU knowledge would incorrectly present an unavailable formula as part of the paper specification.

Building an isolated asset model

build_asset_gru_model accepts one validated AssetModelConfig and an input shape. It delegates layer construction to build_gru_stack, then checks the resulting model's input and output shapes. TensorFlow is imported lazily by the lower-level builder, allowing data and preprocessing components to remain conceptually separate from the optional neural-network runtime.

from stock_forecasting.config import get_final_model_config
from stock_forecasting.models.gru_model import (
    build_asset_gru_model,
    describe_model_contract,
)

# Table 1 paper configuration: AAPL uses three GRU layers.
aapl_config = get_final_model_config("AAPL")
aapl_model = build_asset_gru_model(aapl_config, input_shape=(3, 1))
print(describe_model_contract(aapl_model))

The corresponding MSFT model is constructed independently:

from stock_forecasting.config import get_final_model_config
from stock_forecasting.models.gru_model import build_asset_gru_model

# Table 1 paper configuration: MSFT uses two GRU layers.
msft_config = get_final_model_config("MSFT")
msft_model = build_asset_gru_model(msft_config, input_shape=(3, 1))

describe_model_contract reports the model name, input and output shapes, recurrent layer widths, dropout values, and return_sequences settings. It rejects models without inspectable shapes, without GRU layers, with the wrong scalar output, or with an invalid sequence-return arrangement. The model-building path also raises a domain-specific ConfigurationError if TensorFlow is unavailable or if the constructed model violates the expected shape contract.

Worked example: compare the two architectures

The following inspection workflow expresses the intended comparison without training either model:

from stock_forecasting.config import get_final_model_config
from stock_forecasting.models.gru_model import (
    build_asset_gru_model,
    describe_model_contract,
)

for asset in ("AAPL", "MSFT"):
    config = get_final_model_config(asset)
    model = build_asset_gru_model(config, input_shape=(3, 1))
    contract = describe_model_contract(model)

    print(asset, config.units)
    print(contract["input_shape"], contract["output_shape"])

The expected structural interpretation is that AAPL has three recurrent layers with widths 240, 16, and 16, while MSFT has two with widths 224 and 48. Both accept (None, 3, 1) at the model boundary and emit (None, 1), where None represents a variable batch size. The configurations also carry training settings such as learning rate, loss name, batch size, maximum epochs, and patience, although those settings are applied later during compilation and training.

The generated contract tests check these structural invariants without asserting forecast quality:

def test_assets_are_not_combined() -> None:
    """AAPL and MSFT are built as independent models with no shared layers."""
    aapl_model = build_asset_gru_model(get_final_model_config("AAPL"))
    msft_model = build_asset_gru_model(get_final_model_config("MSFT"))

    assert aapl_model is not msft_model
    assert _layer_units(aapl_model) == (240, 16, 16)
    assert _layer_units(msft_model) == (224, 48)

    aapl_layer_ids = {id(layer) for layer in _gru_layers(aapl_model)}
    msft_layer_ids = {id(layer) for layer in _gru_layers(msft_model)}
    assert aapl_layer_ids.isdisjoint(msft_layer_ids)
    assert len(_gru_layers(aapl_model)) != len(_gru_layers(msft_model))

This test describes the intended invariant: separate models have separate layer objects and preserve their asset-specific widths. It does not establish that training succeeds, that predictions are accurate, or that the paper's reported metrics are reproduced.

Normalized outputs and the next stage

The model operates on scaled inputs and produces a scaled next-price prediction. The output is therefore not yet an adjusted closing price in the original price units. The later evaluation stage must use the same asset-specific training-fitted scaler to inverse-transform predictions and targets before computing price-unit metrics.

The architecture alone cannot establish the paper's reported MSE, MAE, MAPE, or forecast accuracy. Those values also depend on the local data snapshot, scaling convention, training randomness, callback behavior, evaluation subset, and other unspecified protocol details. Under the supplied run policy, this model construction and its tests were not executed, and neither static verification nor semantic code verification was completed.

Hyperband Tuning, Adam Training, and Best Checkpoints

How should we choose and train a GRU without allowing later observations to influence the model? The workflow has two distinct stages. First, optional Hyperband tuning compares candidate asset-specific configurations using validation loss. Second, the selected configuration is trained with Adam while a separate chronological validation partition controls early stopping and checkpoint selection.

The paper describes this procedure, but it does not provide enough tuner metadata to reconstruct the original search exactly. The generated implementation therefore supports both a direct reproduction path using the reported Table 1 configurations and an optional, explicitly documented local Hyperband search.

Paper facts and implementation decisions

The paper reports separate searches and models for AAPL and MSFT. The final configurations are:

• AAPL: GRU widths (240, 16, 16);

• MSFT: GRU widths (224, 48);

• dropout 0.0 for both assets;

• Adam learning rate 0.001;

• MSE loss;

• batch size 8;

• maximum 100 epochs; and

• early-stopping patience 7.

The paper also identifies validation loss as the Hyperband selection objective. Training uses the 156-sample training partition, while the separate 52-sample validation partition is supplied for monitoring. The 52-sample testing partition is not passed to model fitting or callback monitoring.

Several details remain unspecified: the exact Hyperband search space, bracket and iteration settings, random seed, tuner stopping criteria, Keras validation arrangement, callback filename, monitor mode, and whether best weights were restored automatically. The generated code makes these choices explicit rather than presenting them as recovered paper facts. In particular, it uses shuffle=False, minimizes val_loss, restores best weights, and saves weights to a .weights.h5 file.

A resource-aware search, without claiming exact tuner recovery

Hyperband is a resource-aware search strategy. It begins with multiple candidate configurations and allocates more training resources to candidates that appear promising. In this reproduction, a candidate may vary in recurrent depth, unit counts, dropout, learning rate, and loss name. The search is run separately for each asset, so AAPL and MSFT observations never enter the same tuner.

The search-space container documents the local choices:

@dataclass(frozen=True)
class TuningSearchSpace:
    """Candidate values used by the optional Hyperband reproduction.

    The paper reports that Hyperband searched layer depth, recurrent widths,
    dropout, learning rate, and loss, but it does not provide the original
    candidate ranges.  Consequently, the values returned by
    :func:`default_search_space` are explicit implementation decisions rather
    than recovered paper facts.
    """

    layer_options: tuple[int, ...]
    unit_options: tuple[int, ...]
    dropout_options: tuple[float, ...]
    learning_rates: tuple[float, ...]
    loss_options: tuple[str, ...]

TuningSearchSpace contains finite candidate values and validate_search_space rejects malformed options such as nonpositive widths, invalid dropout values, duplicate entries, or unsupported losses. The generated default_search_space includes the reported widths and learning rate so the Table 1 configurations are representable, but its ranges are implementation decisions.

The search function receives a DatasetSplits object whose scaled training inputs have shape (156, 3, 1) and targets have shape (156, 1). Its validation inputs and targets have shapes (52, 3, 1) and (52, 1). It passes only the training and validation arrays to Keras Tuner:

tuner.search(
    train_data.train_inputs,
    train_data.train_targets,
    validation_data=(
        train_data.validation_inputs,
        train_data.validation_targets,
    ),
    epochs=100,
    batch_size=8,
    shuffle=False,
    verbose=0,
)

The objective is explicitly val_loss with direction "min". The testing partition remains distinct. If TensorFlow or Keras Tuner is unavailable, the function raises a configuration error rather than silently substituting another algorithm. It also rejects invalid shapes, wrong sample counts, non-finite values, and searches that produce no finite validation score.

The paper's final architecture can be selected directly when the original tuner metadata is unavailable:

from stock_forecasting.tuning.hyperband import select_reported_configuration

reported_aapl_config = select_reported_configuration("AAPL")
reported_msft_config = select_reported_configuration("MSFT")

select_reported_configuration returns the Table 1 configuration. It does not rerun Hyperband and does not claim that the generated tuner recovered the paper's search. This direct path is the appropriate default for the requested reproduction target.

Normalized-space MSE and model compilation

During training, predictions and targets remain on the scaler's normalized price scale. The model returns one scalar prediction per sample, so predictions and targets should use compatible scalar-batch shapes, preferably (batch, 1). The models.losses module validates this contract and rejects unintended broadcasting or mismatched batch lengths.

The paper selects MSE for both final asset models. No canonical loss equation was supplied in the paper context, so this tutorial does not reconstruct or display one. Instead, resolve_loss("mse") delegates the numerical loss behavior to the configured Keras/TensorFlow version. This keeps the framework's reduction behavior in one place and avoids presenting an invented formula as a paper equation.

Compilation constructs Adam with the configured learning rate and attaches the resolved loss:

# The paper-selected MSE loss operates on normalized scalar predictions
# and targets; inverse transformation is intentionally deferred to
# evaluation.
optimizer = Adam(learning_rate=config.learning_rate)
loss = resolve_loss(config.loss_name)

try:
    model.compile(optimizer=optimizer, loss=loss)
except (TypeError, ValueError) as exc:
    raise ConfigurationError(
        f"Keras rejected the model compilation configuration: {exc}"
    ) from exc

compile_gru_model accepts an uncompiled Keras-compatible model and an AssetModelConfig, then returns the same model after compilation. It validates the learning rate and supported loss before importing TensorFlow. A missing TensorFlow installation or a model without a callable compile method is treated as a clear configuration failure.

The validation loss values reported by the paper are not explicitly labeled as normalized-space or original-price-unit values. The generated code treats training and validation loss as model-space quantities and reserves price-unit MSE, MAE, and MAPE for the inverse-transformed evaluation stage. This is an implementation interpretation, not a confirmed explanation of the paper's reported validation-loss units.

Training with chronological validation

Once a model configuration is selected, train_gru_with_callbacks compiles the model, builds the callbacks, and calls Keras fit. The important data-flow rule is that only the training partition updates weights:

history = compiled_model.fit(
    splits.train_inputs,
    splits.train_targets,
    validation_data=(splits.validation_inputs, splits.validation_targets),
    epochs=config.max_epochs,
    batch_size=config.batch_size,
    shuffle=False,
    callbacks=callbacks,
    verbose=0,
)

Here, splits.train_inputs has shape (156, 3, 1) and splits.train_targets has shape (156, 1). The validation arrays have corresponding shapes (52, 3, 1) and (52, 1). shuffle=False is an implementation decision that preserves chronological sample order; the extracted paper does not explicitly state its shuffle setting.

The configuration supplies the paper's batch size, epoch limit, and patience. The helper checks these shapes and counts before training, rejects non-finite arrays, and verifies that training and validation dates match their expected partition sizes. It never passes test_inputs or test_targets to fit, so the testing data cannot influence parameter updates or callback selection.

Early stopping and best checkpoints

Early stopping answers a practical question: when validation loss stops improving, should training continue merely because the maximum epoch count has not been reached? The generated callback policy monitors val_loss, minimizes it, tolerates seven non-improving epochs, and restores the best in-memory weights. A matching ModelCheckpoint saves the best weights locally.

# The paper specifies validation-loss monitoring and best-state retention;
# mode='min' and restore_best_weights=True make those choices explicit.
early_stopping = keras.callbacks.EarlyStopping(
    monitor=_VALIDATION_MONITOR,
    mode="min",
    patience=config.patience,
    restore_best_weights=True,
    verbose=0,
)
checkpoint = keras.callbacks.ModelCheckpoint(
    filepath=str(checkpoint_path),
    monitor=_VALIDATION_MONITOR,
    mode="min",
    save_best_only=True,
    save_weights_only=True,
    verbose=0,
)
return [early_stopping, checkpoint]

The paper specifies patience 7 and checkpointing, but not the monitor mode, restoration flag, or file format. The generated implementation chooses mode="min" because lower validation loss is preferred, uses restore_best_weights=True, and requires a checkpoint path ending in .weights.h5. build_training_callbacks creates the parent directory and raises an error for an invalid path or missing TensorFlow.

After fitting, train_gru_with_callbacks requires the checkpoint file to exist and reloads it. The returned TrainingResult contains the model, training history, and checkpoint path. Thus, forecasting uses the best validation-loss state rather than assuming the final epoch is best. This is a runtime contract; no training or checkpoint creation was performed for this article under the run policy.

Worked example: direct reproduction versus optional tuning

For the direct Table 1 path, keep tuning disabled. The pipeline then obtains the reported asset configuration through get_final_model_config, builds the corresponding GRU, trains it with the callback policy, and retains separate artifacts under the selected asset directory:

from pathlib import Path

from stock_forecasting.pipeline import run_asset_pipeline

result = run_asset_pipeline(
    asset="AAPL",
    data_dir=Path("data/raw"),
    output_dir=Path("artifacts"),
    evaluation_subset="validation",
    tune=False,
)

This call is procedural rather than a claimed result. The pipeline loads local AAPL records, preserves the 156/52/52 split, fits the scaler on training data, constructs the reported AAPL configuration, trains with validation monitoring, and evaluates the explicitly selected validation subset. The returned AssetRunResult retains the raw PriceSeries, windowed dataset, scaled splits, scaler, model, training result, forecast, and diagnostic metadata.

To request the optional local Hyperband path from the command line, use:

python scripts/run_reproduction.py \
    --data-dir data/raw \
    --output-dir artifacts \
    --assets AAPL MSFT \
    --evaluation-subset validation \
    --tune

Without --tune, the command uses the reported Table 1 configurations. With --tune, it uses the generated search space and records tuner artifacts separately for each asset. Neither invocation silently merges testing and validation, and neither command's numerical outcome is claimed here.

What this stage establishes—and what it does not

This implementation preserves the paper's main training constraints: separate asset models, validation-loss selection, Adam at learning rate 0.001, MSE loss, batch size 8, at most 100 epochs, patience 7, and best-weight checkpointing. It also makes unresolved choices visible: tuner ranges, random seed, Hyperband details, Keras defaults, callback behavior, and evaluation protocol.

The paper's reported AAPL and MSFT validation-loss values and forecast metrics remain reference values, not achieved results. Under the supplied run policy, no training, tuning, code execution, tests, static verification, semantic code verification, tutorial verification, or final quality review was performed.

Inverse Transformation, Forecast Metrics, and Reference Comparison

How do normalized neural-network outputs become interpretable stock-price forecasts? The evaluation stage reverses the training-time scaling, pairs every prediction with the date it forecasts, and computes errors in original adjusted-price units. It also makes one unresolved protocol choice visible: whether to evaluate the testing partition or the validation partition.

The paper describes its reported evaluation as validation-based, but it also defines separate training, testing, and validation subsets. Because the supplied paper description does not explain whether testing was ignored, evaluated separately, or used in another way, the generated implementation never merges these partitions silently. The caller must choose one explicitly.

Paper facts and implementation decisions

The paper reports separate results for AAPL and MSFT. It evaluates next-period adjusted closing prices with MSE, MAE, MAPE, and forecast accuracy. The supplied series contain positive prices, so percentage error is defined for these inputs.

The implementation makes several choices explicit rather than presenting them as recovered paper details:

• validation is the default command-line evaluation choice, but test is also available.

• The scaler is the one fitted from training data only.

• Predictions and targets are inverse-transformed before price-unit metrics are calculated.

• A nonpositive actual price is rejected for MAPE rather than handled with an invented zero-denominator rule.

• The paper's Table 3 values are comparison references, not guaranteed outputs from this implementation.

No canonical metric equations were supplied in the source context. Accordingly, this section explains the metric contracts and their code mappings in prose rather than reconstructing formulas.

1. Select exactly one held-out partition

After windowing and chronological splitting, each asset has 156 training samples, 52 testing samples, and 52 validation samples. The select_evaluation_subset function accepts a DatasetSplits object and returns the inputs, targets, and target dates from exactly one held-out partition.

inputs, scaled_targets, target_dates = select_evaluation_subset(
    splits,
    subset="validation",
)

print(inputs.shape)          # expected: (52, 3, 1)
print(scaled_targets.shape)  # expected: (52, 1)
print(target_dates.shape)    # expected: (52,)

To select the testing partition instead, change only the explicit protocol argument:

test_inputs, test_targets, test_dates = select_evaluation_subset(
    splits,
    subset="test",
)

The function returns three aligned objects:

• input windows with shape (52, 3, 1);

• normalized scalar targets with shape (52, 1); and

• a chronological DatetimeIndex of 52 target dates.

Its failure case is deliberate: any subset other than test or validation raises EvaluationProtocolError. This prevents a misspelled or implicit evaluation choice from producing an apparently valid report. The function also does not concatenate the two held-out partitions.

2. Predict, restore price units, and preserve dates

The GRU receives normalized windows and should produce one normalized scalar per window. evaluate_forecasts validates this contract, calls the model's predict method, converts predictions and targets back through the same training-fitted scaler, and then constructs a ForecastResult.

result = evaluate_forecasts(
    model=model,
    inputs=inputs,
    targets=scaled_targets,
    dates=target_dates,
    scaler=scaler,
)

print(result.dates.shape)       # expected: (52,)
print(result.actuals.shape)     # expected: (52,)
print(result.predictions.shape) # expected: (52,)
print(result.metrics)

The returned ForecastResult contains dates, actuals, predictions, and metrics. The actuals and predictions are one-dimensional price arrays. For either default held-out partition, each has length 52.

The function checks several invariants before asking the model to forecast:

• inputs must have shape (n, 3, 1);

• targets may have shape (n,) or (n, 1), then are normalized internally;

• dates must be unique, nonmissing, and chronological;

• inputs and targets must contain the same number of samples; and

• all numeric values must be finite.

It also requires a model with a callable predict method. A prediction returned as (n, 1) is converted to (n,); an incompatible output shape, a failed prediction, or a non-finite prediction raises EvaluationProtocolError.

The lower-level inverse_transform_forecasts helper performs only the restoration step. It applies the same scaler to normalized predictions and normalized targets, then returns equal-length one-dimensional arrays in adjusted-price units:

from stock_forecasting.evaluation.forecasting import inverse_transform_forecasts

price_predictions, price_targets = inverse_transform_forecasts(
    scaler,
    normalized_predictions,
    scaled_targets,
)

Using one training-fitted scaler for both arrays is important. Applying a separately fitted evaluation scaler would change the meaning of the predictions and could introduce information from the held-out data. The paper specifies training-only fitting, but it does not resolve whether fitting used the raw training prices or flattened training windows and targets. That convention must therefore be recorded with the result artifact.

3. Understand the metric units

Once arrays are back in price units, compute_forecast_metrics delegates to the individual metric functions. The public interface is concise:

from stock_forecasting.evaluation.metrics import compute_forecast_metrics

metrics = compute_forecast_metrics(
    actuals=result.actuals,
    predictions=result.predictions,
)

print(f"MSE: {metrics.mse:.2f}")
print(f"MAE: {metrics.mae:.2f}")
print(f"MAPE: {metrics.mape:.2f}%")
print(f"Forecast accuracy: {metrics.forecast_accuracy:.2f}%")

The fields have different interpretations:

• MSE is mean squared error. Its unit is squared adjusted-price units, so large errors receive disproportionately more weight.

• MAE is mean absolute error in adjusted-price units. It is easier to read as a typical price-distance measure than MSE.

• MAPE is mean absolute percentage error, reported in percentage points.

• Forecast accuracy follows the paper's convention of taking 100 minus MAPE, also in percentage points.

The last quantity is not classification accuracy. It does not measure whether the model correctly predicted an upward or downward movement. It is simply the paper's complement of a percentage error.

The metric functions validate equal-length, one-dimensional, finite arrays. compute_mape additionally requires every actual price to be strictly positive. This matches the supplied AAPL and MSFT data and avoids silently choosing how to divide by zero or a negative price. The generated implementation raises EvaluationProtocolError when that assumption is violated.

4. Worked example: validation forecast and date-aligned plot

The following sequence makes the evaluation protocol visible: choose validation data, evaluate it, and plot the returned date-aligned result. The model and scaler are assumed to have been produced by the preceding training pipeline; this excerpt does not claim that training or forecasting was executed.

from pathlib import Path

from stock_forecasting.evaluation.forecasting import (
    evaluate_forecasts,
    select_evaluation_subset,
)
from stock_forecasting.eda.plots import plot_forecast, save_figure

inputs, scaled_targets, target_dates = select_evaluation_subset(
    splits,
    subset="validation",
)

result = evaluate_forecasts(
    model=model,
    inputs=inputs,
    targets=scaled_targets,
    dates=target_dates,
    scaler=scaler,
)

axis = plot_forecast(result)
save_figure(
    axis.figure,
    Path("artifacts/forecasts/AAPL_validation_forecast.png"),
)

plot_forecast reads the ForecastResult without mutating it. It checks that dates, actuals, and predictions have equal lengths, then draws both series against the target dates. Those dates are not the dates at the beginning of each input window: they identify the following observation that served as the target. This distinction keeps a forecast visually aligned with the price it was intended to predict.

To inspect testing separately, repeat the same process with subset="test" and write a distinct artifact name. Do not label a validation result as a test result, since the two partitions represent different chronological portions of the data.

5. Compare with Table 3 without claiming reproduction

The paper reports these reference values:

| Asset | MSE | MAE | MAPE | Forecast accuracy | |---|---:|---:|---:|---:| | AAPL | 55.80 | 6.11 | 3.37% | 96.63% | | MSFT | 144.44 | 9.41 | 2.55% | 97.45% |

compare_to_reference retrieves the appropriate paper reference, records the computed value, reference value, and signed difference for every metric, and can optionally calculate absolute-tolerance indicators. format_reference_report renders those fields as a readable report.

from stock_forecasting.evaluation.reference import (
    compare_to_reference,
    format_reference_report,
)

comparison = compare_to_reference(
    asset="AAPL",
    metrics=result.metrics,
)

print(format_reference_report({"AAPL": comparison}))

A comparison with an optional tolerance is still descriptive rather than a proof of reproducibility:

comparison = compare_to_reference(
    asset="MSFT",
    metrics=msft_result.metrics,
    tolerance=1.0,
)

print(format_reference_report({"MSFT": comparison}))

The report keeps units explicit: MSE and MAE differences use price-based units, while MAPE and forecast-accuracy differences use percentage points. Without a tolerance, the report states that tolerance was not assessed. With one, it reports whether each absolute difference falls within that numerical threshold; it does not establish that the data, training process, or evaluation protocol matched the paper.

6. Offline artifact reporting

The generated scripts/report_results.py script reads JSON forecast artifacts and reports their generated metrics separately from Table 3. Its load_result_artifact function requires dates, actuals, predictions, and all four metric fields. It rejects missing fields, invalid dates, unequal array lengths, non-finite values, and nonpositive actuals.

A typical command shape is:

python scripts/report_results.py \
    --artifact-dir artifacts/results \
    --assets AAPL MSFT \
    --tolerance 1.0

This reporter is intentionally offline. It does not download data or infer a missing evaluation subset from the JSON. The artifact-producing workflow should retain the asset name, selected subset, scaler convention, model configuration, target dates, and metric units alongside the arrays. That metadata is necessary when a discrepancy with Table 3 could arise from more than the GRU itself.

7. Interpreting absolute and relative errors

The paper reports larger absolute errors for MSFT—MSE 144.44 and MAE 9.41—than for AAPL—MSE 55.80 and MAE 6.11. MSFT nevertheless has the lower MAPE, 2.55% compared with AAPL's 3.37%, and therefore the higher reported forecast accuracy, 97.45% compared with 96.63%.

These values illustrate why both absolute and relative measures matter. A stock with a higher price level can have a larger dollar error while that error represents a smaller fraction of the actual price. The metrics answer different questions: MAE asks about price-unit distance, whereas MAPE asks about proportional distance.

These figures remain paper references in this reproduction. Exact numerical agreement is not guaranteed because the supplied context leaves the source-data version, weekly aggregation, scaler-fitting convention, random seed, complete Hyperband search space, Keras defaults, callback details, and reported evaluation-subset procedure incomplete.

8. Evaluation checklist

Before interpreting a forecast artifact, confirm that:

• the asset is recorded as either AAPL or MSFT and was modeled independently;

• one explicit subset, test or validation, is recorded;

• inputs have shape (52, 3, 1) for the selected default held-out partition;

• dates, actuals, and predictions have equal length and chronological alignment;

• the scaler was fitted on training data only and reused for inverse transformation;

• metrics were calculated after returning predictions and targets to price units;

• MSE and MAE are reported with price-based units;

• MAPE and forecast accuracy are reported as percentages;

• forecast accuracy is interpreted as 100 minus MAPE, not as classification accuracy; and

• Table 3 values are labeled as references rather than achieved results.

The generated metric tests describe these invariants, including inverse transformation, positive-price validation, the forecast-accuracy complement, and date preservation. However, the run policy disabled execution, test generation, static verification, semantic code verification, tutorial verification, and final review. No test result or numerical reproduction claim is made here.

Limitations, Static and Semantic Verification Status, and Reproduction Checklist

What would make this reproduction trustworthy—and what can it not establish? The workflow is a useful, structured experiment: it keeps AAPL and MSFT separate, preserves chronological order, prevents later values from fitting the scaler, and records the choices needed to turn the paper description into Python. It is not a guarantee that a GRU will forecast future markets reliably.

The study uses only 263 weekly adjusted closing-price observations for each of two technology stocks. Each model sees only three preceding prices and predicts one next price. That narrow design is faithful to the paper's scope, but it limits generalization. Market conditions can change, and a price-history-only model may smooth abrupt movements or behave differently under distribution shift. The resulting forecasts should therefore support analysis and financial-risk decisions, not operate as an autonomous trading system.

A final separation of evidence

A responsible reproduction distinguishes three categories:

• Paper facts: the separate AAPL and MSFT models, adjusted-price inputs, three-period windows, chronological partitions, Table 1 architectures, training settings, and Table 3 reference metrics.

• Derived structure: 263 raw observations minus a three-period window produces 260 supervised samples, which are partitioned as 156 training, 52 testing, and 52 validation samples.

• Implementation decisions: local CSV input instead of live retrieval; the selected scaler convention; decomposition, ADF, and autocorrelation parameters; Keras defaults not specified by the paper; callback details; and the locally defined Hyperband search space.

This distinction is especially important when comparing numbers. The paper's reference values—AAPL MSE 55.80, MAE 6.11, MAPE 3.37%, and forecast accuracy 96.63%; MSFT MSE 144.44, MAE 9.41, MAPE 2.55%, and forecast accuracy 97.45%—are targets for comparison. They are not evidence that this generated code achieved those values.

What can prevent exact numerical reproduction

Even with the same broad workflow, results can differ because the supplied paper description does not fully specify:

• the exact Yahoo Finance data version, retrieval query, weekly aggregation rule, and calendar handling;

• whether the scaler was fitted on raw training prices or flattened training windows and targets;

• random seeds and framework initialization;

• GRU activations, initializers, sequence-output settings, and the terminal output construction;

• the Hyperband search space, iterations, seed, and stopping rules;

• early-stopping and checkpoint details; and

• whether the reported evaluation used validation only, testing only, or another unresolved protocol.

The generated pipeline makes these uncertainties visible. It keeps the testing and validation partitions distinct and requires an explicit evaluation_subset choice. It also records the selected scaler convention and diagnostic parameters rather than implying that they were recovered from the paper.

Verification status: skipped, not passed

The run policy disabled execution, test generation, local static verification, semantic code verification, tutorial verification, and final quality review. The local verification record consequently reports verification_skipped, not a successful check. Semantic code verification was also skipped. No claim is made here that the package imports correctly, that a model trained, that tests passed, or that any metric was reproduced.

Static verification and semantic verification would answer different questions if they were later enabled. Static checks could inspect Python parsing, imports, dependency order, configuration consistency, tensor shapes, chronology, and leakage boundaries without training a model. Semantic review would examine whether the implementation's behavior actually matches the intended method—for example, whether only training values fit the scaler, whether testing data stays out of training, and whether dates remain aligned with target observations. Neither kind of check was completed in this run.

Worked example: the complete local workflow

The intended workflow begins with local appendix-style files. The repository does not download data or pretend to reproduce an unspecified Yahoo Finance retrieval procedure. Prepare AAPL.csv and MSFT.csv under a local input directory, with date and adjusted_close columns:

python scripts/prepare_local_data.py \
  --input-dir appendix \
  --output-dir data/raw

The preparation script validates the local records and writes the normalized schema. Its public function preserves the paper's 263-row contract:

def prepare_asset_file(
    source_path: Path,
    destination_path: Path,
    asset: str,
) -> None:
    """Validate a local appendix CSV and write its normalized raw-data form.

    The input is never downloaded or augmented.  The loader accepts the paper's
    date and adjusted-close fields (including documented column aliases), while
    this function writes the stable schema expected by later pipeline stages:
    ``date``, ``adjusted_close``.  Records are ordered chronologically and must
    contain exactly the paper's 263 observations.
    """

Next, generate non-mutating diagnostics for both assets:

python scripts/run_diagnostics.py \
  --data-dir data/raw \
  --output-dir artifacts/diagnostics

These reports and figures describe the data; they do not add decomposition components or other diagnostics to the GRU inputs. The generated command records implementation choices such as the decomposition period, ADF settings, and autocorrelation lag range.

Finally, run the reproduction with an explicit evaluation partition:

python scripts/run_reproduction.py \
  --data-dir data/raw \
  --output-dir artifacts \
  --evaluation-subset validation

To keep the testing partition separate but inspect it instead, choose test:

python scripts/run_reproduction.py \
  --data-dir data/raw \
  --output-dir artifacts \
  --evaluation-subset test

The --tune flag is another explicit choice. Without it, run_asset_pipeline uses the reported Table 1 configuration directly. With it, the implementation runs its documented, locally selected Hyperband search space; that search is not claimed to be the paper's unrecovered search space.

At the Python level, one asset follows the same sequence of responsibilities:

result = run_asset_pipeline(
    asset=asset,
    data_dir=data_dir,
    output_dir=output_dir / asset,
    evaluation_subset=args.evaluation_subset,
    tune=args.tune,
)

The returned AssetRunResult is intended to retain the raw PriceSeries, diagnostic record, WindowedDataset, scaled DatasetSplits, scaler, model, training artifact, and ForecastResult. This makes the pipeline inspectable: a reviewer can trace an asset from its 263 observations through 260 windows and the 156/52/52 partition to its selected held-out forecast. AAPL and MSFT are processed in separate calls, so their scalers, models, checkpoints, and metrics do not get combined.

Practical reproduction checklist

Use the following checklist when execution and verification are available:

1. Prepare inputs. Supply one local CSV per asset with date and adjusted_close; do not download or infer missing records.

2. Validate raw data. Confirm 263 rows, chronological dates, finite positive adjusted prices, and no missing values.

3. Run diagnostics. Save descriptive statistics, plots, decomposition, ADF, and autocorrelation outputs. Treat them as diagnostic artifacts only.

4. Create windows. Confirm 260 samples with inputs shaped (260, 3, 1), targets shaped (260, 1), and dates aligned to target observations.

5. Check partitions. Confirm contiguous chronological sizes of 156 training, 52 testing, and 52 validation samples.

6. Check scaling. Confirm that only training values fit the chosen scaler and that later subsets are transformed without refitting.

7. Build models separately. Confirm AAPL widths (240, 16, 16) and MSFT widths (224, 48), with zero dropout and one scalar output per sample.

8. Train carefully. Confirm Adam at learning rate 0.001, MSE loss, batch size 8, at most 100 epochs, patience 7, validation monitoring, and best-checkpoint use.

9. Select evaluation data explicitly. Record whether the held-out subset is validation or test; never merge them silently.

10. Evaluate in price units. Inverse-transform predictions and targets with the training-fitted scaler before computing MSE, MAE, MAPE, and forecast accuracy.

11. Compare transparently. Use compare_to_reference to report differences from Table 3, while labeling those values as paper references rather than pass/fail verification.

12. Record the environment. Preserve data provenance, dependency versions, random seeds, scaler convention, tuner settings, callback settings, and the evaluation choice.

The paper's conclusions also suggest what a stronger future study would need: more assets and time periods, rolling-origin evaluation, benchmark models, uncertainty estimates, confidence intervals, statistical significance analysis, and potentially technical, sentiment, volatility, or macroeconomic features. Those are sensible extensions, but they are outside this reproduction and must not be presented as part of the implemented method.

The central lesson is procedural rather than predictive: document the evidence boundary, preserve time order, expose unresolved choices, and never turn an unexecuted or unverified pipeline into a claimed result.

Use the button or URL below to download the source code.

Read more

Download Research Paper

The paper proposes a feature-wise compositional recurrent neural network approach for multivariate stock-price forecasting. The five OHLCV inputs are modeled separately with stacked LSTM, GRU, or SRU recurrent layers, regularized with dropout, fused by concatenation, optionally processed by additional recurrent layers and dense layers, and optimized using either Random Search (RS) or Grey Wolf Optimizer (GWO). The study evaluates 54 configurations on daily Hang Seng Index data from Yahoo Finance. The reported best result is an LSTM-GWO configuration, followed by GRU-GWO and SRU-GWO configurations.

Implementation Assumptions

• The five input channels are ordered Open, High, Low, Close, Volume.

• The 20-day and 40-day values are treated as input window lengths, not forecast horizons.

• Because the paper does not identify the target or horizon, the implementation exposes targetcolumn and forecasthorizon as configuration choices; the tutorial must label these as implementation decisions.

• Scaling defaults to training-only fitting over [0, 0.95] to reduce leakage, while documenting that the paper does not specify the fitting partition.

• The paper's opaque configuration labels such as LSTM-GWO (1-1-0-1) are retained as benchmark labels and are not decoded into architecture parameters.

• The training loss, gradient optimizer, search spaces, GWO budget, and metric formulas are configurable implementation decisions rather than asserted paper facts.

• SRU is represented behind a local interface because the paper does not specify an SRU variant or API.

• Yahoo Finance acquisition is optional for offline reproducibility; local OHLCV files and deterministic synthetic data are supported.

• No reported benchmark is treated as verified reproduction output.

• No canonical equations were supplied, so no equation-level LaTeX or equation_id mapping is asserted.

• Verification is limited to planned static and semantic checks; execution, generated tests, and local verification are disabled by policy.

Scope, evidence status, and reproduction target

What should happen when five market measurements describe the same trading day? The paper’s central idea is to avoid sending all of them immediately into one undifferentiated recurrent pathway. Instead, Open, High, Low, Close, and Volume each receive their own temporal pathway. The resulting representations are then joined before the final forecast. A useful intuition is to treat the five pathways as specialists: each studies one signal over time, and a prediction head combines their reports.

Here, OHLCV means the ordered features Open, High, Low, Close, and Volume. The generated model expects tensors in batch × time × feature layout. For a 20-day input window, for example, one batch has shape batch × 20 × 5; for the alternative window, the shape is batch × 40 × 5. The final dimension must retain the canonical OHLCV order.

What the paper establishes

The supplied paper context supports a feature-wise compositional recurrent method. Each OHLCV stream is processed independently by a stacked LSTM, GRU, or SRU family. Dropout is used for regularization, the five representations are fused through concatenation, and optional recurrent and dense layers produce the stock-price prediction. The paper also compares two outer hyperparameter strategies:

• Random Search (RS) samples candidate configurations stochastically.

• Grey Wolf Optimizer (GWO) maintains a population of candidates and updates them according to their relative objective values.

The paper reports 54 evaluated configurations and identifies an LSTM-GWO configuration as its best reported result. It also reports GRU-GWO and SRU-GWO benchmark configurations. These are useful comparison targets, but they do not fully describe how to construct the corresponding models.

The phrase “univariate encoding” in the extracted paper should be read carefully. The implementation target is not one single-variable forecasting model. It is feature-wise encoding: five separate one-channel sequences are modeled and then fused into a multivariate representation.

What remains underdetermined

An exact reproduction cannot be recovered from the supplied context alone. Several decisions needed by executable code are absent or ambiguous:

• The Yahoo Finance symbol, date range, adjustment policy, and missing-row treatment are not uniquely specified.

• The target column and forecast horizon are not identified. The paper names stock-price prediction, but does not establish that the target is Close or that the horizon is one day.

• The recurrent widths, exact layer counts, hidden-state extraction rule, dense widths, activations, and SRU implementation are missing.

• The training loss, gradient optimizer, learning-rate schedule, stopping rule, and checkpoint policy are unspecified.

• RS bounds, sampling distributions, trial counts, and validation details are absent.

• GWO population size, iteration budget, bounds, initialization, discrete-parameter encoding, and update specification are absent.

• Metric formulas and conventions, including percentage-error zero handling and evaluation scale, are not supplied.

The labels LSTM-GWO (1-1-0-1), GRU-GWO (2-1-1-1), and SRU-GWO (2-2-0-0) therefore remain opaque benchmark labels. They must not be decoded into presumed layer counts, dropout switches, or dense-layer structures.

How the generated package responds

The generated package treats the paper as a method specification plus a set of reported targets, not as a complete executable recipe. DataConfig exposes the target, horizon, window length, split, and scaling range. ModelConfig exposes the recurrent family and architecture choices. TrainingConfig, SearchConfig, and EvaluationConfig make training, search, and metric conventions explicit. validate_reproduction_config provides a common boundary for checking these records.

The model assembly is represented by CompositionalRNN and build_compositional_model. Their intended responsibilities are to create five independent recurrent encoders, concatenate their representations, apply the configured fusion treatment, and produce the configured output dimension. RS and GWO remain separate outer-search interfaces, but both are intended to evaluate candidates through a shared training and validation objective.

This separation is important. A derived implementation decision—such as using Close as the target, a one-step horizon, or a particular loss—can be changed when better evidence becomes available without changing the paper’s feature-wise architecture or the public configuration interfaces. It also prevents a convenient default from being mistaken for a fact reported by the paper.

Worked example: an explicit tutorial configuration

For a small paper-oriented setup, choose window_length=20, target_column='Close', and forecast_horizon=1 in DataConfig. These values are tutorial choices: the paper supports 20-day input windows, but it does not specify Close as the target or a one-step forecast horizon. A corresponding ModelConfig can select LSTM and an explicitly documented number of stream layers and units, while a TrainingConfig records the chosen loss, optimizer, batch size, epochs, and seed.

The resulting input contract is batch × 20 × 5, and the configured output dimension determines the prediction shape. That shape contract describes the scaffold; it does not demonstrate that the configuration reproduces the paper’s reported LSTM-GWO result.

Evidence status

This section and the generated package distinguish three kinds of statements:

1. Paper facts: feature-wise OHLCV processing, recurrent-family comparison, concatenation fusion, RS and GWO comparison, and the reported benchmark labels and values.

2. Derived explanations: the “five specialists” intuition and the interpretation of separate pathways as feature-wise rather than single-variable modeling.

3. Implementation decisions: target and horizon defaults, exact architecture settings, preprocessing policies, training choices, search budgets, and metric conventions.

The run policy disabled execution, local static verification, semantic code verification, generated test execution, tutorial verification, and final quality review. Consequently, this package should be described as a configurable reproduction scaffold, not as a verified implementation or a successful reproduction of the published results.

Data contract: HSI OHLCV acquisition and provenance

How can a forecasting model learn from market data if the source rows, feature order, and cleaning decisions are unclear? The data contract answers that question before any recurrent layer is built. It specifies which five measurements enter the model, preserves their chronological order, and records choices that the paper does not disclose.

What the paper specifies—and what it does not

The paper identifies daily Hang Seng Index history obtained through Yahoo Finance/YFinance and names the raw Open, High, Low, Close, and Volume fields as its OHLCV inputs. In the generated package, the canonical order is fixed as Open, High, Low, Close, Volume. A raw table therefore has shape num_days × 5: each row represents one observation and each column represents one input channel.

The paper does not provide a unique Yahoo Finance ticker, date range, download options, adjusted-price setting, missing-value policy, or duplicate-row policy. These are not minor details. Different tickers or date ranges change the observations, while adjusted and unadjusted prices can produce different price histories. The generated implementation consequently makes these choices explicit and stores them in AcquisitionMetadata rather than presenting a particular choice as recovered paper methodology.

The reproduction scope also stays deliberately narrow. It retains the five raw OHLCV fields and does not add technical indicators or other engineered predictors. Technical indicators are mentioned by the paper as possible future work, not as part of the evaluated method.

Canonical columns and provenance

CANONICAL_OHLCV_COLUMNS is the package-level contract used by acquisition and model-facing code:

# Canonical feature order required by the paper-oriented data and model APIs.
OHLCVColumns: TypeAlias = tuple[str, str, str, str, str]
CANONICAL_OHLCV_COLUMNS: OHLCVColumns = (
    "Open",
    "High",
    "Low",
    "Close",
    "Volume",
)

This excerpt is copied from src/compositional_rnn_stock/types.py. OHLCVColumns describes a five-name tuple, while CANONICAL_OHLCV_COLUMNS supplies the concrete order. Preserving this order matters because later model code interprets the final input axis as five channels. Reordering columns—for example, placing Volume before Close—would change the meaning of the model input without changing its shape.

The acquisition record captures the choices surrounding those columns:

@dataclass(frozen=True)
class AcquisitionMetadata:
    """Provenance and cleaning-policy record for an OHLCV acquisition.

    The paper does not specify the Yahoo Finance symbol, date range, adjustment
    mode, missing-value policy, or duplicate-row policy.  These fields preserve
    the choices made by a local reproduction instead of treating them as paper
    facts.
    """

    source: str
    symbol: str | None
    start: str | None
    end: str | None
    adjusted: bool | None
    frequency: str
    requested_columns: OHLCVColumns
    missing_value_policy: str
    duplicate_policy: str
    ordering_policy: str

This is an implementation record, not an additional claim about the paper. For a network acquisition, source, symbol, start, end, and adjusted identify the request. requested_columns records the five retained fields, and the remaining policy fields describe how the returned table was accepted. For an offline archive, some fields such as symbol may be None, but the artifact still records that the source was a local CSV.

Two acquisition paths

The generated adapter supports a network path through load_hsi_ohlcv_from_yfinance and an offline path through load_ohlcv_csv. The dispatcher requires exactly one source, preventing an ambiguous call that supplies both a symbol and a local file:

def acquire_hsi_ohlcv_data(
    *,
    symbol: str | None = None,
    local_path: Path | None = None,
    start: str | None = None,
    end: str | None = None,
    adjusted: bool = False,
) -> tuple[pd.DataFrame, AcquisitionMetadata]:
    """Dispatch to an explicit local-file or Yahoo Finance acquisition mode."""

    if (symbol is None) == (local_path is None):
        raise ValueError("provide exactly one of symbol or local_path")
    if local_path is not None:
        return load_ohlcv_csv(local_path, CANONICAL_OHLCV_COLUMNS)
    return load_hsi_ohlcv_from_yfinance(symbol=symbol, start=start, end=end, adjusted=adjusted)

This excerpt is copied from src/compositional_rnn_stock/data/acquisition.py. The function returns a pandas table plus its provenance record. In local mode, it delegates to load_ohlcv_csv; in Yahoo Finance mode, it delegates to load_hsi_ohlcv_from_yfinance. The optional yfinance import is isolated inside the network function, so an offline workflow does not need that package merely to load an archived file.

For a local archive, a typical call is:

frame, metadata = load_ohlcv_csv(Path("data/hsi_ohlcv.csv"), CANONICAL_OHLCV_COLUMNS)

The generated CSV loader looks for a conventional Date, date, Timestamp, or timestamp column and uses it as the index when present. It then selects and orders only the canonical OHLCV fields. An archive should therefore preserve its date column whenever the identity of trading days matters. If no recognized date column exists, the loader retains the file's row index; that is a documented limitation of the local input rather than evidence that dates were absent from the original paper dataset.

For Yahoo Finance, the caller must supply the symbol and may supply explicit date bounds and an adjustment choice. The paper does not identify these values, so a tutorial command should not silently substitute a supposedly canonical ticker or time range. The command-line entry point instead requires explicit source and target-related settings, while the acquisition function records the selected source parameters.

Validation and cleaning

Acquisition first validates the structural contract. validate_ohlcv_table requires a nonempty pandas table, all five columns, numeric finite values, an increasing index, and no duplicate index values. It does not infer missing observations or create rows. The relevant responsibility is summarized by its public interface:

def validate_ohlcv_table(frame: pd.DataFrame) -> None:
    """Validate a chronological table containing exactly the five OHLCV fields.

    Validation deliberately rejects missing values and duplicates rather than
    fabricating observations.  The paper leaves those policies unspecified, so
    callers must clean or otherwise resolve such records before acquisition
    results are accepted.
    """

The implementation's cleaning layer makes the missing and duplicate policies explicit:

cleaned = clean_ohlcv(frame, missing_policy="drop", duplicate_policy="reject")

This call is the focused usage pattern exposed by src/compositional_rnn_stock/data/cleaning.py. The `

Scaling, target definition, and sliding windows

How does an ordered table of daily market observations become training data without allowing future information to leak backward? The preprocessing pipeline answers this in three stages: scale the five OHLCV channels, construct overlapping time windows, and split the resulting supervised samples chronologically.

The paper states that the features are normalized to the interval [0, 0.95] and that the model uses 20-day and 40-day temporal windows. It does not supply a canonical scaling equation, identify the target column, or specify the forecast horizon. The generated implementation therefore makes those choices explicit instead of presenting them as recovered paper facts.

Scaling five channels independently

FeaturewiseMinMaxScaler treats Open, High, Low, Close, and Volume as five separate numeric channels. It stores one minimum and maximum for each channel, so the much larger numerical scale of Volume does not determine the scaling of price features. Its final feature axis must have length five, while arbitrary leading dimensions are allowed. Thus it can process both a raw table shaped num_days × 5 and model inputs shaped samples × window_length × 5.

The conventional feature-wise min-max mapping used by the generated scaler is an implementation decision because no canonical equation was supplied. The implementation also chooses to fit statistics on training observations only. This is a leakage-prevention policy: test observations are transformed using training metadata, rather than helping define that metadata. The paper states the target interval but does not state which partition fits the normalization statistics, so this choice must remain visible in provenance.

A focused part of the generated scaler shows the fitting contract:

scaler = FeaturewiseMinMaxScaler(feature_range=feature_range)
scaler.fit(training_values)
return scaler

Here, training_values is expected to contain only the earlier chronological observations selected by the caller. The scaler records feature_min_, feature_max_, and feature-specific scale metadata. Constant training features are mapped to the lower bound and restored to their fitted constant during inverse transformation, avoiding division by zero. Values outside the fitted extrema are not clipped by transform; the generated code documents this as an intentional choice so extrapolation remains visible.

The fit, transform, and inverse_transform methods preserve the input shape. That matters later when a prediction is converted back from normalized units to original price units. If evaluation is performed on original prices, the target column must be identified so that the corresponding feature scaling metadata can be applied.

Target and horizon are explicit choices

The paper describes stock-price prediction and lists OHLCV predictors, but it does not say whether the target is Close, adjusted Close, another price field, or a multivariate output. It also does not specify how far into the future the target lies. The generated windowing API therefore requires both target_index and forecast_horizon.

window_length describes how many past observations enter one model input. For paper-oriented runs, it is treated as either 20 or 40 trading days. forecast_horizon is different: it describes the offset from the end of the input window to the target observation. The 20-day and 40-day values should not be interpreted as prediction horizons merely because they are temporal quantities.

For a start position s, the generated implementation uses the input rows from s through the end of the selected window, then takes the target at the configured future position. With window_length=3 and forecast_horizon=1, the first input contains rows 0, 1, and 2, and its target is row 3. A horizon of 2 would instead select row 4 for that same first input.

The key implementation fragment is copied from make_sliding_windows:

sample_count = day_count - window_length - forecast_horizon + 1
if sample_count <= 0:
    raise ValueError(
        "insufficient observations for the requested window and horizon: "
        f"num_days={day_count}, window_length={window_length}, "
        f"forecast_horizon={forecast_horizon}"
    )

inputs = np.empty((sample_count, window_length, _FEATURE_COUNT), dtype=np.float64)
targets = np.empty(sample_count, dtype=np.float64)

target_timestamps = None if timestamp_array is None else np.empty(sample_count, dtype=timestamp_array.dtype)

for start in range(sample_count):
    end = start + window_length
    target_position = end + forecast_horizon - 1
    inputs[start] = array[start:end]
    targets[start] = array[target_position, target_index]
    if target_timestamps is not None:
        target_timestamps[start] = timestamp_array[target_position]

The resulting WindowedDataset records inputs with shape samples × window_length × 5 and scalar targets with shape samples. It also records the selected target column, horizon, and optional target timestamps. The five channels remain in canonical Open, High, Low, Close, Volume order; window construction does not reorder or mix them.

The following focused example is the same alignment pattern used by the generated contract tests. It uses synthetic arrays only and does not represent Hang Seng Index data:

dataset = make_sliding_windows(
    values,
    target_index=3,
    window_length=3,
    forecast_horizon=2,
    timestamps=timestamps,
)

assert dataset.inputs.shape == (4, 3, 5)
assert dataset.targets.shape == (4,)
np.testing.assert_array_equal(dataset.inputs[0], values[0:3])
np.testing.assert_array_equal(dataset.targets, values[4:8, 3])

This example uses target index 3, which corresponds to Close in the canonical order. That index is an example configuration choice, not evidence that the paper definitively forecasts Close.

Chronological splitting

After windows and targets have been aligned, chronological_split divides complete supervised samples without shuffling. Its boundary is the floor of sample_count * train_fraction. The first partition contains earlier target timestamps, and the second contains later ones. The function rejects a split that would leave either partition empty.

The generated test expresses the central ordering invariant:

train, test = chronological_split(dataset, train_fraction=0.6)

expected_train_count = int(dataset.sample_count * 0.6)
assert train.sample_count == expected_train_count
assert test.sample_count == dataset.sample_count - expected_train_count
assert train.sample_count > 0
assert test.sample_count > 0
assert train.timestamps is not None
assert test.timestamps is not None
assert int(train.timestamps[-1]) < int(test.timestamps[0])

These are contract tests supplied in the generated repository, but they were not executed under the authoritative run policy. They describe intended behavior rather than reporting a completed verification.

The paper reports an 80%/20% chronological partition and gives sample counts of 4,678 training samples and 1,170 test samples. Those counts are useful reproduction targets, but they do not uniquely reveal the raw date range, the number of overlapping windows, or whether the 20-day and 40-day datasets were constructed separately.

How the prepared-data pipeline orders its work

prepare_data records the selected target, horizon, window length, split ratio, scaling range, and cleaning policies. Its relevant flow is:

observation_split = _observation_split_index(
    raw_values.shape[0], data_config.split_ratio
)

scaler = fit_scaler_on_training_observations(
    raw_values=raw_values,
    split_index=observation_split,
    feature_range=data_config.scaling_range,
)
scaled_values = scaler.transform(raw_values)
dataset = make_sliding_windows(
    values=scaled_values,
    target_index=_target_index(data_config.target_column),
    window_length=data_config.window_length,
    forecast_horizon=data_config.forecast_horizon,
    timestamps=timestamps,
)
train_dataset, test_dataset = chronological_split(
    dataset,
    train_fraction=data_config.split_ratio,
)

Notice the distinction between the observation boundary used to fit the scaler and the later sample split. The generated pipeline first determines an observation-level boundary, fits the scaler on the leading observations, transforms the ordered series, constructs windows, and then partitions the completed samples. This is the package's documented leakage-avoidance design, not a uniquely specified procedure from the paper. Because overlapping windows can span an observation boundary, a stronger reproduction should confirm the intended split order from the original experiment details.

The pipeline also records scaler_fit_policy as training_observations_only, along with the target column, horizon, observation counts, window sample counts, and scaling range. These records make it possible to replace an assumption later without changing the public interfaces.

Returning to original units

Normalized inputs are useful for model training, but reported price errors may need to be calculated in original units. FeaturewiseMinMaxScaler.inverse_transform restores all five channels, while a target-specific evaluation adapter can select the configured target feature. This distinction is important because the paper does not specify whether its metrics were calculated on normalized values or inverse-transformed prices.

The generated offline demonstration follows the same policy on synthetic data: it fits the scaler on an earlier chronological portion, transforms the complete ordered sequence, and constructs 20-day windows. The demonstration is useful for checking shape contracts conceptually, but it is not HSI data and cannot establish the paper's reported performance.

In summary, the preprocessing contract is precise about shapes and ordering but deliberately honest about unresolved semantics. The implementation preserves five feature channels, treats 20 and 40 as input lengths, aligns each window with an explicitly configured future target, and prevents test observations from fitting the scaler. The target column, forecast horizon, split order, and normalization convention remain choices that must be documented for any reproduction run.

The feature-wise compositional recurrent model

How can five related market signals contribute to one forecast without being mixed too early? The paper’s answer is feature-wise composition: Open, High, Low, Close, and Volume each travel through an independent recurrent pathway. Their learned summaries are then placed side by side and passed to a prediction head. This is not five separate forecasting models. The final forecast is multivariate because the five representations are fused before prediction.

The paper supports this overall structure, but it does not uniquely specify the recurrent widths, exact layer counts, representation extraction rule, dense-layer widths, or activation functions. The generated package therefore implements these details through ModelConfig. The code is a configurable reproduction scaffold, not a verified reconstruction of every hidden architecture choice.

Tensor flow through five independent streams

The public model input, x, has shape batch × time_steps × 5. The final dimension follows the canonical order Open, High, Low, Close, Volume. A 20-day run therefore uses batch × 20 × 5; a 40-day run uses batch × 40 × 5.

FeatureWiseEncoder splits that final dimension into exactly five tensors. Each slice has shape batch × time_steps × 1, so one recurrent stack receives one feature channel rather than the complete OHLCV vector. The five encoders are separate module instances, which means their parameters are not shared.

The following excerpt shows the stream-level contract. Notice that the recurrent stack is configured with return_sequences=False; this local choice selects the final time-step representation for fusion. The paper does not say whether the final hidden state, a complete sequence, or another aggregation was used.

class FeatureStreamEncoder(nn.Module):
    """Encode one OHLCV channel with its own recurrent stack.

    The input contract is ``(batch, time_steps, 1)`` and the output contract is
    ``(batch, hidden_size)``.  A separate instance is created for every OHLCV
    channel, so parameters are not shared between feature streams.
    """

    def __init__(
        self,
        recurrent_family: RecurrentFamily | str,
        hidden_size: int,
        layers: int,
        dropout_rate: float = 0.0,
        feature_name: str | None = None,
    ) -> None:
        super().__init__()
        if isinstance(hidden_size, bool) or not isinstance(hidden_size, int) or hidden_size <= 0:
            raise ValueError(f"hidden_size must be a positive integer; received {hidden_size!r}")
        if isinstance(layers, bool) or not isinstance(layers, int) or layers <= 0:
            raise ValueError(f"layers must be a positive integer; received {layers!r}")
        if feature_name is not None and feature_name not in CANONICAL_OHLCV_COLUMNS:
            raise ValueError(
                f"feature_name must be one of {CANONICAL_OHLCV_COLUMNS!r}; received {feature_name!r}"
            )

        self.feature_name = feature_name
        self.input_size = 1
        self.hidden_size = hidden_size
        self.layer_count = layers
        self.recurrent_family = RecurrentFamily.coerce(recurrent_family)
        self.recurrent = build_recurrent_stack(
            family=self.recurrent_family,
            input_size=self.input_size,
            hidden_size=hidden_size,
            layers=layers,
            return_sequences=False,
        )
        self.dropout = DualDropout(stream_rate=dropout_rate, fusion_rate=0.0)

The important invariant is input_size=1: every stream receives one feature channel. After recurrent processing, each stream produces a fixed-width tensor with shape batch × hidden_size. Stream dropout is then applied. Dropout is a training-time regularizer: it masks values while the model is training and is disabled when the module is in evaluation mode.

FeatureWiseEncoder.forward preserves the channel order with a one-element split along the final axis:

        # Each slice retains its one-channel axis: (batch, time_steps, 1).
        feature_slices = torch.split(x, split_size_or_sections=1, dim=-1)
        if len(feature_slices) != 5:
            raise RuntimeError(f"expected five feature slices, received {len(feature_slices)}")

        encoded_streams = tuple(
            encoder(feature_slice)
            for encoder, feature_slice in zip(self.encoders, feature_slices)
        )

This code does not perform early averaging, summation, or mixing. The five encoded outputs remain separate until the fusion module receives them.

One interface for LSTM, GRU, and SRU

A recurrent layer processes a sequence while maintaining a learned state that carries information across time steps. LSTM and GRU are the two standard recurrent families exposed by the generated wrapper. The paper also evaluates SRU, but the supplied paper context does not identify an SRU variant or Python API.

RecurrentFamily provides a common selector, and build_recurrent_stack returns a RecurrentStack with the same input and output contract for each family. For a stream, the input is batch × time_steps × 1. With final-time-step reduction, the output is batch × hidden_size.

class RecurrentFamily(str, Enum):
    """Supported recurrent-family selectors.

    ``SRU`` is included because it is one of the families evaluated by the
    paper.  The paper does not identify an SRU variant or Python API, so the
    local adapter below is an explicit implementation choice rather than a
    claim of exact SRU reproduction.
    """

    LSTM = "LSTM"
    GRU = "GRU"
    SRU = "SRU"

For LSTM and GRU, the wrapper constructs batch-first PyTorch stacks. The local SRUAdapter is different: it is a dependency-free compatibility adapter supplied by the generated implementation. Its documentation explicitly says that it is not a canonical SRU implementation. Consequently, selecting SRU makes the interface available for experimentation, but it must not be described as proof of paper-equivalent SRU behavior.

The wrapper also validates rank, feature width, positive time steps, and floating-point input. Invalid inputs fail explicitly rather than being silently reshaped. This matters because changing batch × time_steps × features to another ordering would change the meaning of the recurrent computation.

Dropout and concatenation fusion

The paper describes dual dropout: one stage after feature-specific recurrent processing and another stage around feature fusion. The first stage is clear enough to place after each stream encoder. The second stage is not: “around fusion” does not establish whether it is before concatenation, after concatenation, or both. DualDropout and ConcatenationFusion expose this as fusion_placement.

Concatenation means placing the representations end to end along the final dimension. If each of the five streams has width hidden_size, the fused representation has width 5 × hidden_size. The operation preserves the batch dimension and does not combine channels by summation.

        # The paper's stated fusion operation is concatenation along the
        # representation dimension; stream representations are never summed.
        fused = torch.cat(processed_streams, dim=-1)

        if self.fusion_placement in {"after_concatenation", "both"}:
            fused = self.fusion_dropout.apply_fusion(fused)
        return fused

validate_stream_shapes requires exactly five floating-point tensors, matching leading dimensions, a positive representation width, the same dtype, and the same device. A mismatch such as one stream having a different batch size is rejected before concatenation. The default generated placement applies fusion dropout after concatenation, but that default is an implementation decision rather than a recovered paper setting.

Optional post-fusion processing and prediction

After fusion, PostFusionHead can either send the fused vector directly to dense layers or apply additional recurrent processing first. The paper allows post-fusion recurrent layers but does not specify how a fused vector should be treated as a sequence. The generated head makes one explicit choice: it adds a single time step, giving a tensor shaped batch × 1 × fused_features, and then uses the final representation from that one-step recurrent stack.

Dense layers then produce the configured target output. The generated implementation uses ReLU between configurable dense layers, followed by a final linear layer. Both the widths and the activation choice are local implementation decisions. The output shape is batch × output_dimension, where output_dimension must agree with the separately configured target definition.

Assembling the complete model

CompositionalRNN connects the stages in order: validate the input, encode five streams, concatenate them, apply the optional post-fusion head, and validate the prediction shape. Its central forward path is deliberately explicit:

        # Feature-wise recurrent encoding: five tensors of shape
        # (batch, stream_units), preserving the canonical OHLCV order.
        encoded_streams = self.feature_encoder(x)
        if len(encoded_streams) != self.input_feature_count:
            raise RuntimeError(
                f"expected {self.input_feature_count} encoded streams, "
                f"received {len(encoded_streams)}"
            )

        # Paper architecture stage: fuse representations by concatenation.
        fused = self.fusion(encoded_streams)
        expected_fused_shape = (x.shape[0], self.fused_representation_size)
        if tuple(fused.shape) != expected_fused_shape:
            raise RuntimeError(
                "fusion returned an unexpected shape: "
                f"received {tuple(fused.shape)}, expected {expected_fused_shape}"
            )

        # Optional post-fusion recurrence and dense prediction are delegated to
        # the head; its recurrent representation is a configurable choice.
        prediction = self.post_fusion_head(fused)

The model boundary requires exactly five channels and a positive batch and time dimension. It returns batch × output_dimension; an unexpected output shape raises an error. These checks enforce the paper-supported architecture without pretending that unspecified widths or layer counts have been recovered.

Focused configuration example

The following excerpt is adapted exactly from the generated model-contract test. It demonstrates a small local configuration with one recurrent layer, eight stream units, one dense layer, and one output. Those values are demonstration settings, not the paper’s reported best architecture. The 20 in the input shape represents the paper-oriented window choice, while the target semantics remain configured elsewhere.

def _make_model_config(
    family: RecurrentFamily | str = RecurrentFamily.LSTM,
    *,
    stream_dropout: float = 0.0,
    fusion_dropout: float = 0.0,
) -> ModelConfig:
    """Build a small deterministic configuration for contract tests.

    These tests verify structural invariants of the implementation rather than
    the paper's unresolved architecture details or opaque benchmark labels.
    """
    return ModelConfig(
        recurrent_family=family,
        stream_layers=1,
        stream_units=8,
        stream_dropout=stream_dropout,
        fusion_dropout=fusion_dropout,
        fusion_placement="after_concatenation",
        post_fusion_layers=0,
        post_fusion_units=8,
        dense_layers=(8,),
        output_dimension=1,
    )

A caller would construct the model and prepare an input with shape batch × 20 × 5 as follows:

model = build_compositional_model(_make_model_config())
inputs = _make_inputs(batch_size=4, time_steps=20)

predictions = model(inputs)

The intended contract is predictions with shape 4 × 1 for this configuration. This excerpt was not executed under the run policy, so it demonstrates the interface and expected shapes only; it does not establish runtime correctness or paper-level reproduction.

The opaque labels in the reported results—LSTM-GWO (1-1-0-1), GRU-GWO (2-1-1-1), and SRU-GWO (2-2-0-0)—are deliberately not translated into ModelConfig fields. The supplied paper context does not define what those four positions mean. Keeping them as benchmark labels prevents an attractive but unsupported architecture claim.

The generated test plan reinforces structural invariants such as five independent streams, additive fusion width, common family dispatch, and training-only dropout. Those tests were not executed: local static verification, semantic verification, and code execution were disabled or skipped by policy.

Training one candidate without hiding missing specifications

What does it mean to train one candidate model in this reproduction? The candidate receives batches of chronological windows, produces predictions, computes a selected scalar loss, and updates its trainable parameters with gradients. After each epoch, a separate validation pass records loss with dropout disabled and without changing the parameters. This inner training loop is distinct from Random Search and Grey Wolf Optimizer (GWO), which choose different candidate configurations outside the loop.

Paper facts and implementation choices

The paper says that recurrent models are trained and compared using validation performance, and it names learning rate, batch size, and training epochs among the optimized hyperparameters. However, the supplied paper context does not specify the training loss, base gradient optimizer, initialization, learning-rate schedule, random seed, batch-shuffling policy, early-stopping rule, or checkpoint-selection rule. It also does not clearly define how the reported 80%/20% partition relates to the validation loss discussed in the method.

The generated code makes these gaps visible. TrainingConfig supplies the local choices, while ForecastLoss supports explicit mse, mae, and huber objectives. These are available implementation options, not claims about which loss reproduced the paper. Likewise, make_optimizer supports explicit gradient optimizers for a candidate; this inner optimizer should not be confused with RS or GWO, which operate at the outer configuration-search level.

The generated pipeline uses the prepared test partition as the validation loader for a one-candidate run. Its source documentation explicitly calls this an implementation decision because the paper does not define a separate validation construction. Consequently, this scaffold should not be described as recovering the paper’s exact validation protocol.

Validating predictions and targets

A forecasting loss is meaningful only when each prediction is paired with its corresponding target. validate_loss_inputs requires PyTorch tensors with floating-point dtypes, identical shapes, at least one element, and finite values. A mismatch, empty batch, integer tensor, or non-finite value raises an error rather than allowing silent broadcasting or an invalid objective.

The public loss interface is ForecastLoss. Its output is a scalar tensor, so it can participate in backpropagation. The generated implementation uses mean reduction for each supported local objective. The following excerpt shows the explicit dispatch; it is copied from src/compositional_rnn_stock/losses.py.

        if self.name == "mse":
            loss = F.mse_loss(predictions, targets, reduction="mean")
        elif self.name == "mae":
            loss = F.l1_loss(predictions, targets, reduction="mean")
        else:
            loss = F.smooth_l1_loss(
                predictions,
                targets,
                beta=self.huber_delta,
                reduction="mean",
            )

The inputs have the model’s configured target shape. In the scalar-target pipeline, the model returns batch × 1, while window datasets expose targets as batch. The training loop performs only this unambiguous single-output reshape; other mismatches are rejected. That safeguard matters because accidental broadcasting could produce a finite-looking loss with incorrect semantics.

One epoch: update in training mode, evaluate in validation mode

During training, train_one_candidate puts the model in training mode, reads a batch shaped batch × time_steps × 5, computes predictions, aligns the targets, backpropagates the scalar loss, and calls the selected optimizer. The five-channel input contract is checked before the model is used. A training loader with no batches or non-finite values is rejected.

The core update sequence is shown below. This excerpt is copied from src/compositional_rnn_stock/training/loops.py.

        for batch in train_loader:
            inputs, targets = _prepare_batch(batch, device, "training")
            optimizer.zero_grad(set_to_none=True)
            predictions = model(inputs)
            targets = _align_targets(predictions, targets)
            loss = loss_fn(predictions, targets)
            loss.backward()
            optimizer.step()

After the training batches, evaluate_loss switches the model to evaluation mode and wraps inference in torch.no_grad(). Evaluation mode is important for the compositional model because dropout must be disabled when validation predictions are measured. The function restores the model’s previous training state afterward, computes a value-weighted mean over the validation targets, and rejects an empty or non-finite result.

    was_training = model.training
    model.eval()
    total_loss = 0.0
    total_values = 0
    try:
        with torch.no_grad():
            for batch in loader:
                inputs, targets = _prepare_batch(batch, device, "validation")
                predictions = model(inputs)
                targets = _align_targets(predictions, targets)
                loss = loss_fn(predictions, targets)
                value_count = int(targets.numel())
                total_loss += float(loss.detach().cpu().item()) * value_count
                total_values += value_count
    finally:
        model.train(was_training)

This separation is the main training invariant: validation loss is observed, not optimized directly. The validation loader is never passed to backward() or optimizer.step() by this workflow. The paper’s exact split and model-selection policy remain unresolved, so the generated code retains the final state after the configured epochs rather than silently inventing a best-checkpoint rule.

Optimizer and reproducibility settings

make_optimizer constructs the inner gradient optimizer from the configured name and learning rate. The generated implementation accepts adam, adamw, and sgd in this function. It validates that the learning rate is positive and finite, that the model has trainable parameters, and that the optimizer name is supported. The paper does not identify which of these, if any, was used.

    if normalized == "adam":
        return Adam(parameters, lr=learning_rate)
    if normalized == "adamw":
        return AdamW(parameters, lr=learning_rate)
    return SGD(parameters, lr=learning_rate)

set_global_seed seeds supported local Python, NumPy, and PyTorch generators. SeedContext additionally snapshots and restores available random states and can request deterministic PyTorch algorithms. These facilities improve the comparability of local experiments, but the paper does not report its seed or deterministic-backend settings. A seed therefore belongs in provenance, not in a claim that the paper used the same setting.

Histories and provenance records

Each completed epoch becomes an EpochRecord containing its epoch number, training loss, and validation loss. TrainingHistory preserves insertion order and rejects non-increasing epoch numbers. Its best_epoch helper can identify the earliest record with the smallest available validation loss, but the training loop does not automatically restore that epoch’s weights. Selecting and restoring a best checkpoint would be an additional explicit implementation decision.

The outer record is TrainingRun. It links the trained CompositionalRNN, its TrainingHistory, the ModelConfig, the TrainingConfig, and a prepared-data identifier. That identifier records acquisition and preprocessing metadata, window length, forecast horizon, target column, and train/test sample counts. This prevents a local loss curve from becoming detached from choices such as Close versus another target, a 20-day versus 40-day window, or a particular scaling policy.

Worked example: defining a local candidate run

The following configuration fragment is copied from scripts/train_model.py. It selects Close and a one-step horizon, but those are tutorial-level implementation choices because the paper does not identify the target column or forecast horizon.

    data_config = DataConfig(
        symbol=args.symbol,
        local_path=args.local_path,
        start=args.start,
        end=args.end,
        target_column=args.target_column,
        forecast_horizon=args.forecast_horizon,
        window_length=args.window_length,
        split_ratio=args.split_ratio,
        scaling_range=(0.0, 0.95),
    )

A focused call path inside train_one_candidate then creates the selected loss and optimizer and trains for the configured number of epochs. This excerpt is copied from src/compositional_rnn_stock/training/loops.py.

    loss_fn = _loss_for_config(config)
    optimizer = make_optimizer(model, config)
    device = _model_device(model)
    history = TrainingHistory()

At the pipeline level, run_training validates the prepared data and configurations, builds chronological loaders, constructs the model, checks that loader inputs have shape batch × time_steps × 5, and delegates to train_one_candidate.

    trained_model, history = train_one_candidate(
        model=model,
        train_loader=train_loader,
        validation_loader=validation_loader,
        config=training_config,
    )

This call defines one local experiment. It does not establish that the loss, optimizer, architecture settings, validation protocol, or final model state match the paper. It also does not execute here: the authoritative run policy disabled code execution and verification.

Why this boundary matters

The gradient loop answers, “How do we fit one selected candidate?” RS and GWO answer a different question, “Which candidate settings should we try next?” Keeping those responsibilities separate allows both search methods to use the same loss, loaders, validation objective, and provenance schema. It also makes missing paper specifications inspectable instead of burying them in defaults.

For this scaffold, the honest output of training is a locally configured model, an epoch-aligned history, and provenance describing the choices. No training result, convergence behavior, or successful reproduction of the paper’s reported benchmarks is claimed. Verification was skipped under policy, so the code excerpts describe the generated interfaces and intended control flow rather than a checked execution outcome.

Random Search and Grey Wolf Optimizer

How should a reproduction choose among many possible recurrent-network recipes? Random Search (RS) samples independent candidates, while Grey Wolf Optimizer (GWO) maintains a population of candidates and moves them using the best-ranked candidates as guides. In both cases, a candidate is more than a model family: it can include recurrent-layer count, recurrent units, dropout rates, learning rate, batch size, and training epochs.

The paper names both RS and GWO and reports that GWO performed better across the LSTM, GRU, and SRU families. However, it does not specify the search bounds, probability distributions, number of trials, wolf population, iteration count, initialization, update settings, invalid-candidate policy, or randomization controls. The generated package therefore treats these as explicit implementation configuration. No displayed setting in this section should be read as the paper's missing experimental setting.

One objective shared by both methods

The outer optimizer does not train a model directly. Instead, it proposes a typed CandidateConfig. The shared candidate objective builds the corresponding compositional model, creates or receives chronological training and validation loaders, trains the candidate, and records its best validation loss. Validation loss is used here as a local implementation choice because the paper discusses validation performance but does not define the loss or the exact selection rule.

CandidateResult stores the candidate configuration, objective, metrics, provenance, success status, and any failure information. SearchResult stores the selected result and the complete search history. This separation matters: a failed candidate remains visible in the history rather than being silently treated as the best candidate, and the held-out test set is not needed for hyperparameter selection.

The objective's public protocol is deliberately small. Its candidate has type CandidateConfig, its data bundle must provide training and validation loaders, and its seed is recorded for reproducibility. The generated implementation explicitly records that test data were not used for selection:

provenance: dict[str, Any] = {
    "seed": seed,
    "objective": "minimum_validation_loss",
    "selection_basis": "lowest validation loss recorded during training",
    "test_data_used_for_selection": False,
}

This excerpt comes from src/compositional_rnn_stock/search/objectives.py. The objective returns a CandidateResult after train_one_candidate supplies a TrainingHistory; the history must contain a validation loss from which the lowest recorded value can be selected. The code does not claim that this objective is the paper's exact loss or checkpoint policy.

The same selection helper serves both optimizers. Its minimize argument makes the direction explicit, which is important because validation loss is normally minimized, whereas some metrics are maximized:

def select_best_candidate(
    results: Sequence[CandidateResult],
    minimize: bool,
) -> CandidateResult:
    """Select a successful candidate using an explicit objective direction."""
    if not isinstance(minimize, bool):
        raise TypeError(f"minimize must be boolean; received {type(minimize).__name__}")
    if not isinstance(results, Sequence):
        raise TypeError("results must be a sequence of CandidateResult instances")

    validated: list[CandidateResult] = []
    for index, result in enumerate(results):
        if not isinstance(result, CandidateResult):
            raise TypeError(
                f"results[{index}] must be a CandidateResult; "
                f"received {type(result).__name__}"
            )
        if result.success:
            if result.objective is None:
                raise ValueError(
                    f"successful results[{index}] must contain an objective"
                )
            validated.append(result)

    if not validated:
        failure_count = sum(1 for result in results if not result.success)
        raise ValueError(
            "cannot select a best candidate: no successful candidate results "
            f"were available ({failure_count} recorded failures)"
        )

    if minimize:
        return min(validated, key=lambda result: float(result.objective))
    return max(validated, key=lambda result: float(result.objective))

The function accepts a sequence of results, rejects malformed successful records, ignores unsuccessful records for selection, and raises an error when no successful candidate exists. That failure behavior is a derived implementation safeguard, not a reported paper procedure.

Typed search spaces

A search space must represent different kinds of values. Recurrent-layer counts, units, batch sizes, and epochs are integer-valued. Dropout rates and learning rates are continuous. A model family is categorical when one search spans LSTM, GRU, and SRU. ParameterDomain gives each parameter one of these meanings and validates its bounds or allowed values.

The following excerpt is copied from src/compositional_rnn_stock/search/space.py and shows the domain constructors exposed by the generated package:

    @classmethod
    def continuous(cls, lower: float, upper: float) -> "ParameterDomain":
        return cls("continuous", lower=lower, upper=upper)

    @classmethod
    def integer(cls, lower: int, upper: int) -> "ParameterDomain":
        return cls("integer", lower=lower, upper=upper)

    @classmethod
    def categorical(cls, values: Sequence[Any]) -> "ParameterDomain":
        return cls("categorical", values=tuple(values))

    @classmethod
    def discrete(cls, values: Sequence[Any]) -> "ParameterDomain":
        return cls("discrete", values=tuple(values))

continuous and integer domains use inclusive lower and upper bounds. categorical and discrete domains hold explicit values. The distinction between categorical and discrete values is useful when documenting intent, although both are represented by positions when a vector-based optimizer needs numeric coordinates.

SearchSpace keeps the named domains in a stable order and exposes encoded bounds. CandidateConfig then groups the decoded values into ModelConfig, TrainingConfig, and any explicitly declared extra parameters. candidate_from_mapping validates a named mapping before constructing that typed record. The paper's list of optimized hyperparameters tells us what should be represented, but not the actual ranges; those ranges must be supplied by the caller.

Random Search: independent proposals

RS draws one value from every domain for each trial. The generated random_search function accepts a SearchSpace, a callable CandidateObjective, and a search configuration containing a seed, trial count, data bundle, and objective direction. Each trial receives a derived candidate seed, and each result is appended to the history, including failures.

The core loop is implemented as follows in src/compositional_rnn_stock/search/random_search.py:

    history: list[CandidateResult] = []
    for trial_index in range(trial_count):
        candidate = sample_random_candidate(space, rng)
        candidate_seed = seed + trial_index
        try:
            result = objective(candidate, data_bundle, candidate_seed)
            if not isinstance(result, CandidateResult):
                raise TypeError(
                    "objective must return CandidateResult; "
                    f"received {type(result).__name__}"
                )
        except Exception as exc:
            result = CandidateResult(
                configuration=candidate,
                objective=None,
                metrics={},
                provenance={
                    "method": "random_search",
                    "trial_index": trial_index,
                    "seed": candidate_seed,
                    "failure_type": type(exc).__name__,
                },
                success=False,
                error=f"{type(exc).__name__}: {exc}",
            )
        else:
            result.provenance.update(
                {
                    "method": "random_search",
                    "trial_index": trial_index,
                    "sampler_seed": seed,
                    "candidate_seed": candidate_seed,
                }
            )
        history.append(result)

Notice the two seeds recorded in successful results: the sampler seed identifies the sequence of sampled candidates, while the candidate seed identifies the training run. This is a useful provenance design for comparing methods locally. It does not establish that the paper used the same seeding scheme.

A tutorial experiment might declare three trials for a quick demonstration, but that would be a tutorial budget only. The paper's statement that 54 configurations were evaluated must not be reverse-engineered into a presumed RS trial count or into a presumed division between model families and optimizers.

GWO: population-guided proposals

GWO works with a matrix of wolf positions. A position is a numeric vector whose coordinates correspond to the ordered parameters in SearchSpace. The best three successful candidates are called alpha, beta, and delta in the generated implementation. Their encoded vectors guide a new population. After each update, positions are clipped to the declared bounds and decoded back into valid typed candidates.

The generated update function accepts positions with shape population × encoded_dimension. Each leader vector has shape encoded_dimension. The update is intentionally documented as a local GWO-style choice because no canonical GWO equations or settings were supplied in the paper:

def update_wolf_positions(
    positions: Any,
    alpha: Any,
    beta: Any,
    delta: Any,
    iteration: int,
    total_iterations: int,
    rng: random.Random,
) -> np.ndarray:
    """Return one local GWO-style update for every wolf.

    ``positions`` has shape ``(population, encoded_dimension)`` and each leader
    has shape ``(encoded_dimension,)``.  The implementation uses the usual
    linearly decreasing exploration coefficient and independent random draws.
    These update rules are implementation decisions, not equations supplied by
    the paper.
    """
    current = np.asarray(positions, dtype=float)
    leader_arrays = [np.asarray(value, dtype=float) for value in (alpha, beta, delta)]
    if current.ndim != 2:
        raise ValueError("positions must have shape (population, encoded_dimension)")
    if current.shape[0] < 1:
        raise ValueError("positions must contain at least one wolf")

The remainder of the function validates leader shapes and iteration bounds, generates independent random coefficients, averages the three leader-guided proposals, and returns an array with the same population-by-dimension shape. The linearly decreasing exploration coefficient, random update details, and alpha/beta/delta ranking are implementation decisions rather than recovered paper settings.

Decoding mixed parameters safely

A continuous vector cannot directly be used as a batch size or a model-family name. CandidateCodec bridges that gap. It encodes typed values into a stable vector ordering and decodes them by clipping to the domain, rounding integer-like coordinates, selecting categorical positions, and constructing validated ModelConfig and TrainingConfig records.

The public decoding contract is concise:

    def decode(self, vector: Sequence[float] | np.ndarray) -> CandidateConfig:
        array = np.asarray(vector, dtype=float)
        if array.ndim != 1 or array.shape[0] != self.dimension:
            raise ValueError(
                f"encoded candidate must have shape ({self.dimension},); received {array.shape}"
            )
        if not np.all(np.isfinite(array)):
            raise ValueError("encoded candidate must contain only finite values")
        values = {
            name: self.space.domains[name].decode_value(float(array[index]))
            for index, name in enumerate(self.space.parameter_names)
        }
        return self._build_candidate(self.space.validate_candidate(values))

For example, a vector coordinate intended for batch sizes (8, 16, 32) is clipped to the valid index range and rounded to one of those positions. That prevents GWO from producing an invalid fractional batch size. It also means that several nearby continuous positions can decode to the same typed candidate; this is an expected consequence of using a continuous optimizer for mixed parameters.

Worked configuration example

The following excerpt is a small configuration-only example based on the generated search-contract test. It demonstrates integer, continuous, and discrete domains without asserting that these bounds came from the paper:

PARAMETER_DOMAINS = {
    "stream_layers": ParameterDomain.integer(1, 3),
    "units": ParameterDomain.integer(8, 32),
    "dropout": ParameterDomain.continuous(0.0, 0.5),
    "learning_rate": ParameterDomain.continuous(1.0e-4, 1.0e-2),
    "batch_size": ParameterDomain.discrete((8, 16, 32)),
    "epochs": ParameterDomain.integer(1, 4),
}

space = SearchSpace(domains=PARAMETER_DOMAINS)
codec = CandidateCodec(space)

Here, stream_layers, units, batch_size, and epochs are discrete choices, while dropout and learning_rate vary continuously. codec can translate a validated CandidateConfig into a numeric vector for GWO and decode a clipped vector back into a typed candidate. The displayed ranges are deliberately modest tutorial settings; they are not paper-reported bounds.

In a complete local run, the same space and shared objective can be passed to RS or GWO. The pipeline functions run_random_search and run_gwo_search bind both methods to a chronological training/validation split created from the prepared training partition. Their documented contract excludes PreparedData.test from selection, so the held-out test data are reserved for evaluation after a candidate has been chosen.

Practical limits of the search reproduction

A fair local comparison requires RS and GWO to use the same feature order, scaling policy, candidate-training routine, validation partition, objective, and failure policy. Only the proposal mechanism should differ. The generated pipeline records the validation fraction as a local choice and shares the candidate objective, but this does not recover the paper's undisclosed validation protocol.

The paper's GWO superiority claim remains a reported paper result, not a result established by this scaffold. In particular, the generated GWO update, population size, iteration count, bounds, initialization, and discrete decoding cannot be labeled as the paper's implementation. The four-part labels in LSTM-GWO (1-1-0-1), GRU-GWO (2-1-1-1), and SRU-GWO (2-2-0-0) also remain opaque; the search code does not decode them.

The related contract tests are intended to check local invariants such as candidate decoding, seeded RS sampling, GWO shape and bound handling, and the common objective interface. They were not executed in this run. Static verification and semantic code verification were also skipped, so the generated files should be reviewed before relying on them operationally. No search, training run, or reproduction result is claimed here.

Held-out evaluation, metrics, and residual analysis

How do we know whether a forecast is useful? First, each prediction must be paired with the correct held-out observation. Then both arrays must be evaluated under the same scale and metric conventions. The evaluation layer therefore aligns predictions and targets, optionally restores original price units, computes the seven metrics named by the paper, and summarizes residual behavior.

This section distinguishes three kinds of statements. The paper names the metrics and discusses error distributions, but it does not provide canonical formulas, denominator rules, residual signs, or plotting conventions. The generated implementation supplies explicit conventional choices for those missing details. The resulting values would be local evaluation outputs—not verified reproductions of the paper's reported results.

From model output to aligned evaluation arrays

The generated predict_dataset function runs inference in loader order. It switches the model to evaluation mode so dropout is disabled, uses no-gradient inference, collects scalar predictions and targets batch by batch, and restores the model's previous training-mode flag afterward. Its scalar-target contract accepts arrays shaped samples or samples × 1. It does not reorder or silently truncate samples.

The key alignment operation is exposed separately as align_predictions_and_targets. It rejects unequal sample counts and preserves the order emitted by the loader. This is why held-out loaders should normally use shuffle=False: a shuffled loader can still produce pairs within a batch, but it no longer represents the original chronological output order.

The following excerpt is copied from predict_dataset. Notice the two independent checks: the model output and the batch targets are normalized to the scalar-target representation, and their batch counts must agree before either is appended.

                outputs = model(inputs)
                if not isinstance(outputs, Tensor):
                    raise TypeError(
                        f"model output for batch {batch_index} must be a tensor"
                    )
                predictions, _ = _normalise_scalar_targets(outputs, "predictions")
                batch_targets, _ = _normalise_scalar_targets(targets, "targets")
                if predictions.shape[0] != batch_targets.shape[0]:
                    raise ValueError(
                        f"batch {batch_index} prediction and target counts differ: "
                        f"{predictions.shape[0]} != {batch_targets.shape[0]}"
                    )
                prediction_parts.append(predictions)
                target_parts.append(batch_targets)

The function returns predictions in loader order, while the targets are retained internally for alignment validation. The model input remains the package-wide batch × time × 5 contract; this evaluation helper specifically supports the configured scalar-output case. A multivariate target would require an explicit extension rather than an implicit interpretation.

Choosing the evaluation scale

The paper states that inputs are scaled to [0, 0.95], but it does not say whether its reported metrics were computed in normalized space or after inverse transformation to price units. The generated pipeline exposes both choices through EvaluationConfig.

In normalized evaluation, actual_targets and predictions remain in scaled space. RMSE then has normalized units. In original-unit evaluation, inverse_transform_target embeds each scalar target into a five-feature row, applies FeaturewiseMinMaxScaler.inverse_transform, and extracts the configured target column. RMSE can then be interpreted in price units. MAPE, RMSPE, and PBIAS remain percentage-valued under the local conventions, while agreement metrics remain unitless.

The target column itself is also unresolved by the paper. The generated pipeline requires a configured choice such as Close; it does not assume that the paper's phrase “stock price” uniquely means Close or adjusted Close. The forecast horizon is likewise recorded separately from the input window_length.

The following excerpt is copied from inverse_transform_target and shows why the scaler needs a target index even though the prediction contains only one value per sample.

    embedded = np.zeros((flat_values.shape[0], scaler.n_features), dtype=np.float64)
    embedded[:, target_index] = flat_values
    restored = scaler.inverse_transform(embedded)[:, target_index]

    if len(original_shape) == 1:
        return restored
    return restored.reshape(original_shape)

This operation preserves the prediction shape while using the scaler's five feature-specific statistics. It is an implementation mapping, not a formula recovered from the paper. The scaler must already be fitted, and its fitting policy is recorded as training-only in the generated preprocessing pipeline to reduce leakage.

The seven named metrics

The paper names R², RMSE, MAPE, RMSPE, PBIAS, Willmott Index, and NSE. Because no canonical equation records were supplied, the generated metrics.py module documents and implements conventional local definitions. The important practical point is consistency: every candidate must use the same scale, alignment rule, sign convention, and zero-denominator policy.

• R² summarizes explained variation relative to an actual-value mean baseline. The generated function stores the conventional result on the unit-interval scale by default. A display convention can multiply it by 100, which supports the paper's percentage-style reporting. Thus a paper display of 99.2427% is distinct from the underlying unit-interval representation 0.992427.

• RMSE measures the typical magnitude of prediction error in the units supplied to the function. It is therefore scale-sensitive: normalized inputs yield normalized-unit RMSE, while inverse-transformed prices yield price-unit RMSE.

• MAPE averages absolute relative errors and returns a percentage. It cannot divide by zero actual values, so EvaluationConfig selects whether such observations raise an error, are ignored, or receive a defined local treatment.

• RMSPE is the root-mean-square version of relative error and uses the same zero-actual policy.

• PBIAS summarizes signed aggregate bias. The generated default residual direction is actual_minus_predicted, so positive PBIAS indicates underprediction under that convention.

• Willmott Index measures agreement using the generated module's selected standard convention. Its denominator can be undefined for degenerate data, in which case the local implementation returns NaN.

• NSE, or Nash–Sutcliffe Efficiency, compares squared forecast error with variation around the actual-series mean. Like R², it is undefined when the actual series has no variation.

The module's report assembler keeps these choices together. The following excerpt is copied from compute_metric_report; it shows that one configuration controls the conventions used by the individual metric functions.

    _validate_zero_policy(str(zero_policy))
    return {
        "r_squared": r_squared(actual_vector, predicted_vector, str(r2_convention)),
        "rmse": rmse(actual_vector, predicted_vector),
        "mape": mape(actual_vector, predicted_vector, str(zero_policy)),
        "rmspe": rmspe(actual_vector, predicted_vector, str(zero_policy)),
        "pbias": pbias(actual_vector, predicted_vector, str(pbias_convention)),
        "willmott_index": willmott_index(
            actual_vector,
            predicted_vector,
            str(willmott_convention),
        ),
        "nash_sutcliffe_efficiency": nash_sutcliffe_efficiency(actual_vector, predicted_vector),
    }

Before reaching this block, _aligned_arrays converts supported array-like inputs to finite one-dimensional vectors and rejects mismatched shapes. This failure behavior matters: a metric calculated from mispaired samples can look numerically plausible while describing the wrong forecast errors.

A small illustrative interface example

The following excerpt is adapted directly from the generated metric test's synthetic setup. It demonstrates the intended interface with short local arrays; it is not HSI data, and no numerical result from this example should be compared with the paper's benchmarks. The test uses EvaluationConfig() to select the generated defaults rather than claiming that those defaults were specified by the paper.

    actual = np.array([100.0, 105.0, 110.0, 115.0], dtype=float)
    predicted = np.array([99.0, 106.0, 109.0, 116.0], dtype=float)

    # Defaults are deliberately supplied by EvaluationConfig rather than
    # inferred from the paper, whose metric conventions are unspecified.
    report = compute_metric_report(actual, predicted, EvaluationConfig())

A corresponding residual summary can be created from the same aligned arrays. In the generated implementation, compute_residuals(actual, predicted) returns actual - predicted with the same shape, and summarize_residuals flattens the result before calculating its count, mean, population standard deviation, standardized skewness, and excess kurtosis. Fewer than three observations, or zero variance, leave skewness undefined; fewer than four observations, or zero variance, leave excess kurtosis undefined. These estimator and insufficiency rules are implementation decisions because the paper only mentions mean, standard deviation, skewness, and kurtosis without defining their conventions.

Residuals and visual analysis

Residuals make systematic behavior easier to inspect than a single score. Under the generated actual_minus_predicted convention, positive residuals indicate underprediction and negative residuals indicate overprediction. compare_error_distributions creates a stable table for named residual groups, preserving empty groups with an explicit count and undefined summary values rather than inventing observations.

The visualization adapters implement the analysis types mentioned by the paper:

• plot_error_boxplots shows grouped residual spread and central tendency.

• plot_error_violins shows distribution shape.

• plot_metric_comparison gives each metric its own panel so percentage and unitless measures are not placed on one misleading axis.

• plot_pbias_radar displays signed PBIAS values using a locally selected radial convention.

• plot_taylor_summary uses a locally selected correlation-angle and standard-deviation representation.

These functions operate on local predictions and reports. They do not contain the paper's plotted points, styling, axes, or source predictions. Therefore they can support analogous analysis, but they cannot claim exact reproduction of the paper's figures.

Evaluate only after model selection

The evaluation pipeline enforces the intended experiment order conceptually: train candidates and compare them using the configured validation objective, select a candidate, and only then read the chronological held-out partition for final metrics. evaluate_training_run evaluates a retained trained model. evaluate_search_result requires that a selected trained model be retained; it does not silently retrain from an incomplete search record.

The resulting EvaluationRun stores predictions, aligned actual targets, residuals, the metric report, and provenance. Provenance includes the evaluation scale, target column, forecast horizon, window length, metric conventions, residual sign, held-out sample count, and preprocessing metadata. This makes a future comparison auditable: a difference in RMSE can be investigated as a possible scale, target, or convention difference rather than treated as unexplained model behavior.

The paper's seven metrics and its discussion of residual distributions are therefore represented, but not overclaimed. The formulas and conventions are local implementation choices, the target and evaluation scale remain configurable, and no execution or verification occurred in this run. Any future local report must be labeled as a computed result and kept separate from the paper's reported benchmark records.

Reported LSTM-GWO, GRU-GWO, and SRU-GWO benchmarks

How should you use the paper’s reported numbers when the original experiment cannot yet be reconstructed exactly? Treat them as reference records, not as expected outputs that automatically validate a new run. The generated package stores the reported values separately from locally computed predictions and metrics, so a later experiment can be compared with the paper without confusing the two.

What the paper reports

The paper identifies its strongest reported configuration as LSTM-GWO (1-1-0-1). The four-part label is opaque: the supplied paper context does not define whether its fields represent layer counts, dropout settings, dense layers, or another encoding. The label must therefore remain a benchmark identifier rather than being translated into ModelConfig values.

The reported LSTM-GWO metrics are:

• R²: 99.2427%

• RMSE: 339.3902

• MAPE: 1.1721%

• RMSPE: 1.6221%

• PBIAS: 0.0523

• Willmott Index: 0.9981

• NSE: 0.9924

The paper also reports the following best labels and associated values:

• GRU-GWO `(2-1-1-1)`: R² 99.2322%, RMSE 341.7225, MAPE 1.1821%, and PBIAS −0.1357.

• SRU-GWO `(2-2-0-0)`: R² 99.2009%, RMSE 348.6384, and MAPE 1.2080%.

The remaining GRU and SRU metric associations are not clear in the supplied extraction. They are intentionally left unavailable rather than inferred from nearby text or reconstructed tables. Likewise, the configuration labels remain undecoded.

These values also use mixed display conventions. R² is shown by the paper as a percentage, whereas Willmott Index and NSE are shown on a unit-interval scale. A local metrics implementation may store R² internally as 0.992427 and display it as 99.2427%, but that conversion must be recorded explicitly before comparing values.

How the generated code preserves the evidence

PaperBenchmark is an immutable record containing the recurrent family, optimizer, opaque configuration label, metric values, units, and provenance. Its metric schema always contains the seven named metrics, but unavailable values are represented by None. The following excerpt is copied from reported_benchmarks in src/compositional_rnn_stock/evaluation/benchmarks.py:

def reported_benchmarks() -> tuple[PaperBenchmark, ...]:
    """Return the three benchmark records supplied in the paper context.

    Missing GRU and SRU associations remain ``None`` rather than being inferred from
    other reported values. R² is stored in the paper's percentage display convention.
    """
    return (
        PaperBenchmark(
            family="LSTM",
            optimizer="GWO",
            configuration_label="1-1-0-1",
            metrics=_benchmark_metrics(
                r_squared=99.2427,
                rmse=339.3902,
                mape=1.1721,
                rmspe=1.6221,
                pbias=0.0523,
                willmott_index=0.9981,
                nash_sutcliffe_efficiency=0.9924,
            ),
            metric_units=dict(_METRIC_UNITS),
        ),

The function returns records marked as supplied paper targets. It does not train a model, generate predictions, or assert that any local candidate achieved these values. The LSTM record is complete, while the later GRU and SRU records retain None for unavailable associations.

The generated contract test makes this distinction explicit when it retrieves the records:

benchmarks = reported_benchmarks()

assert [(item.family, item.optimizer, item.configuration_label) for item in benchmarks] == [
    ("LSTM", "GWO", "1-1-0-1"),
    ("GRU", "GWO", "2-1-1-1"),
    ("SRU", "GWO", "2-2-0-0"),
]

This is a planned test file, not evidence of an executed check. Under the authoritative run policy, code execution and verification were disabled.

Comparing a future local report

The compare_to_report function accepts a local metric mapping and one PaperBenchmark. It produces rows only when both sides have a value. Differences are expressed as local minus reported in the benchmark’s displayed units. Thus a missing GRU RMSPE value does not become a guessed comparison row, and a local unit-interval R² can be converted to the paper’s percentage convention when the local report has not declared another unit.

A comparison is meaningful only when the upstream experiment is also aligned. The provenance should identify the target column, forecast horizon, scaling range and fitting policy, window length, chronological split, recurrent-family configuration, seed, training settings, search settings, evaluation scale, and metric conventions. Without those fields, a numerical difference cannot reveal whether the models or datasets were actually comparable.

The evaluation pipeline places these comparisons alongside a local EvaluationRun, whose predictions, aligned targets, residuals, metric report, and provenance are kept together. This connects benchmark comparison to the feature-wise compositional model: the local prediction must first come from the five independent OHLCV streams, concatenation fusion, and the selected recurrent and dense head. A benchmark difference cannot repair an architecture or target choice that the paper leaves unspecified.

Reported evidence versus local conclusions

The paper states that GWO outperformed Random Search across the evaluated LSTM, GRU, and SRU families. That statement is reported evidence from the paper. It is not a result established by this unexecuted scaffold. A local RS-versus-GWO comparison would require the same preprocessing, validation objective, candidate space, and declared search budget for both methods, followed by actual execution and evaluation.

The safe interpretation is therefore:

1. reported_benchmarks() preserves the supplied reference records.

2. compare_to_report provides a labeled comparison mechanism for future local outputs.

3. Missing metrics and opaque configuration labels remain unresolved.

4. No local metric is claimed to match the paper.

The benchmark records are useful precisely because they retain their provenance and limitations. They provide targets for a future, better-specified reproduction rather than proof that the current configurable implementation reproduces the published experiment.

Offline synthetic demonstration and artifact provenance

How can you learn the repository’s data and model interfaces without downloading market data? Use the offline demonstration. It creates clearly labeled synthetic OHLCV rows, applies the same five-channel scaling and windowing contracts, builds a configurable compositional model, and inspects the expected tensor shapes. This is a plumbing demonstration only: the synthetic series is not Hang Seng Index data and cannot validate forecasting performance.

Why use an offline demonstration?

The paper describes daily Hang Seng Index history obtained from Yahoo Finance, but the supplied context does not identify a unique ticker, date range, adjustment policy, or download configuration. Yahoo Finance data can also change as providers revise historical records. An offline example avoids those dependencies while making every local choice visible.

The generated example uses a deterministic pseudo-random generator when given a selected seed. It creates five columns in the canonical order Open, High, Low, Close, Volume. The values are synthetic prices and volumes; they are not intended to reproduce the empirical distribution of the HSI.

The example’s generator is defined in examples/offline_synthetic_demo.py:

def make_synthetic_ohlcv(num_days: int, seed: int) -> pd.DataFrame:
    """Create deterministic, clearly labeled synthetic OHLCV observations.

    The returned table has shape ``(num_days, 5)`` and canonical column order:
    Open, High, Low, Close, Volume. Values are generated only for this
    demonstration; they are not intended to model the empirical HSI series.
    """
    if isinstance(num_days, bool) or not isinstance(num_days, int) or num_days <= 0:
        raise ValueError("num_days must be a positive integer")
    if isinstance(seed, bool) or not isinstance(seed, int) or seed < 0:
        raise ValueError("seed must be a non-negative integer")

    rng = np.random.default_rng(seed)
    daily_returns = rng.normal(loc=0.0004, scale=0.012, size=num_days)
    close = 18_000.0 * np.exp(np.cumsum(daily_returns))
    open_noise = rng.normal(loc=0.0, scale=35.0, size=num_days)
    open_price = close + open_noise
    intraday_spread = np.abs(rng.normal(loc=55.0, scale=18.0, size=num_days))
    high = np.maximum(open_price, close) + intraday_spread
    low = np.minimum(open_price, close) - intraday_spread
    volume = np.maximum(
        1.0,
        1_000_000.0 + rng.normal(loc=0.0, scale=120_000.0, size=num_days),
    )

    values = np.column_stack((open_price, high, low, close, volume))
    return pd.DataFrame(
        values,
        index=pd.date_range("2020-01-01", periods=num_days, freq="D"),
        columns=list(CANONICAL_OHLCV_COLUMNS),
    )

The function accepts a positive number of rows and a non-negative integer seed. Its output is a table with shape num_days × 5. The validation errors are useful failure cases: a boolean is not accepted as an integer, and invalid row counts or seeds are rejected rather than silently corrected.

Scaling an earlier chronological portion

The paper states the intended feature range [0, 0.95], but it does not specify exactly which observations fit the scaling statistics. The demonstration chooses an earlier chronological portion for fitting. This is a leakage-prevention decision made by the implementation, not a recovered paper detail. The remaining observations are transformed with the already fitted scaler.

The relevant portion of main is:

data_config = DataConfig(
    target_column="Close",
    forecast_horizon=1,
    window_length=20,
    split_ratio=0.8,
    scaling_range=(0.0, 0.95),
)
raw_frame = make_synthetic_ohlcv(num_days=96, seed=7)
raw_values = raw_frame.loc[:, list(CANONICAL_OHLCV_COLUMNS)].to_numpy(
    dtype=np.float64
)

# Fit on an earlier chronological portion to demonstrate a leakage-avoiding
# choice. The paper states the [0, 0.95] range but does not specify fit scope.
scaler_split = int(raw_values.shape[0] * data_config.split_ratio)
scaler = FeaturewiseMinMaxScaler(feature_range=data_config.scaling_range)
scaler.fit(raw_values[:scaler_split])
scaled_values = scaler.transform(raw_values)

Here, target_column='Close', forecast_horizon=1, and window_length=20 are tutorial configuration choices. The paper does not identify the target column or forecast horizon; it only describes 20-day and 40-day input windows. Similarly, the demonstration’s 96 synthetic rows and seed 7 are not paper data.

FeaturewiseMinMaxScaler requires a final feature axis of length five. It stores separate scaling metadata for each OHLCV channel, so Volume does not determine the scaling of the price columns. Its inverse_transform method is available when later evaluation should return predictions to original price units. The example does not calculate or present forecast metrics.

Building windows and inspecting model shapes

After scaling, the example selects the configured target column and constructs overlapping supervised windows:

target_index = list(CANONICAL_OHLCV_COLUMNS).index(data_config.target_column)
windowed = make_sliding_windows(
    scaled_values,
    target_index=target_index,
    window_length=data_config.window_length,
    forecast_horizon=data_config.forecast_horizon,
    timestamps=raw_frame.index.to_numpy(),
)

# The model follows the paper's feature-wise structure: five independent
# recurrent streams, concatenation fusion, and a one-value prediction head.
model_config = ModelConfig(
    recurrent_family="LSTM",
    stream_layers=1,
    stream_units=16,
    stream_dropout=0.0,
    fusion_dropout=0.0,
    use_post_fusion_recurrent=False,
    dense_layers=(16,),
    output_dimension=1,
)
model = build_compositional_model(model_config)
model.eval()

batch_size = min(4, windowed.sample_count)
batch_inputs = torch.as_tensor(windowed.inputs[:batch_size], dtype=torch.float32)
with torch.no_grad():
    predictions = model(batch_inputs)

make_sliding_windows returns inputs with shape samples × window_length × 5 and scalar targets with shape samples. With this tutorial configuration, each input contains 20 time steps and five channels. The target is the selected Close value one step after the input window according to the implementation’s explicit alignment convention.

build_compositional_model then constructs the feature-wise recurrent model. Each batch input has shape batch × 20 × 5; the model separates the five channels, encodes them independently, concatenates their representations, and returns a prediction with shape batch × output_dimension. The model configuration shown here is an educational choice. It does not decode the paper’s opaque benchmark label LSTM-GWO (1-1-0-1), nor does it claim to recover the paper’s exact layer widths or training recipe.

The source example prints these contracts:

print("Synthetic demonstration only; no paper result is being reproduced.")
print(f"raw OHLCV shape: {raw_values.shape}")
print(f"scaled OHLCV range: {data_config.scaling_range}")
print(
    "window input shape (samples, time_steps, features): "
    f"{windowed.inputs.shape}"
)
print(f"window target shape: {windowed.targets.shape}")
print(
    "model batch input shape (batch, time_steps, features): "
    f"{tuple(batch_inputs.shape)}"
)
print(f"model prediction shape (batch, output_dimension): {tuple(predictions.shape)}")

These statements document intended dimensions; this run did not execute the example. In particular, no output numbers, predictions, or paper-comparison result should be inferred from the excerpt.

Recording provenance and arrays locally

A demonstration becomes more useful when its assumptions travel with its outputs. Artifact provenance should identify at least the data source, cleaning policy, scaling range and fit scope, window length, target column, forecast horizon, model family, architecture settings, seed, and evaluation conventions. For real HSI work, the archived raw file and the Yahoo Finance query parameters should also be retained.

The generated artifact module provides local persistence through save_json_record, save_array, and load_json_record. The README illustrates the JSON-record call as follows:

save_json_record(report, Path('artifacts/report.json'))

A JSON record is appropriate for serializable configuration, acquisition metadata, histories, benchmark labels, and metric reports. Numeric predictions and residuals should be stored with save_array, which preserves their numeric shape and dtype. load_json_record reads a previously saved top-level JSON object. The artifact implementation uses local filesystem operations and rejects implicit overwrites, helping prevent an old experiment from being silently replaced.

For example, a future evaluation record could contain a local run identifier, window_length, target_column, forecast_horizon, scaler settings, model configuration, seed, metric conventions, and a separate reference to prediction and residual arrays. It should distinguish computed local values from the paper’s reported benchmark records. The generated interfaces provide the storage operations, while the exact record contents remain an experiment-provenance responsibility.

What this demonstration does not establish

The synthetic path confirms the intended software contracts conceptually: five ordered channels can be scaled, transformed into windows, and supplied to a feature-wise compositional model. It does not establish that the model forecasts the HSI accurately, that the chosen target and horizon match the paper, or that the reported LSTM-GWO, GRU-GWO, or SRU-GWO values can be reproduced.

No example execution, artifact creation, test execution, or verification occurred under the run policy. A stronger reproduction would archive the original Yahoo Finance data query and downloaded rows, supply the missing architecture and optimization details, run the configured experiment, and then evaluate predictions under explicitly documented metric conventions. Until then, the offline example is a reproducibility aid—not evidence of paper-level performance.

Verification status, limitations, and responsible reproduction claims

How can a carefully organized implementation be useful without overstating what it proves? The answer is to separate invariants, planned checks, and verified results. The generated package records the paper’s intended data flow, model structure, search interfaces, evaluation conventions, and provenance. However, this run did not execute code or perform verification. Its correct status is therefore verification_skipped: a configurable reproduction scaffold with explicit gaps, not a validated reproduction of the published experiment.

What the implementation is designed to preserve

Several structural requirements are clear enough to review independently of the paper’s missing experimental details:

• The input feature order is exactly Open, High, Low, Close, Volume.

• Model inputs use batch × time × 5 ordering.

• Preprocessing constructs chronological windows and keeps training samples before later samples.

• The scaler targets the stated interval [0, 0.95]. Fitting it on earlier training observations only is a leakage-prevention implementation choice; the paper does not specify its fitting partition.

• The model creates exactly five independent feature streams.

• Stream representations are fused by concatenation, rather than by early summation or mixing.

• Dropout is intended for training mode and should be inactive during evaluation.

• Predictions and targets must remain sample-aligned before metric calculation.

• Search candidates must decode to valid typed values for layer counts, units, dropout rates, learning rates, batch sizes, and epochs.

• RS and GWO should use the same candidate objective, preprocessing, and validation protocol when compared locally.

• Benchmark labels such as LSTM-GWO (1-1-0-1) remain opaque identifiers and are not translated into architecture settings.

These are implementation contracts and interpretation safeguards. They are not evidence that a particular model reached the paper’s reported metrics.

Planned contract tests are not completed checks

The repository contains generated test files intended to make these invariants reviewable later. For data handling, test_scaler_range_and_inverse_round_trip, test_window_target_alignment, and test_chronological_split_has_no_reordering address scaling behavior, window-to-target indexing, and chronological partitioning. The model tests include test_model_accepts_five_feature_input, test_fusion_dimension_is_additive, test_recurrent_family_dispatch_is_consistent, and test_dropout_differs_only_in_training_mode.

The remaining planned tests cover local metric conventions, candidate encoding, search bounds, and provenance. Examples include test_rmse_zero_for_equal_arrays, test_percentage_metric_zero_policy, test_candidate_codec_validates_discrete_parameters, test_gwo_positions_stay_in_bounds_after_decoding, test_reported_benchmarks_preserve_supplied_values, and test_incomplete_metrics_remain_missing.

These names describe intended checks, not completed checks. Under the authoritative run policy, test generation and execution were disabled. No test result, passing assertion, successful import, or model output should be inferred from the presence of these files.

Static and semantic verification status

Static verification and semantic verification address different questions. Static review would inspect syntax, imports, public interfaces, tensor-shape contracts, and configuration propagation without running the experiment. Semantic review would assess whether the implementation’s behavior matches the intended method, including feature independence, concatenation, dropout mode changes, search-objective consistency, and metric conventions.

Both forms of verification were skipped here. Local static verification reports the status verification_skipped, and semantic code verification was also skipped. Code execution, test execution, tutorial-section verification, and final quality review were disabled by policy. Consequently, this section must not claim that the generated code is correct, that the tests pass, or that any local result matches the paper.

Why exact reproduction remains underdetermined

The supplied paper context does not uniquely determine several decisions that materially affect results:

• The Yahoo Finance ticker or symbol, date range, download options, adjustment policy, missing-value handling, and duplicate-row handling.

• The forecast target and horizon. The paper names “stock price” and lists OHLCV inputs, but does not identify whether the target is Close, adjusted Close, another price, or a multivariate output.

• The exact interpretation of the 20-day and 40-day windows beyond treating them as input lengths.

• The normalization formula, fitting partition, clipping behavior, and inverse-transformation procedure.

• Recurrent layer counts, widths, activations, hidden-state extraction, dense-layer structure, and exact placement of both dropout stages.

• The SRU variant and Python implementation or API.

• The training loss, gradient optimizer, initialization, learning-rate schedule, seed, shuffling policy, stopping rule, and checkpoint-selection rule.

• RS search domains, distributions, trial budget, and validation protocol.

• GWO population size, initialization, bounds, iteration count, update schedule, objective, and encoding of discrete parameters.

• Metric formulas, percentage conventions, zero-denominator handling, bias signs, and the scale used for evaluation.

The paper’s reported total of 54 configurations does not resolve these omissions. Nor can the opaque four-part labels in the LSTM-GWO, GRU-GWO, and SRU-GWO records be safely decoded from the supplied evidence.

A responsible review checklist

Before treating a future local run as evidence, review the following items and record each one in the experiment provenance:

1. Data shape: confirm that the prepared inputs use batch × time × 5 and the canonical OHLCV order.

2. Chronology: confirm that the split and window-target alignment preserve temporal order and do not use future observations during fitting.

3. Scaling: record the [0, 0.95] range, the fitting partition, constant-feature policy, and any inverse transformation.

4. Target semantics: record the target column and forecast horizon instead of assuming that the paper specifies them.

5. Model structure: record the five independent encoders, recurrent family, representation choice, dropout placement, fusion operation, post-fusion layers, and output dimension.

6. Training: record the loss, gradient optimizer, learning rate, batch size, epochs, seed, validation protocol, and checkpoint policy.

7. Search: record the typed domains, objective direction, trial or wolf budget, bounds, discrete decoding, and failure handling.

8. Evaluation: record whether metrics use normalized or original units, every metric convention, percentage display rules, and zero handling.

9. Benchmarks: preserve reported values separately from local values, keep missing GRU and SRU associations missing, and leave configuration labels opaque.

10. Verification: record whether code and tests were actually executed and reviewed; do not substitute planned test names for evidence.