Overview

CodeFrog is a powerful testing and analysis tool, but like all automated tools, it has inherent limitations. This page documents what CodeFrog cannot detect so you know when to supplement with other tools or manual review.

Accessibility Testing

WCAG Rating Requires Manual Testing

CodeFrog’s accessibility tests use the axe-core engine to perform automated WCAG checks. However, automated testing typically catches only 20-30% of issues in real-world audits (up to ~57% in controlled lab tests). WCAG conformance ratings (A, AA, or AAA) cannot be determined by automated tools alone. To achieve a true WCAG conformance rating, you must also perform:

• Manual keyboard testing — Can all interactive elements be reached and operated with keyboard alone?
• Screen reader testing — Does the page make sense when read aloud by VoiceOver, NVDA, or JAWS?
• Cognitive review — Is the content clear, consistent, and predictable for users with cognitive disabilities?
• Color and contrast review — Are there color-only indicators that need additional cues?
• Media review — Do videos have captions? Do audio files have transcripts?
• Zoom and reflow testing — Does content remain usable at 200% zoom and on narrow viewports?

Important: A passing automated accessibility score does not mean your site is WCAG compliant. It means no automated violations were found. Many critical accessibility issues (like poor tab order, missing alt text context, or confusing navigation) require human judgment to detect.

See Accessibility Best Practices and WCAG Levels for guidance on manual testing.

Console Errors Test

The Console Errors test loads pages in the platform’s native headless WebView (WKWebView on macOS and iOS, WebView2 on Windows, Android WebView on Android) and captures JavaScript errors, warnings, and network failures. Due to differences between WebView engines and full browsers like Chrome, some categories of errors cannot be detected:

Chrome-Internal Permissions Policy Violations

Chrome 115+ internally deprecated the unload event and progressively blocks pages from using addEventListener('unload', ...). When Chrome blocks this, it shows:

[Violation] Permissions policy violation: unload is not allowed in this document.

This violation is generated entirely within Chrome’s engine — there is no HTTP header, no JavaScript event, and no DOM API that exposes it. No WebView engine (WKWebView, WebView2, Android WebView) reports these violations, so CodeFrog cannot detect them.

Workaround: Check the Chrome DevTools Console directly for permissions policy violations.

Authenticated API Failures

The headless WebView has no user session, cookies, or login state. API calls that require authentication (e.g., graph.microsoft.com/v1.0/me/photos returning 404 because no user is logged in) will either not be made at all or will fail differently than in a logged-in browser session.

Workaround: Test authenticated flows in Chrome DevTools while logged into the site.

Cross-Origin Opaque Responses

When JavaScript makes a fetch() or XMLHttpRequest to a cross-origin server, and that server doesn’t include CORS headers on error responses (e.g., returns 404 without Access-Control-Allow-Origin), the browser treats the response as “opaque” — the HTTP status code is hidden (appears as 0). CodeFrog’s interceptor cannot read the actual status code in these cases.

Cached Function References

Some JavaScript bundles save references to window.fetch or EventTarget.prototype.addEventListener at module initialization time, before CodeFrog’s interceptor script runs. Requests made through these cached references bypass our monitoring.

Code Analysis Tests (Desktop Only)

The following tests require command-line tools installed on the local machine and are only available on macOS, Windows, and Linux — not on iOS or Android:

• Secrets Detection (Gitleaks) — Requires gitleaks CLI
• Supply Chain Scanning (OSV) — Requires osv-scanner CLI
• Static Analysis (Semgrep/OpenGrep) — Requires semgrep or opengrep CLI

If these tools are not installed, the tests are skipped with a message indicating the tool is unavailable.

Security Scanning

Security scanning in the Mega Report performs automated checks only. It does not replace:

• Penetration testing — Active exploitation testing by security professionals
• Security code review — Manual review of authentication, authorization, and data handling logic
• Dynamic application security testing (DAST) — Full crawl-and-attack testing tools like OWASP ZAP or Burp Suite

Note: Security scanning requires explicit authorization for the target. CodeFrog will prompt you to confirm authorization before running security tests. See Security Scanning Policy for details.

General Limitations

Single-Page Snapshot

Most web tests analyze a single page load at a specific point in time. They do not:

• Test user interactions (clicking buttons, filling forms, navigating between pages)
• Monitor pages over time for intermittent issues
• Test pages behind authentication (login walls)
• Execute tests in multiple browser engines simultaneously

Network-Dependent Results

Test results depend on network conditions at the time of testing. Results may vary between runs due to:

• CDN caching behavior
• Server response times
• Third-party service availability
• Geographic location of the testing device

Recommendations

For comprehensive quality assurance, combine CodeFrog with:

• Chrome DevTools — For console errors, network analysis, and performance profiling
• Screen readers (VoiceOver, NVDA, JAWS) — For manual accessibility testing
• Pa11y CI — For automated accessibility regression testing in CI pipelines
• OWASP ZAP or Burp Suite — For dynamic security testing

See Recommended Tools for more details.