tick-collection-ops

MT5 Tick Collection Operations

Self-Evolving Skill: This skill improves through use. If instructions are wrong, parameters drifted, or a workaround was needed — fix this file immediately, don't defer. Only update for real, reproducible issues.

⚠️ HOST SUPERSEDED — 2026-08-07

MetaTrader 5 no longer runs on bigblack. It runs exclusively on the FXView broker VPS (Tailscale host fxview-mt5, 100.126.202.32, Windows), on demo account 400364. The Mac-local Wine bench was deleted 2026-08-06; bigblack's Wine terminal + its xvfb/xfce/x11vnc desktop and fxview-sidecar are being decommissioned.

On the VPS the topology is Windows scheduled tasks, not Linux systemd:

Concern	bigblack (retired)	FXView VPS (current)
Terminal	mt5.service under Wine, DISPLAY=:99	terminal64.exe, native Windows, real session
Tick rings	/dev/shm/tick_ring_<SYM>	C:\odb-tickrings\tick_ring_<SYM>
Ring → bus	—	odb-tapegateway.exe → task OdbTickRingToNatsTapeGateway
Bus	—	nats-server.exe :4222 → task OdbTapeBusNatsJetStreamServer
Orders	—	order-service :50060 → task ODB-MT5-OrderService
Symbols	EUR/GBP/XAU/XAG	EUR/GBP/XAU only (silver excluded per directive 2026-05-26)
Service control	systemctl	Get-ScheduledTask / Start-ScheduledTask

Reach it with ssh fxview-mt5. PowerShell over SSH eats $_ under shell quoting — pass commands base64-encoded via powershell -NoProfile -EncodedCommand <base64-utf16le>.

Everything below still describes the Linux/Wine deployment. It is retained because the EA, the tick_writer.dll ring format and the gap-detection reasoning are unchanged and still authoritative — only the HOST and the service manager moved. Do not use it to stand up a new Linux MT5.

Canonical: opendeviationbar-patterns/docs/adr/2026-08-07-retire-bigblack-as-a-metatrader-host-…md and docs/decision-register-metatrader-hosting-and-market-data-source-provenance.md.

Operate, monitor, and troubleshoot the zero-gap tick collection system running on Linux via Wine.

When to Use This Skill

Use this skill when:

• Deploying or restarting tick collection on bigblack
• Diagnosing tick gaps or missing data
• Troubleshooting Wine or MT5 issues
• Understanding systemd service topology
• Investigating crash recovery files
• Checking tick collection health

System Architecture

TickCollector EA (MQL5)  -->  tick_writer.dll (Rust)  -->  Parquet files
     OnTick() callback         C-ABI FFI                   ZSTD compressed
     CopyTicksRange()          Buffered writes              One file per day
     Watermark dedup            Row group flush

• TickCollector EA runs on MT5 via Wine on headless Linux (bigblack)
• EA captures every tick via OnTick() callback + OnTimer() fallback (1-second interval)
• Ticks buffered in EA, flushed to Rust DLL (tick_writer.dll) via C-ABI FFI
• Rust DLL writes Parquet files with ZSTD level 3 compression
• One EA instance per symbol per chart -- this is a CRITICAL constraint (single symbol per EA)

Production Topology

Systemd Service Chain

xvfb (virtual framebuffer)
  --> xfce (desktop environment)
    --> x11vnc (VNC access)
      --> mt5 (MetaTrader 5 via Wine)

All are user-level systemd units. MT5 depends on the display stack. The chain must start in order -- xvfb first, mt5 last.

Daily File Rotation

• New Parquet file created each trading day
• Filename pattern: {SYMBOL}_{YYYYMMDD}.parquet
• EA detects day change and calls tw_close (finalize current file) + tw_init (open new file)
• Trading day runs Sunday open to Monday open (forex hours)

Crash Recovery

On EA restart:

1. EA calls tw_get_last_time_msc to read resume watermark from the last Parquet file footer
2. Watermark is the timestamp of the last tick written -- resume from there
3. If an existing file is found for today, new file gets _1, _2 suffix (not overwrite)
4. All recovery segments are valid, complete Parquet files with footers
5. Resume watermark prevents duplicate ticks after restart
6. Gap between shutdown and restart is backfilled via CopyTicksRange from the watermark

The Rust DLL renames corrupt/incomplete files to _partial suffix rather than attempting recovery.

Gap Detection

Use DuckDB to check for gaps between consecutive ticks:

WITH ticks AS (
    SELECT time_msc,
           lead(time_msc) OVER (ORDER BY time_msc) - time_msc AS gap_ms
    FROM read_parquet('{base_path}/FXVIEW_{SYMBOL}/{YYYY}/{SYMBOL}_{YYYYMMDD}.parquet')
)
SELECT time_msc, gap_ms,
       make_timestamp(time_msc * 1000) AS ts
FROM ticks
WHERE gap_ms > 60000  -- gaps > 1 minute
ORDER BY gap_ms DESC;

Interpreting results:

• Weekend gaps are normal (Friday close to Sunday open)
• Mid-session gaps indicate collection interruption
• Gaps < 1 minute are normal during low-activity periods
• Check systemd journal for correlating restart events

Wine Gotchas

These are CRITICAL anti-patterns -- violations cause silent crashes or data loss:

• NEVER use %I64u or %I64d format specifiers in MQL5 -- crashes Wine silently. Use IntegerToString() instead.
• Wine cannot follow symlinks for file access -- use physical file copies for EA and DLL deployment
• Headless Linux requires Xvfb virtual framebuffer -- MT5 GUI needs a display even when no monitor is connected
• compile.sh uses Wine CLI for compilation -- no MetaEditor GUI needed
• MetaEditor returns exit code 1 even on success under Wine -- check compile log instead
• Compile log is UTF-16LE -- use iconv -f UTF-16LE -t UTF-8 to read it
• DISPLAY=:99 must be set (Xvfb virtual display number)

Deployment Pipeline

Use /mql5:mql5-ship slash command for full deployment:

• --ea-only for EA source changes
• --dll-only for Rust DLL changes
• Full pipeline: commit -> push -> pull on bigblack -> compile -> validate

Detailed deployment steps (SSH commands, file copies, compilation) are in the headless-mt5-remote local skill. Not duplicated here to avoid drift.

Monitoring Commands

Check MT5 service status

systemctl --user status mt5

Check today's tick files

ls -la ~/.cache/opendeviationbar/ticks/FXVIEW_EURUSD/$(date +%Y)/

Count ticks in latest file

duckdb -c "SELECT count(*) FROM read_parquet('path/to/file.parquet')"

Check for recovery segments

ls ~/.cache/opendeviationbar/ticks/FXVIEW_EURUSD/$(date +%Y)/*_[0-9].parquet 2>/dev/null

View systemd service chain

systemctl --user list-units --type=service | grep -E "xvfb|xfce|x11vnc|mt5"

Check MT5 journal for errors

journalctl --user -u mt5 --since "1 hour ago" --no-pager

Scope Boundary

• This skill: Operations, monitoring, troubleshooting
• headless-mt5-remote local skill: Deployment steps (SSH, file copy, compilation)
• fxview-parquet-consumer skill: Data consumption patterns (schema, DuckDB queries)

Troubleshooting

Post-Execution Reflection

After this skill completes, check before closing:

1. Did the command succeed? — If not, fix the instruction or error table that caused the failure.
2. Did parameters or output change? — If the underlying tool's interface drifted, update Usage examples and Parameters table to match.
3. Was a workaround needed? — If you had to improvise (different flags, extra steps), update this SKILL.md so the next invocation doesn't need the same workaround.

Only update if the issue is real and reproducible — not speculative.