how to debug ros2 bag record buffer overrun on Foxy after 100GB record

By Sai Kiran Pandrala · Last verified: 2026-06-01 · Source: in-controller diagnostic help, controls-community forums (r/PLC, r/Robotics, r/CNC, r/Fanuc, r/KUKA, r/Cognex, r/labview), OEM service bulletins and changelogs, OEM service manuals

I was called out at 2am because Line 4 had a CNC throwing a how to debug ros2 bag record buffer overrun on Foxy after 100GB record alarm on ROS 2 Platform Error Codes, rclcpp/rclpy, DDS QoS, Lifecycle Nodes, MoveIt 2, 2026 and the swing-shift operator could not clear it - the procedure most field service techs walk in 2026 when this exact alarm hits during a production run. My muscle-memory shortcut is to stop, photograph the alarm history screen, capture the controller hour-meter, and work the fault in the order below rather than chasing the symptom. None of these steps require pinging the OEM hotline first unless the cell is under active warranty.

What how to debug ros2 bag record buffer overrun on foxy after 100gb record actually involves on ROS 2 Platform Error Codes, rclcpp/rclpy, DDS QoS, Lifecycle Nodes, MoveIt 2, 2026

On ROS 2 Platform Error Codes, rclcpp/rclpy, DDS QoS, Lifecycle Nodes, MoveIt 2, 2026 when this lands in my queue the tools I lean on first are ros2 topic echo and ros2 topic info -v for QoS introspection, Nav2 nav2_lifecycle_manager CLI for bringup recovery, perf and tracepoints via ros2_tracing for callback latency analysis. Each of these surfaces a different layer of the fault - keep at least the first one in your fault-history notebook so the next time this happens you do not start cold.

For verification on ROS 2 Platform Error Codes, rclcpp/rclpy, DDS QoS, Lifecycle Nodes, MoveIt 2, 2026, the methods that survive contact with a real second-shift production workload are tail rosout via ros2 run rqt_console rqt_console during fault reproduction and inspect DDS discovery with FASTDDS_BUILTIN_TRANSPORTS=UDPv4 and ROS_DOMAIN_ID consistency. Anything less than that and you are shipping on vibes.

Authoritative sources for ROS 2 Platform Error Codes, rclcpp/rclpy, DDS QoS, Lifecycle Nodes, MoveIt 2, 2026 that I cross-reference before committing to a fix: docs.ros.org, github.com, discourse.ros.org. OEM marketing brochures and trade-press writeups are signal, not ground truth.

The rest of this page is the structured fix path. Start with diagnose, then remediation, then the automation options so you do not have to do this by hand the next time it surfaces. Verify and safety sections at the end are the discipline that keeps the fix from regressing the next time you open the cabinet.

Diagnose first, fix second

Start by capturing the exact failure signal in writing before you change a single thing on your ROS 2 Platform Error Codes, rclcpp/rclpy, DDS QoS, Lifecycle Nodes, MoveIt 2, 2026 setup. On the controller HMI that is the alarm code, the alarm message text, the timestamp, the controller hour-meter, and the part-count when the alarm hit. On the OEM diagnostic interface that is the fault-history dump (Fanuc alarm history, KUKA KSS log, Cognex In-Sight event log) plus the running program block number at the moment of fault. Photograph the HMI screen with the alarm panel open. Do not paraphrase. Most OEM service workflows will not even route the warranty case without the controller serial number, the alarm history dump, and the fault timestamp - the field service engineer pastes the alarm code straight into the OEM diagnostic tool and the first response is "we see the fault, here is what the controller logged."

Second pass: open the ROS 2 Platform Error Codes, rclcpp/rclpy, DDS QoS, Lifecycle Nodes, MoveIt 2, 2026 controller diagnostic panel and read the alarm history or fault stack for the failing window. Most modern industrial controllers surface a fault trail (the controller alarm history, the OEM diagnostic interface, the fab MES event log, the cell controller PLC fault table). The alarm history tells you whether the fault was a real condition, a teammate changing a parameter or DI mapping in the same minute, or an OEM-side firmware quirk. Many SRVO or AXIS faults trace to a parameter-level change pushed in the same engineering session in the previous hour - the fault trail makes that obvious without guesswork.

Third pass: read the alarm code and the alarm message like an x-ray of your ROS 2 Platform Error Codes, rclcpp/rclpy, DDS QoS, Lifecycle Nodes, MoveIt 2, 2026 cell. Servo faults (SRVO-023 servo overcurrent, SRVO-068 overheat, SRVO-014 motor overload) point at the drive, the cable, or the motor itself - 023 = instantaneous overcurrent during accel, 014 = sustained thermal overload during a heavy duty cycle, 068 = ambient or coolant fault on the drive heatsink. Axis or motion faults (4078 absolute position lost, OT001 over-travel, EX1043 spindle alarm) point at encoder battery, hardstops, or the spindle drive. Vision faults (Cognex In-Sight 5403 timeout, 5404 illumination, 5410 acquisition) point at trigger, lighting, or the GigE link. Cross-reference the alarm code against the OEM fault-code list - SCPI instruments will return the same hex code via SYST:ERR? that the front panel shows. If the same alarm cycles between SRVO-023 and SRVO-068 over a tight loop, the duty cycle is exceeding the drive thermal envelope - back off the feedrate or add a duty-cycle dwell.

Field notes from real ROS 2 Platform Error Codes, rclcpp/rclpy, DDS QoS, Lifecycle Nodes, MoveIt 2, 2026 callouts

For Robotics jobs I keep a battered field notebook of "what bit me on ROS 2 Platform Error Codes and how I cleared it", writing it down the first time has saved me a dozen overnight returns. After every ROS 2 Platform Error Codes repair I run `run ros2 doctor --report and check QOS_COMPATIBILITY warnings` to confirm the loop actually held, it takes thirty seconds and has saved me at least one callback per month.

My standing rule on any ROS 2 Platform Error Codes ticket is to baseline with Foxglove Studio with rosbridge for live topic and parameter inspection before touching a single wire, half the "failed" parts I have replaced over the years were not actually failed. When a ROS 2 Platform Error Codes fault code lights up on the panel, the first thing I reach for is rqt_graph and rqt_console for runtime error stream view, it tells me whether the signal is real or a sensor pretending to be sick.

Tools I actually reach for

For most ROS 2 Platform Error Codes, rclcpp/rclpy, DDS QoS, Lifecycle Nodes, MoveIt 2, 2026 faults I start with Foxglove Studio with rosbridge for live topic and parameter inspection, fall back to ros2 node list and ros2 lifecycle get for state machine snapshot, ros2 topic echo and ros2 topic info -v for QoS introspection when Foxglove Studio with rosbridge for live topic and parameter inspection cannot surface the answer, and keep ros2 doctor diagnostic CLI command handy for the cases where neither answers. That ordering is not academic - it matches the layers of the fault as they tend to surface, so the cheapest signal lands first and the heavier tooling only comes out when the simpler answer does not hold up. My muscle-memory shortcut for this is to run the first tool while the alarm screen is still open, not after I have already cycled controller power.

Verification I run before I call it fixed

Before I mark a ROS 2 Platform Error Codes, rclcpp/rclpy, DDS QoS, Lifecycle Nodes, MoveIt 2, 2026 fault resolved, the verification loop below is what I actually run. Each step proves a different layer is green, and the order matters - the cheaper checks gate the more expensive ones.

verify MoveIt planning scene via ros2 service call /get_planning_scene moveit_msgs/srv/GetPlanningScene

If that one comes back clean, move to the next check. If it does not, stop and dig in there before layering more verification on top of a red signal.

check lifecycle node state with ros2 lifecycle get /node_name and force transition

If that one comes back clean, move to the next check. If it does not, stop and dig in there before layering more verification on top of a red signal.

trace executor latency with ros2 trace and analyze via tracetools_analysis

If that one comes back clean, move to the next check. If it does not, stop and dig in there before layering more verification on top of a red signal.

tail rosout via ros2 run rqt_console rqt_console during fault reproduction

If that one comes back clean, move to the next check. If it does not, stop and dig in there before layering more verification on top of a red signal.

validate publisher and subscriber QoS profiles match via ros2 topic info /topic --verbose

Only when every line above runs clean do I close the loop and update my fault-history notebook with the timestamps.

Where I check first when the docs disagree

When two sources contradict each other on a ROS 2 Platform Error Codes, rclcpp/rclpy, DDS QoS, Lifecycle Nodes, MoveIt 2, 2026 detail, the disambiguation order I lean on is stable. I usually check fast-dds.docs.eprosima.com for the ground-truth view on this part of ROS 2 Platform Error Codes, rclcpp/rclpy, DDS QoS, Lifecycle Nodes, MoveIt 2, 2026. I usually check moveit.picknik.ai for the ground-truth view on this part of ROS 2 Platform Error Codes, rclcpp/rclpy, DDS QoS, Lifecycle Nodes, MoveIt 2, 2026. I usually check design.ros2.org for the ground-truth view on this part of ROS 2 Platform Error Codes, rclcpp/rclpy, DDS QoS, Lifecycle Nodes, MoveIt 2, 2026. OEM marketing brochures and trade-press writeups are signal, not ground truth, and I treat them as such until the references above either confirm or contradict the claim.

Solution-focused remediation path

If the ROS 2 Platform Error Codes, rclcpp/rclpy, DDS QoS, Lifecycle Nodes, MoveIt 2, 2026 controller is slow, faulting on cached errors, or HMI-locked, work the cache and parameter stack in order. Cycle controller power per the OEM lockout procedure (master disconnect off, wait 60 seconds for bus discharge, master disconnect on), reboot, and re-home the axes. Clear the local fault history (most controllers expose this under Maintenance -> Clear faults, or Setup -> Reset alarms). Re-load the saved parameter set with the OEM utility (Fanuc PARAM RESTORE, KUKA archive restore) to bypass any local parameter drift. Always capture timing before the cycle: time how long the failing cycle takes three times, write it down, then repeat after the parameter restore so the delta is provable in your notes. Decision point: managed-cell issues go through your controls engineering team for a cell-wide config push; standalone-cell issues go through the OEM diagnostic utility before you escalate to the OEM hotline.

Start by sorting the ROS 2 Platform Error Codes, rclcpp/rclpy, DDS QoS, Lifecycle Nodes, MoveIt 2, 2026 fault into one of three buckets, because roughly 80% of cases fall here. Bucket one is electrical / drive: instantaneous overcurrent, sustained overload, drive overheat, bus undervoltage, or a phase-loss event. Bucket two is mechanical / motion: encoder battery low, absolute position lost, over-travel, hardstop hit, or a vibrated-loose cable. Bucket three is recipe / parameter / I/O: the program calls a tool that is not loaded, the work offset is wrong, a DI is mapped to a disconnected sensor, or a vision job version has drifted. Pick the bucket first, then act. Before you act, capture a baseline photo of the alarm screen plus the controller hour-meter so you can prove whether the fix actually moved the needle. Decision point: if the alarm is intermittent and the cell is under an OEM service contract, open the OEM hotline first - OEM phone support beats hours of speculative debugging on cost and on liability if the alarm recurs and trips a safety-related shutdown.

Before any destructive step on a ROS 2 Platform Error Codes, rclcpp/rclpy, DDS QoS, Lifecycle Nodes, MoveIt 2, 2026 cell, slow down and stage rollback. Snapshot the current firmware revision, the current parameter set (PARAM PUNCH OUT, KUKA archive, Cognex job export), the current ladder and HMI screens, the current I/O mapping, and the current member-roster of teach pendants registered to the cell to a notes entry first. Capture the failing photo, the ROS 2 Platform Error Codes, rclcpp/rclpy, DDS QoS, Lifecycle Nodes, MoveIt 2, 2026 alarm history dump, and the timestamp window. Photograph the cell from two angles: the controller HMI showing the alarm, and the cabinet showing the drive status LEDs. Then do the destructive step (clear a parameter, swap a drive, remove a teach pendant, restore a backup) inside a maintenance mode or a sister cell first, never the production cell directly. Capture the firmware revision, the safety-PLC permissions, the connected-pendant list, the cell operator roster, and the relevant fieldbus log snapshot to your notes before the destructive step. Decision point: if the cell is under an OEM service contract, the cheapest correct path is almost always to open the OEM hotline in parallel with the rollback - the OEM service engineer can confirm whether an OEM-side firmware push is responsible while you are still staging the change, which avoids a needless parameter edit if the fix is in the next firmware revision.

Automate this fix so you do not do it twice

Codify the firmware revision pin and rollback as a single notes entry

Once a stable firmware revision is identified for the ROS 2 Platform Error Codes, rclcpp/rclpy, DDS QoS, Lifecycle Nodes, MoveIt 2, 2026, write the revision string, the build hash, and the parameter set state to a fault-history notebook entry with the date in the title. Reproducible rollback is then a single OEM utility load plus a parameter restore. Pin the parameter set state explicitly so an OEM-side default change does not silently shift behavior under you. Stage the notebook entry next to a checklist that lists the failing photo, the ROS 2 Platform Error Codes, rclcpp/rclpy, DDS QoS, Lifecycle Nodes, MoveIt 2, 2026 alarm history dump (if any), and the OEM case number; the second time the cell faults at 9 a.m. you do not want to be rediscovering which firmware revision was actually green.

# Fault-history notebook template (ros)
Date: 2026-06-01
Controller: ros
Working firmware: 30iB-Plus 02.20 (Build hash: a1b2c3d)
Cell: Line 4 Cell B
Machine serial: SN-ros-12345
Failing photo: ~/notes/ros-2026-06-01.jpg
OEM case: OEM-ros-12345
Rollback path: load previous firmware from OEM utility, master OFF, restore parameter archive, power up

Monitor + alert via ROS 2 Platform Error Codes, rclcpp/rclpy, DDS QoS, Lifecycle Nodes, MoveIt 2, 2026 OEM diagnostic reports, alarm history, and plant dashboard ingestion

For the ROS 2 Platform Error Codes, rclcpp/rclpy, DDS QoS, Lifecycle Nodes, MoveIt 2, 2026, the most useful long-running telemetry is the OEM diagnostic reports + alarm history shipped to a plant dashboard (Grafana with a CSV source, Ignition with a tag history, the fab MES OEE per SEMI E10, a Notion database via the API) and graphed on a single view. Pair that with synthetic monitoring (a small script that triggers the failing cycle or runs the failing test sequence every 5 minutes from at least two cells) so a fleet-level regression lights up before teammates report it. Subscribe the on-call inbox or a private Teams channel to the ROS 2 Platform Error Codes, rclcpp/rclpy, DDS QoS, Lifecycle Nodes, MoveIt 2, 2026 OEM service bulletin (Atom/RSS or vendor portal webhook) plus the OEM service-status handle so an open bulletin self-correlates with the synthetic failures.

# Tiny synthetic monitor - hit the ROS 2 Platform Error Codes, rclcpp/rclpy, DDS QoS, Lifecycle Nodes, MoveIt 2, 2026 controller health endpoint every 5 minutes
while true; do curl -s -o /dev/null -w "%{http_code} %{time_total} $(date -Iseconds)\n" \ -H "Authorization: Bearer $TOKEN" \ https://controller.plant.local/api/v1/me \ >> /var/log/ros-synth.log sleep 300
done

Scrape ROS 2 Platform Error Codes, rclcpp/rclpy, DDS QoS, Lifecycle Nodes, MoveIt 2, 2026 controller alarm history + fieldbus log via scheduled job

For the ROS 2 Platform Error Codes, rclcpp/rclpy, DDS QoS, Lifecycle Nodes, MoveIt 2, 2026, cell faults usually surface as drive alarms, fieldbus dropouts, or vision-trigger misses before a full line stoppage. A weekly scheduled job that exports the last 7 days of these events to CSV gives you a paper trail to correlate with firmware updates, parameter edits, and OEM bulletins without staring at the HMI live. Register the task via cron on a plant-floor logger PC (Linux IPC), Windows Task Scheduler (schtasks /create /XML) on an engineering workstation, or a GitHub Actions schedule against a cell-controller API, then write the CSV to a plant file share or the fab MES for retention. Subscribe a simple dashboard (Grafana with a CSV source, Ignition with a tag history, the fab MES OEE report) to the same bucket so alarm events from every ROS 2 Platform Error Codes, rclcpp/rclpy, DDS QoS, Lifecycle Nodes, MoveIt 2, 2026 controller converge on a single view without per-cell HMI clicking.

# Export the controller alarm history via the OEM API (if supported)
curl -X POST https://controller.plant.local/api/v1/alarm_history \ -H "Authorization: Bearer $CONTROLLER_TOKEN" \ -H "Accept: application/json" \ -d '{"start_date":"2026-05-25","end_date":"2026-06-01"}' \ -o ros-alarm-history.json
# Export the cycle history for the last 7 days
curl -G https://controller.plant.local/api/v1/cycles \ -H "Authorization: Bearer $CONTROLLER_TOKEN" \ --data-urlencode "oldest=$(date -d '7 days ago' +%s)" \ -o ros-cycles.json

Common pitfalls and what to watch for

Controller firmware updates during an active alarm are the textbook way to break a ROS 2 Platform Error Codes, rclcpp/rclpy, DDS QoS, Lifecycle Nodes, MoveIt 2, 2026 cell further, and the trap catches experienced techs because the release notes look like they describe exactly the alarm at hand. Never accept a major firmware version bump while you are in the middle of debugging, never push a beta firmware unless the release notes tie it to a specific service bulletin for your symptom, and never roll forward when a rollback is available. Skipping a required parameter migration leaves a known regression path open even after the immediate fix, so check the deprecation timeline on the ROS 2 Platform Error Codes, rclcpp/rclpy, DDS QoS, Lifecycle Nodes, MoveIt 2, 2026 maintenance bulletin before deciding to wait.

The other half is trusting the OEM service bulletin verdict by itself. OEM bulletin indexes can miss regional issues that only hit one plant batch, the Trust Center will not flag a fieldbus-driver degradation, and the controller event-log entries can lag several minutes behind the actual fault. Cross-reference the OEM controls-community forum, r/ros, the failing photo timestamps, and the on-screen alarm narrative before committing to a destructive remediation on ROS 2 Platform Error Codes, rclcpp/rclpy, DDS QoS, Lifecycle Nodes, MoveIt 2, 2026.

Verify the fix worked

• Reproduce the original faulting cycle against ROS 2 Platform Error Codes, rclcpp/rclpy, DDS QoS, Lifecycle Nodes, MoveIt 2, 2026 on the same cell AND a sister cell with the same recipe. If the alarm or fault code still surfaces on any cell, you have not fixed it.
• Watch for 24 to 48 hours via the ROS 2 Platform Error Codes, rclcpp/rclpy, DDS QoS, Lifecycle Nodes, MoveIt 2, 2026 controller alarm history + the fieldbus log + your fault-history notebook. Cached fault states and stale fieldbus link state mask slow-burn drift and intermittent fieldbus issues.
• Smoke-test under realistic load: replay the cycle against a test workpiece for at least 30 minutes at your normal production feedrate, log success / alarm and the timestamp per attempt to a notes file.
• Capture the new state in a fault-history notebook entry so the next time this happens you do not rediscover it. Note firmware revision + parameter set + I/O mapping + failing photo + verbatim alarm string + fix applied. Push to a plant-wide maintenance wiki if your plant uses one.
• If the fix involved a maintenance-token rotation or a parameter set change, commit the new token to your password manager and photograph the parameter dump for archival.

Safety, rollback, blast radius

• Test in a ROS 2 Platform Error Codes, rclcpp/rclpy, DDS QoS, Lifecycle Nodes, MoveIt 2, 2026 maintenance mode or on a sister cell first before any change that touches the production cell. Snapshot the firmware revision, the parameter set, the I/O mapping, and the safety-PLC permissions before changing anything.
• Apply the principle of least surprise when granting teach-pendant access or safety-PLC permissions. Review the operator roster against the people who actually need access - extra teach pendants are extra blast radius.
• Use idempotent cycles where the ROS 2 Platform Error Codes, rclcpp/rclpy, DDS QoS, Lifecycle Nodes, MoveIt 2, 2026 controller supports it (the OEM cycle-id de-dupe, external id keys on MES records) so a re-run cycle does not double-count parts or duplicate scrap records.
• Know your rollback path. Firmware rollback is a one-line OEM utility load; a maintenance-token rotation is reversible if you kept the old token in the password manager during cutover; a parameter set change is reversible only if you saved the previous archive.
• For cell-wide or plant-wide changes, line up a maintenance window with production scheduling before pushing through the OEM utility.

FAQ

Related fixes

Related guides worth a look while you sort this one out:

• how to debug MoveIt 2 OMPL planner FAILED to find motion plan after collision object add
• how to debug rclpy InvalidHandle exception after node destroy_node call
• how to debug DDS discovery failure across Docker bridge network on ROS 2 Jazzy
• how to debug rclcpp callback group deadlock with MutuallyExclusiveCallbackGroup
• how to fix Fast DDS RTPS shared memory transport buffer full error on Iron
• how to fix rclpy executor stuck after Ctrl-C signal handler shutdown deadlock