Skip to content
Website
🧱 CFC 238Construction site coordination and safety for telematics techniciansChapter 6 of 10 in the section

Working on a live production site: window, notice and rollback

Intervening on the telecommunications infrastructure of an operating hospital or banking site requires a rigorous method. This chapter details the negotiation of the written maintenance window, communication channels to the affected services, the drafting of the technical fallback plan and the precise determination of the point-of-no-return time ensuring business continuity.

Key takeaways

  • The maintenance window requires formal signed approval specifying the time slot and the impacted network services.
  • The user notice details the estimated duration, service interruptions and internal emergency numbers.
  • The rollback plan is drafted and technically verified before touching the first cable.
  • The point of no return sets the exact deadline mandating a switchback to the initial configuration.

Negotiate the written time slot and organise the notice to operators

Replacing a core switch or restructuring an optical distribution frame in a live production environment tolerates no improvisation. In a Geneva private bank or a Vaud hospital centre, a cut in a data link immediately freezes financial transactions or access to electronic patient files. Such a procedure is not a mere technical action, but a strict organisational process. The outage must strictly be covered by a written agreement between the telematics technician and the client's information systems management, specifying the execution date, the exact minute of commencement, the deadline for completion and the perimeter of the equipment involved.

Operating constraints and standard maintenance windows in French-speaking Switzerland
Client environmentTypical maintenance windowMajor operational constraints
Hospital or clinic facilityWeekday nights between 01:00 and 04:30Mandatory maintenance of telemetry streams, IP nurse calls and emergency telephony.
Banking or financial institutionSaturday from 14:00 to Sunday 18:00Total freeze of stock market transactions, core server closure, secure access to computer rooms.
Administrative or commercial buildingWeekday evenings between 19:00 and 23:00Shutdown of office workstations, nightly backups and access control operational by morning.

Communication to end users is the responsibility of the client, but the network team leader must provide the technical details of the message at least five working days prior to the intervention. The information note must specify the list of patch panels involved, the temporarily unavailable local subnets, possible workarounds and the contact details of the standby duty team during the operation.

Build the rollback plan and lock in the point of no return

The fallback plan, or rollback procedure, is the sine qua non condition for any manipulation on an active infrastructure. It is a technical document describing the chronological sequence of operations required to restore the initial functional state if the new hardware or software configuration fails. According to art. 364 CO, the liability of the contractor is subject to the same rules as that of the employee. The tradesperson is liable for the proper performance of the entrusted services and must implement technical measures of due diligence suitable for protecting their client's equipment.

  1. Exhaustive configuration backup and physical labellingBefore removing a single fibre or copper patch cord, export the switch configuration files to a secure external medium. Photograph the patching arrangement and affix numbered tags to each RJ45 connector or optical plug.
  2. Recording of deployment steps and individual time allowancesQuantify the time required for installation, connection and commissioning of the new active equipment. Each manipulation must include a realistic time estimate, supplemented by a margin for connectivity checks.
  3. Irrevocable determination of the point of no returnCalculate the exact deadline by subtracting the time required to reassemble the old installation from the end time of the window. If the new equipment is not operational at that exact moment, work must be aborted to initiate the reinstallation of the original hardware.
  4. Functional check and handover restoration reportCheck addressing tables, inter-building links and port status LEDs. Test the reachability of building automation gateways before signing off on the handover of the infrastructure.

The point of no return allows for no emotional or financial compromise. When the clock reaches the deadline set in the outage protocol, the on-site team immediately ceases all troubleshooting attempts on the new equipment. Continuing the intervention beyond this time boundary exposes the building to an extended service disruption when business resumes in the morning.

Real-time control of the outage and compliance with the intervention clock

In a cantonal hospital centre or a financial institution in French-speaking Switzerland, shutting down a production network switch tolerates no improvisation. The outage paralyses operating theatres, the distribution of medical fluids over IP or critical financial transactions. The telematics technician's intervention never starts without a written maintenance window, signed by the IT infrastructure manager and the operator. This contractual document sets the authorised outage schedule, the permitted impact on links and the maximum deadline beyond which no unavailability is permitted.

Prior to any physical intervention on patch panels, the notice to users must be confirmed with the network operations centre. Distributed according to a negotiated notice period, this communication warns the affected departments of the temporary shutdown of data and IP telephony. On site, the telematics technician ensures that traffic failover to backup routes has been performed by system administrators. Only after this formal green light does the unplugging of RJ45 patch leads or fibre-optic jumpers begin.

Standard schedule for a 240-minute outage window on a critical site
MilestoneTelematics actionTechnical inspectionDecision
T0 (22:00)Isolation of the switch and backupConsole access and LED statusOutage green light
T0 + 45 minMechanical replacement of chassis in rackPower supplies and earthingInstallation validated
T0 + 90 minRe-patching of fibre and copper linksContinuity and speed autonegotiationLayer 1 compliant
T0 + 120 minMilestone assessment (Point of no return)Stability of routing and VLANsProceed or roll back

Triggering the rollback plan and return to operation

Establishing a rollback plan before starting is an absolute requirement on a live network. This written protocol details every step to restore the initial situation if the new equipment experiences a hardware fault, a firmware bug or a refusal to communicate with the core network. The document records the complete backup of prior configurations, the port-by-port cabling schedule and the power-up sequence for the decommissioned equipment. The Code of Obligations (CO) reiterates the contractor's duty of due diligence. Working without a formalised rollback procedure engages the installer's liability in the event of an extended operational stoppage.

The point of no return identified in the intervention file is the absolute time limit of the operation. Aligned with the intervention clock at exactly half of the granted window, this time milestone tolerates no compromise. If at that exact minute, synchronization LEDs, MAC table population or access to sensitive subnets are not completely stable, the telematics technician immediately stops the integration of the new switch. Continuing manipulations beyond this point to attempt troubleshooting consumes the time reserved for restoring the original system, risking a service disruption by morning.

  1. Halting work on the new equipmentAs soon as the point of no return is reached without full validation, log the integration failure, disconnect the power supplies of the new hardware and notify the operator of the rollback execution.
  2. Physical reconnection of the old switchReinstall the old switch in the rack and reconnect each optical jumper and copper patch cord according to the initial labelling recorded in the cabling schedule.
  3. Software restoration and power restorationRe-engage the power supplies of the original unit, verify the absence of startup errors and load the backup configuration saved before work commenced.
  4. Joint continuity inspectionValidate the nominal recovery of critical links, network loops and telephony with the operations centre to certify the normal resumption of traffic.

Once continuity is verified, on the validated new switch or after a complete rollback execution, the telematics technician formally closes the construction site. The final announcement of link availability is transmitted to operations for broadcast to users. A joint inspection report is drawn up and signed on site with the site manager. It records compliance with the outage schedule, the measured transmission values and the objective grounds that dictated either validation or rollback.

Crisis management and rollback recording: procedure in case of failure

When the point of no return identified during preliminary studies is crossed or if the rollback plan established before starting fails to restore links by the set time, the operation shifts into critical operational incident management. In a banking establishment, data centre or cantonal hospital, the unforeseen outage of a distribution switch beyond the agreed schedule blocks workstations, IP telephony and access to medical records. Faced with a hardware block, the team leader and their installers must never improvise software reconfiguration or alter the IP addressing scheme under user pressure. The priority action consists of freezing the patch panel: stop all unplugging of fibre-optic jumpers or RJ45 patch leads, record the state of interface status LEDs and scrupulously log the status of physical ports.

  1. Notify IT operationsAlert the site operations manager via direct line, notify them of the failure to achieve nominal recommissioning and transmit the technical diagnostics of the faulty links.
  2. Trigger crisis communicationThe notice to users is then relayed by the site management, which triggers its business continuity plan and informs the relevant departments of the extension of the outage.
  3. Execute rollback to original equipmentMethodically reconnect the jumpers to the original switch according to the joint labelling recorded prior to the works, or switch uplinks over to the passive standby infrastructure.
  4. Log acceptance measurementsVerify physical continuity using an optical power meter or copper certifier, confirm the clearance of loop alarms with the administrator and freeze the configuration in degraded mode.

Once the network is restored or stabilised in degraded mode, the telematics technician draws up a joint inspection report with the client's systems administrator. The document logs the exact timestamp of outages, the sequence of manipulations and the photometric measurements of optical links. The written maintenance window agreed in advance by management forms the legal framework of the operation. According to art. 97 CO, the debtor is liable to pay damages if an obligation is not performed, unless they prove that no fault is attributable to them. This precise report, accompanied by configuration backups performed prior to the work, proves that the service outage resulted from an unforeseeable hardware malfunction and not from faulty workmanship by the installer.

Frequently asked questions

What should be done if a switch refuses to reboot ten minutes before the end of the authorised window?

The team leader applies the fallback plan without waiting for the time limit to expire. Ten minutes before the deadline, software troubleshooting attempts are abandoned to protect operations. The technician reinstalls the old equipment or switches the patch cords over to the standby equipment documented in the fallback procedure. The client's network administrator is alerted immediately to observe port status and confirm the minimum restoration of links.

Who takes responsibility for aborting the rollback to continue the installation?

Only the client's operations manager holds the authority to order the continuation of work beyond the point of no return. This agreement requires written validation provided by timestamped message or countersigned in the intervention log. If the client refuses to extend the outage or cannot be reached, the telematics technician halts modifications and executes the scheduled rollback to release the infrastructure before business resumes.

How can it be proven that an extended network outage is not due to installer error?

The company relies on the initial patching schedule, the prior configuration backup and the joint photometric survey performed prior to disconnection. These documents establish the functional state of the network before work began. When the fault occurs, the LED readings, console diagnostic traces and event logs of the faulty equipment are appended to the intervention report. This methodical traceability demonstrates compliance with recognised trade standards and the absence of mishandling.

Can a switch be cut without prior notice if the network features physical redundancy?

No, any shutdown of active production equipment requires formal prior notice, even in the presence of a redundancy protocol such as spanning tree or virtual stacking. An automatic failover of links can saturate a remaining uplink, degrade latency or reveal a latent fault on the passive switch. The scheduled window warns users and ensures administrators are present to monitor network traffic convergence.

Texts and standards cited

  • CO art. 97non-performance of the obligation: the debtor is liable to pay damages, unless they prove that no fault is attributable to them

In the same section