1
EXHIBIT 10.23
** This portion of the document has been omitted pursuant to a request for
confidential treatment and this document, including the information, has been
filed separately with such respect.
[INTERACTIVE TELESIS LOGO]
September 14, 1999
Xxxx XxXxxxxxxx
Frontier Communications Services, Inc.
00000 Xxxxx Xxxxx Xxxxxx
Xxxxxxxxxxx, XX 00000-0000
Dear Xxxx:
As requested, enclosed is the Record and Replay Communication Services
Agreement (in triplicate) executed by Interactive Telesis. Would you please
provide one fully executed copy to the undersigned at your convenience.
You will note that we have added the PostView Support and Maintenance Section of
our Business Continuity Plan as Attachment C to the Agreement. I think you will
find this addendum will cover the issues that were raised earlier and for which
Frontier wanted more specific detail than set forth in our Best Practices
document. If you would like any amendments to Attachment C, just let us know
and we will be happy to accommodate.
Yours very truly,
/s/ XXXXXX X. XXXXXXX
-------------------------------
Xxxxxx X. Xxxxxxx
President
Enc.
2
RECORD AND REPLAY COMMUNICATION SERVICES AGREEMENT
1. Agreement between Interactive Telesis Inc. (ITI) and
Name: Frontier Communications Services Inc., d/b/a Frontier ConferTech
Address: 00000 Xxxxx Xxxxx Xxxxxx, Xxxxxxxxxxx, XX 00000-0000
(the Corporation)
2. ITI operates a telecommunications system that provides, inter alia,
digital recording and playback capabilities ("the Network") accessible by
various communication devices, including telephone and computer, as
described in Attachment "A" (the "Communication Services").
3. ITI agrees to provide the Communication Services to the Corporation and to
provide all hardware and software required for delivering the
Communication Services. Services not completed to the Corporation's
satisfaction shall be reperformed at no cost to the Corporation.
4. Upon acceptance of the services provided hereunder and submission of
accurate invoices by ITI, Corporation shall pay ITI for the performance of
the Communication Services in accordance with the schedule contained in
Attachment "B", attached hereto and made a part hereof. The Communication
Services shall be performed to the satisfaction of the Corporation and
shall be in accordance with the highest professional standards commonly
used in the automated call processing industry.
5. Non-exclusive Rights: It is expressly understood and agreed that this
Agreement neither grants to ITI an exclusive right or privilege to sell to
or provide the Communication Services or services of a similar type
described in this Agreement which the Corporation may require. It is
understood that the Corporation may contract with other service providers
for the procurement of comparable services. In addition, the Corporation
shall, at its sole discretion, decide the extent to which the Corporation
will market, advertise, promote, support or otherwise assist in further
offerings of the services contemplated under this Agreement. The
Communication Services shall be provided by ITI in accordance with the
specific Support and Maintenance conditions set forth in Attachment "C."
6. IT IS UNDERSTOOD THAT THE GENERAL CONDITIONS OF BUSINESS ATTACHED TO THIS
AGREEMENT ARE INCORPORATED BY REFERENCE AND FORM PART OF THIS AGREEMENT.
BY EXECUTING THIS AGREEMENT THE CORPORATION CONFIRMS HAVING READ,
UNDERSTOOD AND ACCEPTED EACH OF THE GENERAL CONDITIONS OF BUSINESS.
FRONTIER CONFERTECH INTERACTIVE TELESIS INC.
Signature: /s/ XXXXXX XXXXXXX Signature: /s/ XXXXXX X. XXXXXXX
--------------------- ------------------------
Name: Xxxxxx Xxxxxxx Name: Xxxxxx X. Xxxxxxx
-------------------------- -----------------------------
Title: VP Title: President
------------------------- ----------------------------
Date: 09/15/99 Date: 9-14-99
-------------------------- -----------------------------
3
GENERAL CONDITIONS OF BUSINESS
The following GENERAL CONDITIONS OF BUSINESS are incorporated by reference in
the Record and Replay Communication Services Agreement.
1. AGREEMENT
ITI agrees to provide Communication Services to the Corporation utilizing
the Network.
2. BILLING AND PAYMENT
A. Invoicing Payment. All usage charges are billed on minute increments.
Invoices submitted pursuant to this Agreement shall be deemed correct unless
the Corporation makes written objection stating the reason within 60 days of
receipt by the Corporation of the disputed invoice. ITI shall submit invoices
to Corporation, together with such documentation as Corporation may require,
at the following address:
Frontier ConferTech
00000 Xxxxx Xxxxx Xxxxxx
Xxxxxxxxxxx, XX 00000
Attn: Xxxxxxx Xxxxx
If the Corporation has no objection to an invoice, it shall pay the
invoice in full within forty-five (45) days after receipt of such invoice. If
Corporation objects to an invoice or any portion thereof, it shall notify ITI
of its objections within thirty (30) days after receipt and, within forty-five
(45) days of receipt, it shall pay that portion of the invoice not in dispute.
Any objections or disputes concerning invoices shall be resolved in accordance
with Section 12 herein.
3. AUDIT
ITI shall maintain accurate and complete records of all recording and
replay usage for which payment under this Agreement is to be computed. Such
records shall be maintained in accordance with recognized commercial accounting
practices so they may be readily audited and shall be held until costs have
been finally determined under this Agreement and payment or final adjustment of
payments as the case may be has been made. ITI shall permit Corporation or
Corporation's Representative to examine and audit these records and all
supporting records at all reasonable times. Audits shall be made not later than
two (2) calendar years after the completion of services rendered pursuant to
this Agreement.
4. REPORTING
ITI shall provide usage and billing reports in the format(s) and time
intervals required by the Corporation. ITI shall, at the Corporation's request,
provide secured online call management and call usage access to the Corporation
for the Record and Replay services provided under this Agreement.
5. TITLE TO WORK PRODUCTS
All right, title and interest in and to all tangible and intangible work
and work products developed or produced under this Agreement by or on behalf of
ITI for the Corporation, whether comprising or incorporated in specifications,
samples, data, computer programs, reports, documentation or other technical or
business information, and all right, title and interest in and to patents,
copyrights, trade secrets, trademarks and other intellectual property derived
from such work and work products are the property of ITI. This Provision shall
not apply to the Corporation's confidential information that may be contained
in the work product.
6. WARRANTIES
ITI agrees that it shall provide, in connection with the services provided
pursuant to this Agreement, the standard of care, skill, and diligence normally
provided by a vendor in the performance of similar services and that all such
services shall be performed in accordance with sound and accepted industry
standards and practices. If ITI fails to meet such standards of care, skill and
diligence set forth in this section, ITI shall be liable for all direct damages
caused thereby.
ITI has developed a Business Continuity Plan (the "BCP") which addresses
issues relating to system maintenance, back-up, and disaster recovery relating
to the services to be provided hereunder. The BCP details ITI's procedures and
is attached hereto as Exhibit "C" and incorporated by reference herein.
7. YEAR 2000
ITI warrants that all services, including any software or data processing
support, data exchanges or systems of any kind used by ITI to support
performance of the services, shall be Year 2000 Compliant, as defined below.
This warranty begins upon the commencement of the service and shall continue
thereafter until March 31, 2001, or the expiration of any other warranty period
applicable to the service, whichever is later. In the event of a breach of this
warranty, Corporation shall have any and all rights and remedies provided at
law or in equity. Upon receiving notice thereof, ITI shall promptly correct any
breach of this warranty, including correction of any invalid data or
information resulting therefrom.
"Year 2000 Compliant" means proper performance of any service which is
dependent upon or involves calendar dates before, during and after the Year
2000, including both single century and multi-century formulas, leap year
capabilities and transition from the year 2000 to the year 2001. "Proper
Performance" means the service will not cause an abnormal ending scenario or
generate incorrect values involving dates.
8. INDEMNIFICATION
A. By Corporation. The Corporation shall indemnify and hold harmless ITI
and any affiliated companies from and against any and all claims, damages, or
liability, including reasonable attorneys' fees and the costs and expense of
any legal action, or prelude thereto, for libel, slander, invasion of privacy,
securities law violations, improper trade practices, illegal competition,
infringement of copyright or licenses (if not covered under Section 14
hereof), or any other wrongful conduct resulting from the dissemination of any
information provided by customers of the Corporation and made available on the
Network and not involving negligence or wrongful acts or omissions on the part
of ITI.
B. By ITI. ITI shall indemnify, defend and hold harmless Corporation
from and against any claims, damage or liability including reasonable attorneys
fees and the costs and expenses of any legal action, or prelude thereto,
arising as a result of the negligence or wrongful acts or omissions of ITI, its
officers, employees or agents in performing any services for Corporation
pursuant to this Agreement, including, without limitation, claims under Sections
7 and 14 hereof.
C. Survival. The provision of this paragraph shall remain effective and
inure to the benefit of the respective parties notwithstanding the expiration,
cancellation, or termination of this Agreement.
9. TERMINATION
Either party may cancel this Agreement at any time by giving the other
party 30 days' written notice thereof and specifying the effective date of such
cancellation. ITI may cancel this Agreement if the Corporation has failed to
make any payment when due pursuant to this Agreement upon giving 30 days'
written notice to Corporation and the Corporation does not cure such failure
within 10 days after receipt of ITI's written notice thereof.
10. GOVERNMENTAL REGULATION
This Agreement is subject to the terms of the licenses held by ITI and to
all applicable federal, state, and municipal laws, regulations, and decisions,
either presently in existence or enacted, made or enforced hereafter, including
the regulations and actions of all governmental administrative agencies and
commissions having jurisdiction.
11. ASSIGNMENT AND DELEGATION
Neither party may assign any rights hereunder without the express prior
written consent of the other, which consent shall not be unreasonably delayed
or withheld.
12. MEDIATION
If a dispute arises out of or relates to this Agreement, or its breach,
and the parties have not been successful in resolving such dispute through
negotiation, the parties agree to attempt to resolve the dispute through
mediation by submitting the dispute to a sole mediator selected by the parties
or, at any time at the option of a party, to mediation by the American
Arbitration Association ("AAA"). Each party shall bear its own expenses and,
unless the arbitrator rules otherwise, an equal share of the expenses of the
mediator and the fees of the AAA. The parties, their representatives, other
participants and the mediator shall hold the existence, content and result of
the mediation in confidence. If such dispute is not resolved by such mediation,
the parties shall have the right to resort to any remedies permitted by law.
All defenses based on passage of time shall be tolled pending
4
the termination of the mediation. Nothing in this clause shall be construed to
preclude any party from seeking injunctive relief in order to protect its
rights pending mediation. A request by a party to a court for such injunctive
relief shall not be deemed a waiver of the obligation to mediate.
13. WAIVER
The failure of either party to this Agreement to object to, or to take
affirmative action with respect to, any conduct of the other which is a
violation of the terms of this Agreement shall not be construed as a waiver of
such violation or breach, or of any future breach, violation, or wrongful
conduct.
14. PATENT AND OTHER PROPRIETARY RIGHTS INDEMNITY
ITI warrants that the service provided hereunder does not infringe upon or
violate any patent, copyright, trade secret, or any other proprietary right
of any third party. In the event of any claim for patent/trademark infringement
by any third party against Corporation related to ITI's provision of services
provided hereunder. ITI agrees to defend any suit or proceeding brought against
Corporation, provided ITI is notified promptly in writing and is given complete
authority and information required for the defense of same in Corporation's
name. ITI shall pay all damages and costs (including reasonable attorney fees)
whether or not such claim is successful or awarded against Corporation.
15. EQUAL OPPORTUNITY
ITI will not discriminate against any employee or applicant for employment
because of race, religion, color, sex, age, or national origin. ITI will take
affirmative action to ensure that applicants are employed, and the employees are
treated during employment without regard to their race, religion, color, sex,
age, or national origin. Such action shall include, but not be limited to, the
employment, upgrading, demotion, or transfer; recruitment or recruitment
advertising; layoff or termination; rates of pay or other forms of compensation;
and selection for training, including apprenticeship. ITI agrees to post in
conspicuous places, available to employees and applicants for employment,
notices setting forth the provisions of this Section . ITI will comply with all
provisions of Executive Order No. 11246 of September 24, 1965, as amended, and
of the rules, regulations, and relevant orders of the Secretary of Labor. ITI
will include the provisions of this Section in every subcontract or purchase
order relating to this agreement unless exempted by rules, regulations, or
orders of the Secretary of Labor issued pursuant to Section 204 or Executive
Order No. 11246 of September 24, 1965, so that such provisions will be binding
upon each subcontractor and vendor.
16. CONFIDENTIALITY
Except upon mutual written agreement, or as may be required by law,
neither party shall in any way or form disclose the existence or terms of this
Agreement.
17. NOTICES
Any notice, demand, or request provided for in, or allowed by, this
Agreement shall be in writing and shall be deemed properly made if personally
delivered, delivered by courier or sent via facsimile transmission to the
person at the facsimile number specified below and shall be deemed received, if
personally delivered or delivered by courier, upon delivery, or if sent by
facsimile transmission, upon transmission, or if mailed, on the fifth business
day following deposit in the US mail.
To ITI:
Interactive Telesis
000 Xxxxxxxxx Xxxx., Xxxxx 000
Xxxxxxxxx, XX 00000
Attention: Xxxxx Xxxx, V.P. Operations
Telephone Number: 000-000-0000
Facsimile Number: 760-632-1790
To Corporation:
Frontier ConferTech
00000 Xxxxx Xxxxx Xxxxxx
Xxxxxxxxxxx, XX 00000
Attention: Xxxx XxXxxxxxxx
Telephone Number: 303/000-0000
Facsimile Number: 303/633-3030
Nothing contained in this Section shall preclude the transmission of
routine invoices or correspondence, messages, and information by other methods
between the respective parties or their authorized representatives in the
ordinary course of performing their obligations under this Agreement.
Either party may change any individual designated above without amendment
to this Agreement, by written notice provided to the other party.
18. MODIFICATION
This writing, including the schedules attached hereto, contains the entire
agreement between the parties. No representations were made or relied upon by
either party, other than those expressly set forth herein. No agent, employee,
or other representative of either party is empowered to alter any of the terms
hereof, unless done in writing and signed by the respective parties.
5
ATTACHMENT "A"
The Communication Services to be provided under this Agreement are set forth
in detail under ITI's response dated July 1, 1998, to the Corporation's RFP
dated June 15, 1998, which are incorporated herein by reference. In general,
the services will include the following:
- Services - The digital recording and automated replay of teleconference
calls originating at the Corporation's Westminster, Colorado, and Canadian
operations. The service includes the recording and storage of all calls as
a back-up to analog taping carried out by the Corporation.
- Storage - All calls, whether scheduled for replay or not, shall be
available on the system for retrieval for a period of 30 days from the date
of recording or, in the event of replay, 30 days after the expiration of
the replay, within two business hours.
- Recordings, in any format, will be removed from all media and destroyed no
earlier than 60 days and no later than 90 days after the expiration date of
each conference call.
- Reporting - The call detail reporting format will be determined by the
Corporation and may be modified from time to time at the Corporation's
request.
- Management tools - ITI will develop an online session management interface
(OSMI) providing the Corporation with the ability to remotely manage all
aspects of the replay of each conference call, from set-up to tear-down.
Details of OSMI are set forth in the ITI response at item 4.2.1.1.
- Internet replay - ITI will design, develop and host a web site for the
Corporation dedicated to the replay of audio teleconferences through the
Internet. ITI will modify and enhance this dedicated web site from time to
time at the Corporation's request.
SERVICE STANDARDS
1. The service will initially be provisioned with 288 dedicated ports provided
by the following carriers:
Frontier Communications 8 long distance TIs
AT&T 2 long distance TIs
TCG 2 local access TIs
2. ITI will perform all technical development, testing, installation and
operation of Corporation's system according to the specifications outlined
by the Corporation.
3. ITI has configured the dedicated system to allow routine maintenance and
the addition of system hardware without any interruption of service. If any
interruption of the service is anticipated, ITI will inform the Corporation
of such interruption (if possible) before interrupting service to ensure
that call volume loss is minimized.
4. ITI has configured the existing hardware and software for maximum fault
tolerance and redundancy and, in the event of equipment malfunction or
failure, general service should not be interrupted, although total capacity
will be reduced until repairs to the system have been completed. ITI will
provide its best efforts to replace any lost capacity as quickly as
possible.
6
5. ITI will provide technical support for any and all issues arising from the
ITI equipment and the operation of the Communication Services to be
provided under this Agreement at no additional cost to the Corporation.
6. ITI has preconfigured hot spare work stations to replace any failed
machines. ITI has two highly fault-tolerant servers and a spare server
that is synchronized nightly with the other servers. This system includes
an auto-loading SCSI tape backup system that can store 96GB of data. ITI
performs daily incremental backups and weekly full backups of all data
with tapes being archived off-site.
7. All call usage and billing data shall be available to the client within
one hour of request during usual business hours.
8. ITI has dedicated systems-support staff that performs monthly maintenance
on all machines during non-peak hours. ITI disconnects each machine, one
by one, and runs a full set of diagnostics. This consists of checking
hard drives for logical and physical errors and running both high- and
low-level diagnostics on all system components (such as controller cards,
T1 cards, RAM, etc.).
9. All recordings are stored digitally into 8-bit U-law dialogic VOX format
at a sampling rate of 8,000Hz.
10. All hardware and software are prequalified for Y2K compliance. Software
design and testing incorporates all Y2K issues.
7
ATTACHMENT "B"
The following pricing is to be charged for services performed by Interactive
Telesis Inc. pursuant to this Agreement.
Services Price
Teleconference recording and playback no charge
One-time set-up fee
Per call:
Recording and replay **
Editing $15 per conference call +
Customized greetings $45 per conference call +
Usage reports no charge
Implementation of replay features
Pause, reverse, fast forward no charge
Fast forward to specific location $25 per conference call +
Safe-harbor disclaimer option $15 per conference call +
Fax-on demand
Set-up (up to 7 pages) $75 per conference call
Fax usage $0.40 per page
Message capture/transcription
Message capture option no charge
Transcription (name, address, affiliation) $0.80 per message
Transcription $1.65 per minute of transcription
Detail report (standard format) no charge
Internet replay
Replay (no time restriction) $350 per conference call ++
Record and transmit electronically to client $250 per conference call
Usage report no charge
Web site hosting fee $150 per month
Specialized programming
Specialized programming involving $125 per hour
creation of new applications and (fixed per project pricing is
enhancements to existing applications, available)
including revisions to specialized reports
Internet management systems
Development:
Call usage detail no charge
Customization of call Charge to be determined
management interface according to client specifications.
+ No charge if completed by Customer.
++ Subject to volume/band width requirements; per-minute charge is
available.
** CONFIDENTIAL TREATMENT REQUESTED.
PLEASE REFER TO THE LEGEND ON
THE FRONT PAGE OF THIS DOCUMENT.
8
ATTACHMENT "B" (CONT'D)
Record and Replay Pricing
**
ITI shall be responsible for all hardware and software development costs
necessary to provide the Communication Services contemplated under this
Agreement other than:
(i) the customized development of the online session management interface
(OSMI) and
(ii) creation of any new applications or substantial enhancements to
existing applications which were not envisioned in the original RFP
response dated July 1, 1998.
The Corporation shall be responsible for all telecommunication charges
necessary to provide the Communication Services under this Agreement. These
charges shall include, but are not limited to, the following:
1. Installation and monthly recurring charges for all long distance T1
service;
2. Where ITI arranges for long distance and/or local T1 service from a
second carrier (initially two long distance T1s from AT&T and two
local T1s from TCG), ITI will pass through these expenses, with
appropriate third-party invoices, and the Corporation shall be
responsible for reimbursing ITI at cost for same;
3. Where ITI, at the request of the Corporation, arranges installation
of second-carrier services, ITI shall advise the Corporation of
expected costs prior to incurring any expenses.
** CONFIDENTIAL TREATMENT REQUESTED.
PLEASE REFER TO THE LEGEND ON
THE FRONT PAGE OF THIS DOCUMENT.
9
ATTACHMENT C
INTERACTIVE TELESIS INC [INTERACTIVE TELESIS LOGO]
===============================================================================
BUSINESS CONTINUITY PLAN
----------------------------------------------------------------
Supplement One
Post View Support and Maintenance
--------------------------------------------------------------------------------
10
Overview...................................................................2
Purpose................................................................2
Scope..................................................................2
Definition of Terms....................................................2
Service Ticket Classification..........................................3
Bug Ticket Classification..............................................3
System maintenance.........................................................4
Disk Maintenance.......................................................4
Network Maintenance....................................................4
Other devices..........................................................5
Database maintenance...................................................5
Virus protection.......................................................5
System Backup..........................................................6
System monitoring..........................................................7
System support.............................................................8
Procedures.............................................................8
Escalations............................................................9
Key Contacts...........................................................9
System Capacity planning and upgrade......................................10
11
OVERVIEW
PURPOSE
To provide a comprehensive set of procedures for support and maintenance
of the Frontier Confertech Post View System located in San Diego and
Toronto, Canada and hosted by Interactive Telesis. Used in conjunction
with our Business Continuity Plan it will insure the reliability and
availability of the Post View system.
SCOPE
This document is intended to define the levels of support and what type
of maintenance will be performed as well as the frequency of such
maintenance. Additionally, it describes the communication process should
the need for support arise.
DEFINITION OF TERMS
This section will define the terms used throughout this document.
- Service Ticket Classification: Method of tracking problems and
solutions with the Post View System.
- Bug Classification: The prioritization of known software bugs and the
order in which they will be resolved.
- System Outage: Critical error with system resulting in the system
being unavailable.
- Circuit Outage: T1 voice line non-operational.
- Service degradation: Slow and or erratic performance of Post View
system.
- XxxxxxxXxxxxxx.xxx: ITI's customer service Internet operation (Still
under implementation).
- Preventive maintenance: Proactive maintenance performed on a
scheduled basis and designed to insure the health of the system.
- Scheduled maintenance: Maintenance performed on a scheduled basis
either recurring or one time.
- Unscheduled maintenance: Performed on an irregular basis because of
an abnormal operating condition.
- Automated maintenance: Maintenance performed on a recurring basis by
a software utility or SNMP application. I.E. scheduled disk
defragmentation, virus scan, RAID consistency check, or physical
surface scan.
- Non-intrusive maintenance: Maintenance performed during system
operation that does not result in degradation of system performance,
e.g. system backup, virus scan, or defragmentation.
- Intrusive maintenance: Maintenance resulting in the degradation of
system performance or system outage.
2
12
- SNMP: Simple Network Management Protocol
- PERC: Power Edge RAID Controller
- RAID: Redundant Array of Inexpensive (or Independent) Drives
- Software Bug: Any error in the IVR code, known or unknown, which
causes the system to perform abnormally.
- System Bug: Any error in support software, (i.e. NT Server, SQL7.0,
etc.) protocols, transports, or other commercially available software.
SERVICE TICKET CLASSIFICATION
1. Priority 1: (Critical) The Post View system is down and conference
customers cannot listen to and or record conferences. Restoration
will occur as fast as humanly possible with an objective of 15
minutes or less.
2. Priority 2: (Major) Fifty percent or more of the system is unavailable
for any reason. System is still functional but capacity is severely
reduced. Restoration will occur as fast as humanly possible with an
objective of 1 hour or less.
3. Priority 3: (Minor) Any condition that results in the Post View
system performing at less than optimal performance. This includes;
slow performance during administration, recording, and replay, 1 or
more voice communications lines down, 1 or more IVR workstations
down, disk space reaches 85 percent of capacity, any other trouble
that results in degradation in performance. Restoration will occur as
fast as humanly possible with an objective of 3 hours or less.
BUG TICKET CLASSIFICATION
1. Priority 1: (Critical) Any code error or violation in the Interactive
Telesis developed applications or databases that results in the Post
View system being unavailable. Repair will occur as fast as humanly
possible with an objective of 15 minutes or less.
2. Priority 2: (Major) Any code error or violation in the Interactive
Telesis developed applications or databases that results in the Post
View system operating at a reduced capacity. Repair will occur as
fast as humanly possible with an objective of 1 hour or less.
3. Priority 3: (Minor) Code errors or violations in the Interactive
Telesis developed applications or databases that should be fixed to
enhance service or improve performance. Repair will occur according
to a scheduled implementation plan after a significant amount of
testing has occurred.
4. Priority 4: (warranted) Errors in software developed by companies
other than ITI and used to host the Post View system, e.g. bugs in
Microsoft or any other commercially available software. Repair will
occur in accordance with the software manufacturer's bug fixing plan.
3
13
SYSTEM MAINTENANCE
Whenever possible any system maintenance will be performed in a non-intrusive
manner consistent with insuring maximum availability of the Post View system.
Where available software utilities will be used to automate and enhance the
maintenance procedures. A framework like Unicenter TNG or HP OpenView will be
used to monitor and report on system health and performance. Additional tools
will be deployed to build a layered approach to maintenance. All automated
maintenance will occur as background processes and have no effect on system
performance. The first line of defense is the monitoring of system performance
and capacity. The second line of defense is the proactive maintenance of the
system on a scheduled basis. The third line of defense is the exception
maintenance conducted just before peak periods to insure maximum availability
of the Post View application. All maintenance will be performed under the
supervision of the Manager, Corporate Systems who is directly responsible for
all corporate systems. Technicians qualified on the monitoring and maintenance
procedures of Interactive Telesis will perform monitoring and maintenance.
These steps are detailed below:
DISK MAINTENANCE
All disks will be maintained in the readiest state available.
- On RAID arrays the PERC Flexible Array Storage Tool (FAST) or the PERC SC
RAID Console utility is to be used for consistency monitoring of the
array. Monitoring will be continuous and a specific consistency check will
occur at 12:00 AM every Saturday night.
- A hot swappable standby drive will be ready on all RAID 5 configurations
in case of a drive in the array failing.
- Consistency checks will be performed on non-RAID disk drives using,
Executive Software's, NT utilities. This will occur nightly with full
defragmentation of the drives occurring every Sunday at noon.
NETWORK MAINTENANCE
All network cabling, switch, and hub equipment is maintained on a scheduled
basis.
- Wire management trenches are used to route cabling away from traffic areas.
- Cables will be labeled to identify the devices connected to them. This
list will be maintained in an excel spreadsheet.
- Cables will be inspected on a quarterly basis for fraying, cracking, and
loose connectors. Damaged cables will be replaced immediately.
- Switches will be maintained and monitored through the Network Node
Management station. During peak periods, a sniffer may be deployed to
determine if any bottlenecks are occurring in the network or transport
layer. Corrective action will be taken immediately to relieve any
occurrences of above items.
4
14
OTHER DEVICES
. Board diagnostics are performed on a quarterly basis for all Dialogic
240SC T1 cards.
. Network interface cards will be monitored via SNMP and replaced
immediately if performance drops below 85% of advertised capability.
. Backup tape drives are cleaned after every 80 hours of usage.
. UPS are maintained in accordance with the manufacturer's suggested
maintenance plan.
. CSU's will be monitored via RMON probes and the network management
station.
DATABASE MAINTENANCE
All databases have a weekly scheduled maintenance plan run by the SQL
Agent.
. Transaction logs are dumped nightly at 1:00 AM to a backup device
(disk). During this process a consistency check and file resizing will
occur.
. Databases are dumped to a backup device daily at 12:00 midnight for
user databases and 1:00 AM for system databases. Consistency checks
are performed before databases are being dumped.
. Query optimization and analysis tools are enabled to allow SQL 7.0 to
performance tune itself automatically.
. Databases are configured to grow by 10% when needed and all logs are
truncated on checkpoint. Performance monitoring and index resizing
recommendations are written to a file for analysis and implementation
at the appropriate time and place.
. Transactional replication occurs "in real time" to the backup database
server and is designed to take over if the primary server fails.
VIRUS PROTECTION
The Post View system is a partially closed loop system with connectivity to
the Internet via a firewall. System management is achieved via VPN PPTP
connection over the Internet. There will be no direct downloads of files or
software to the Post View file servers. Only "shrink wrapped" software,
voice files, and ITI application code will be installed or copied to the
Post View system. This will significantly reduce the potential for viruses
effecting system availability. Additionally the following will occur:
. McAfee Netshield and Webscan X are installed and running on the
Firewall.
. A scheduled full scan of all server logical volumes will occur, from
the firewall, every Tuesday starting at 11:00 PM.
. All IVR machines will be scanned on a monthly rotation. Three machines
per week rotating so that each machine gets a full scan at least once
per month. Only the Post View IVR applications will be installed on
the IVR workstations. No other files or software will be installed, on
the workstations, with the exception of diagnostic utilities.
5
15
- If a virus is introduced to a system, that system will be physically
disconnected from the network (LAN cable unplugged from the NIC) and
manually cleaned of the virus. If the Manager, Corporate Systems cannot
certify the system clean then the drive will be formatted and the machine
rebuilt.
- An incident report will be generated and submitted to the Manager,
Corporate Systems within 1 hour of a suspected or confirmed virus
incident.
SYSTEM BACKUP
Without exception, all logical volumes on the servers will have a weekly full
backup and a daily incremental backup. Tapes are changed on a grandfather,
father, and son rotation. Archive jobs will be run on an as needed basis
depending on system volume.
- Full backups with verify will occur every Thursday starting at 12:30 AM.
All volumes are written to tape at this time.
- Incremental backups with verify will occur every day except Thursday at
12:30 AM. Only files that have been changed since the last full backup or
incremental backup are written to tape at this time.
- The backup operator is responsible for reviewing the backup reports to
insure complete and verifiable backups.
- Tapes are secured in an Underwriters Laboratory (UL) approved fire proof
safe both on site and off site. The current working set in the auto
loader, second most current in safe on site, third most current in safe
off site, fourth set in off site storage location.
- One month of normal and incremental tapes will be available for
restoration. Archive tapes are kept for 6 months from date of archive
before being recycled.
- Manager, Corporate Systems is directly responsible for management and
implementation of the data storage procedures.
- During normal business hours, the NOC will be alerted electronically if a
back up job fails. During non-business hours the on call technician will
be paged.
- In the event of critical data loss the most current full backup will be
restored to the disk and current incremental tapes restored to bring the
system up to the existing storage level. The Manager, Corporate Systems
will closely monitor this procedure to insure accurate and rapid
restoration of all corrupted data.
- The Network Administrator will coordinate and monitor the restoration of
individual files on an as needed or requested by the Post View
administrators.
6
16
SYSTEM MONITORING
All critical functions of the Post View system will be monitored on a 24/7
basis. This includes LAN throughput, CPU utilization, storage capacity, memory
allocation, database performance, network availability (T1 lines), capacity, and
general system health. Since our Network Operation Center (NOC) does not operate
on a 24/7 basis, the Computer Associates Framework (Unicenter TNG subset) is the
monitoring tool set we use. From 7:00 AM until 6:00 PM Monday through Friday
(normal operating hours) the NOC will be responsible for system monitoring and
maintenance. All other times the on call technician will be responsible for
system monitoring and maintenance. Alerts are programmed to trigger events at
the warning stage and to page the on call technician during non-business hours.
o Weekly reports will be submitted to the Manager, Corporate Systems for
review. Reports will include storage capacity, intraday system capacity,
performance matrix, and any other general information pertinent to the
operation of the system.
o Alerts have 4 thresholds for reporting purposes;
1. Critical: System has had a major malfunction and system is down.
2. Warning: An abnormal condition has occurred, however the system is
still functional. The system will alarm in the NOC during normal
business hours or page the on call technician during off business
hours.
3. Informational: Event has occurred and information is being logged
detailing the event. System is fully functional.
4. Exception: An event has occurred but no information was logged. A
dialog box may appear for administrative intervention, however no
data is logged to the event log. System is fully functional and no
abnormal condition has occurred.
o The Network Administrator reviews all event logs on a weekly basis for
recurring events and unusual conditions. Additionally, the event logs are
saved to a monthly folder every week and cleared in the on-line event log.
o Manager, Corporate Systems reviews monthly event folders and will submit a
quarterly system status report to the Director of Operations.
o Immediate action will be taken by either the NOC or the on call technician
to resolve events reported by the monitoring utilities.
o If any server volume reaches 85 percent of capacity additional storage
will be added "on the fly" using the FAST tool to initialize the stand by
drive.
o Any unusual or abnormal conditions are to be reported to the Network
Administrator and the Manager, Corporate Systems as soon as possible after
the event.
7
17
SYSTEM SUPPORT
Post View system support is provided via our "XxxxxxxXxxxxxx.xxx" solution
support operation. This new value added service is designed to "surround our
customer in service". It consists of a web site, email support, and an 800
number (0-000-000-0000). We will use it to track all issues and bugs related to
the Post View system. You will receive an issue tracking number and be able to
get status updates via the web site. Future enhancements include access via the
"OSMI" as well.
PROCEDURES
When an issue arises with the Post View system, a service request will be
submitted to the ITI partner support group. The first level technician will
take down pertinent information, open a ticket and perform preliminary
troubleshooting procedures to determine if the issue is on the LAN or the
telecommunications network. If the problem resides on the telecom network, the
Frontier Confertech Network Management Center (NMC) will be conferenced into
the call and the issue will be gracefully passed to them for resolution with
the long haul carriers. If the issue is determined to be within an ITI managed
network the following will occur.
o A case number is assigned in the system and user contact information will
be entered. A detailed description of the issue will be entered and the
issue is prioritized based on the service ticket classification explained
in the previous section.
o For priority 1 issues, the NOC will be notified of the issue as fast as
humanly possible with an objective of 15 minutes or less. If the incident
occurs after hours the on call technician will be notified immediately via
alphanumeric paging. The objective being to return the system to
operational status ASAP.
o For priority 2 issues, the same steps will be taken as in priority 1
however, the ticket will be passed through normal communications channels
with resolution being 1 hour or less. If the issue occurs after hours the
on call technician will receive an alert page and will continue
troubleshooting procedures to determine what level of repair needs to be
accomplished immediately. If service is performing at an acceptable level,
the repairs will commence at the start of the next business day. The
Manager, Corporate Systems will make the decision when determining if a
system is performing at an acceptable level.
o For priority 3 issues, a ticket will be opened and a case number assigned
to the Network Administrator for resolution. The Network Administrator
will assess the situation, determine the best course of action, and
schedule the repair of the issue to insure no interruption of service.
o The same procedures will apply for software bug tracking as well.
Additionally, a priority 4 bug will be recorded and submitted to the
appropriate software vendor for analysis and resolution in accordance with
their established procedures.
o Priority 1 and 2 software bugs will be addressed immediately, priority 3
and 4 bugs will be implemented on a mutually agreed upon schedule.
8
18
ESCALATIONS
All escalations will occur automatically at pre-determined times. Frontier
Confertech and Interactive Telesis employees may request escalation of an
issue if they feel it is warranted. It is important to note that if all
issues are submitted as priority one tickets then priority one issues
become normal an the effectiveness of the escalation process is diminished.
If service level and responsiveness is not meeting your expectations,
please do not hesitate to contact the Director of Operations at
000-000-0000. All escalations will go from a first level technician to the
next appropriate person in the process.
- Priority 1 issues are escalated to the Manager, Corporate Systems if
not resolved within 30 minutes of ticket issue.
- Priority 2 issues are escalated to the Network Administrator if not
resolved within 3 hours of ticket issue. If the issue is not resolved
within 6 hours, the issues are escalated to the Manager, Corporate
Systems.
- Priority 3 issues are escalated to the Network Administrator if not
resolved within 6 hours of ticket issue. If the issues are not
resolved by next business day, the issues are escalated to the
Manager, Corporate Systems.
- Priority 1 bugs are escalated to the Director, It if not resolved
within 30 minutes of ticket issue.
- Priority 2 bugs are escalated to a Sr. Developer if not resolved
within 6 hours of ticket issue. If the bugs are not resolved within 3
days, the bugs are escalated to the Director, IT.
- Priority 3 bugs are not escalated through the normal escalation
process. They will be identified and scheduled for correction through
the Director, IT.
- Priority 4 bugs will not be tracked through the system however, they
will be logged with the MSDN for evaluation.
KEY CONTACTS
Primary point of contact for service and support will be the
XxxxxxxXxxxxxx.xxx group. This may be via email, Internet, and 800 number.
This is the first place to go when you have an issue. If you have a problem
getting your issues resolved the following people can be contacted.
- Director of Operations
Email: xxxxx@xxxxxxxxx.xxx
(000) 000-0000 (24/7 number that forwards to cell phone)
(000) 000-0000 (pager)
- VP, Operations
Email: xxxxx@xxxxxxxxx.xxx
(760) 632-1700 ext. 262
(000) 000-0000 (pager)
(000) 000-0000 (cell phone)
A separate more complete list of key contacts will be provided.
9
19
SYSTEM CAPACITY PLANNING AND UPGRADE
All system upgrades are scheduled to avoid confusion and downtime.
Coordination between the Post View administrators and Interactive
Telesis is critical to a successful upgrade. System and storage capacity
reports will be reviewed regularly, by the management team, to analyze
trends and better prepare for peak seasons.
- When storage reaches 65 percent of capacity, the Network administrator
will institute the standing PO for a new hot swappable hard drive. At
85 percent full the standby drive is initialized and added to the
file systems logical volume. When the replacement drive is received,
it will be put into the standby bay and low level formatted.
- An archive program will be run at least 3 times per week to move
expired voice files to the alternate file system for backup to an
archive tape and deletion from the system.
- When incoming T1 ports reach 75 percent of capacity, provisioning for
additional port capacity will occur. Additional capacity will be
captured from existing IVR systems and or standby IVR workstations
may be deployed to meet capacity during peak system usage.
- Analysis of usage will occur quarterly and after every "IR season" to
determine capacity and plan for the next period of heavy usage.
- IVR Application upgrades will be categorized as either an enhancement
or a fix to an existing problem. Bug fixes will be thoroughly tested
in a development environment and then ported to 1 or 2 production
IVR's. Once the fix has been validated on the product system the
remainder of the IVR's will be brought up to the latest build and a
new release will be generated.
- The same procedures apply for software enhancements, the only
difference being a new version is generated instead of a new release.
Additionally, enhancements are to be scheduled outside of the bug
tracking system.
10
