ISO Log and Data Retention Procedures

This procedures defines data, log and SIEM retention size, schedule and time and references operational documents related to each data store, index, raw log.

ITSO Log and Data Retention Procedures

Procedure Description:  This procedures outlines the operational intent and practice related to data, raw logs, indexed SIEM  (Security Information and Event Management) data with regard to the source, retention method, retention size or time and special circumstances.  The procedure is maintained for standardization and repetition of the process, to define the operational intent and duties and to provide audit practices.

 Date  Modification Type
 Modified by
 1/30/2018  Updated from old format, expanded details
 2/5/2018  Adding Gap services not included in current logs

SIEM: A service sometimes incorporating a raw log correlation engine that allows multiple sources of security information to be collected, tracked and correlated into actionable data.

Raw Logs: Service, system and network data initiated from the source system or service in a standard log format as defined by a service protocol or system. Raw logs catalog the functional operation of a system or service and are used for diagnostic or service evaluation processes.  

Operational Data: Documents, email, work files, configurations, procedures, policies, memos, notes, calendar entries,
minutes, project plans, .. etc. that are data generally created by people and not systems or services as part of the operation of the organization. 

Centralized Logging: A best practice tenant of security logging following separation of duty requirements.  Logs of critical services and systems should be logged to a log service real time so that in the event of a service affecting outage of the service, a copy of the events leading up to the outage could be maintained to support root cause analysis.

Retention Metric: Retention metrics specify the measurement criteria used to limit how data is retained.  There are 2 metrics TIME and SIZE.  All Raw Logs are retained by TIME and by default are kept no longer than 62 days as policy unless requested and approved by management.  SIEM logs are retained by either TIME or SIZE of the log.  Due to factors such as high and/or frequent output rates, restricted storage space, competition for storage space with longer held data, importance hierarchy of some logs, some logs are deleted sooner than others. Logs rotated using size metrics have indeterminate retention times.

Operating Standards: The Office of Information Security has access, visibility and control of many forms of data.  Raw log and SIEM data are defined separately and by nature tend to overlap.  Some data are operational data such as files and emails that are associated with operational and administrative staff  of the security office, some related to investigations, data under e-discovery restrictions and others related to incidents under review.  Operational data is NOT in scope under this procedure, only raw log and SIEM data that is routinely accumulated is covered.  Copies of raw log and SIEM data may become operational with special handling considerations when under investigation.  These copies are out of scope for this procedure.

Centralized Raw Logs: 
All centralized logs will be initiated by a help desk ticket.  The system/service owner will provide the following MINIMUM Required information in the request.
  • Source IP of the log data
  • Requested protocol and port
  • The default retention for all raw logs will be 62 days unless otherwise required and approved by administration
  • The source system MUST utilize institutional NTP services
  • The source log MUST utilize standard syslog services 
  • The source system owner MUST have a syslog procedure and protocol document.

 Owner Org
 Log Source
 Maximum Retention in days
 Also in SIEM
VMWare VCenter
Windows Servers
kadmin 62  No
 IAM ssh_duo
 62  Yes
kdc  62  Yes
 NDCO Radius  62  Yes
 NDCO dhcp  62  Yes
 NDCO ib_dhcp  62  Yes
 NDCO ib_dsn  62  Yes
 NDCO ib_system  62  Yes
 ORG Linux Org Servers
 62  No
 SECI POP  62  No
 SECI Perdition
 62  Yes
 62  Yes
 SECI IMAP  62  Yes
 SECI Stunnel  62  Yes
 SECI nginx  62  Yes
 62  Yes
 SECI Webmail  62  Yes
 SECI Proofpoint  62  Yes
 NDCO netudp_nat
 62  Yes
 NDCONetwork (Core device logs)
 Not Centrally Logged
 NDCONetwork (Dist device logs)
 Not Centrally Logged 
 NDCONetwork (Access device logs)
 Not Centrally Logged 
 CRCStorage (isilon - Research)
 Not Centrally Logged 
 AESSStorage (3par)
 Not Centrally Logged
 AESSStorage FC Network
 Not Centrally Logged 
 AESSStorage (isilon - non-research)
 Not Centrally Logged 
 LEAWS system logs
 Not Centrally Logged 
 IAMAD DC logs
 Not Centrally Logged 
 AESSDatabase Logs BI
 Not Centrally Logged 
 AESSBanner Logs
 Not Centrally Logged 
 AdmissionsSlate Access Logs
 Not Centrally Logged 
 UG Dean
Titanium System Logs
 Not Centrally Logged 
 UG Dean
eClinical Access Logs
 Not Centrally Logged 
 ProvostVRDE VDI System Logs
 Not Centrally Logged 
 SECIRT Help System Logs
 Not Centrally Logged 
 IAMOwlDB System Logs
 Not Centrally Logged 
 SECITelecom System Logs
 Not Centrally Logged 
 SECIVOIP System Logs
 Not Centrally Logged 
 LELMS System Logs
 Not Centrally Logged 
 LECrestron System Logs
 Not Centrally Logged 
 LEComputer Lab System Logs
 Not Centrally Logged 
 CRCCRC Systems Logs
 Not Centrally Logged 
 AESSBackup (Comvault) System Logs
 Not Centrally Logged 
 AESSBackup Logs
 Not Centrally Logged 
 AthleticsVideo Systems Logs
 Not Centrally Logged 
 ParkingParking SkiData
 Not Centrally Logged 
 VariousPCI Terminals
 Not Centrally Logged 
 DevelopmentVarious systems
 Not Centrally Logged 
 RUPDVarious systems
 Not Centrally Logged 
 Not Centrally Logged 
 H&DVarious Systems
 Not Centrally Logged 
 IAMCBORD (Jeff is looking)
 Not Centrally Logged

SIEM Logs:  The SIEM accumulates raw logs and indexes them to make searching faster.  Some data comes from centralized log services, some from API sources and others from flat files. SIEM log retention is configured per source.  SIEM logs are called indexes and are maintained in a retention hierarchy based on age.  The hierarchy container is called a bucket and the tiers are hot, warm, cold and frozen based on time from creation within an index. Default bucket size and rotation periods are set to global settings unless overridden by index's configuration.  Data is automatically rolled from most recent to oldest based on configuration defined by the SIEM admin.  When indexes grow to above 500, GB the oldest data is frozen regardless of the configured 90 day freeze setting. After data is rotated to a frozen state, it is no longer available to searches without manually thawing.

Index age hierarchy and aging policy

  • Hot = 1 day (current days data).  Contains newly indexed raw log data, all data hot only 1 day (10 buckets max)
  • Warm = data rolled from Hot (15 buckets max, Default = 300 if not defined) no real time updates to bucket
  • Cold = Default rotation not defined, rolls when max size limit is met
  • Frozen = Cold 90 days old OR larger than 500 GB, data is frozen and not searchable without thawing (archives)
  • Thawed = data restored from archive - ad hoc
Maximum data storage = 500,000 MB

Index Source
 Size 500 GB
 Maximum Thawed retention time. Default (1+15+90)=106 days
Max Frozen Retention Time. Default = 6 years, by storage size we keep 1 year
 dhcp  Default Default
 imap-proxy  Default Default  
 infoblox_dhcp  Default Default 
 infoblox_dns  Default Default 
 Default Default 
 maillog  Default Default 
 postfix  Default Default 
 nat  750000 MB
 nginx  Default Default 
 proofpoint  Default Default 
 radius  Default Default 
 shibboleth  Default Default 
 stunnel  Default Default 
 webmail  Default Default 
 Shibboleth DefaultDefault 
 cisco-ise DefaultDefault 
 extrahop DefaultDefault 
 trendmicro DefaultDefault 
 cisco_firewall DefaultDefault 
 pan_logs DefaultDefault 
 cisco_cleanacces DefaultDefault 
 cisco_ios DefaultDefault 
 cisco_wireless DefaultDefault 
 symantec_pgp DefaultDefault 
 msad DefaultDefault 
 permon DefaultDefault 
 winevents DefaultDefault 
 windows DefaultDefault 
 wineventlog DefaultDefault 
 os DefaultDefault 
 firedalerts DefaultDefault 
 unix_summary DefaultDefault 
 vmware-esxilog DefaultDefault 
 dmca Default 380 days
 masscan Default 380 days
 tenable Default 380 days
 oit_crc Default183 days
 Default183 days
 Default5 years 15 days
 kerberos Default Default  
 box DefaultDefault 
 duo DefaultDefault 
 pps DefaultDefault 
 googleapps DefaultDefault 
 bro DefaultDefault 

Keywords:Logs, SIEM, Data Retention, procedure itso   Doc ID:79735
Owner:Barry R.Group:Rice University
Created:2018-01-30 07:37 CDTUpdated:2019-04-01 07:21 CDT
Sites:Rice University
Feedback:  0   0