[Scamper-dev] scamper data collection

[Matthew Luckie]

This is the data scamper currently collects as part of the warts file 
format, and how it does it.  The structs that expose the data to users are 
presented.

Where I use 'ttl', substitute 'hlim' for the case where the data collected
is for an IPv6 path.  IPv6 addresses are collected are stored in a
dictionary once, and a 32 bit address id is used for this address family
whenever an address is written to disk.

   typedef struct scamper_trace
   {
     /* source and destination addresses of the trace */
     sa_family_t      af;
     void            *src;
     void            *dst;

     /* when the trace commenced */
     struct timeval   start;

     /* hops array, length specified by hop_count */
     scamper_hop_t  **hops;
     uint8_t          hop_count;

     /* why the trace finished */
     uint8_t          stop_reason;
     uint8_t          stop_data;

     /* trace parameters */
     uint8_t          flags;
     uint8_t          attempts;
     uint8_t          hoplimit;
     uint16_t         sport;
     uint16_t         dport;

     /* if we perform PMTU discovery on the trace, then record data here */
     scamper_pmtu_t  *pmtu;

     /* the current list, cycle, and defaults */
     scamper_list_t  *list;
     scamper_cycle_t *cycle;

     /* this variable is free for any user of this struct to use */
     void            *param;
   } scamper_trace_t;

The scamper_trace struct is the fundamental structure read and written
from a warts file.  Each trace record has the src address, dst address,
address family, start time, stop reason, stop data, list id, and cycle id
stored unconditionally.

The hops variable is an array of pointers to hop structs, ordered by the
distance from the source - i.e. the responses, if any, to a probe with ttl
1 are stored in hops[0].  The hop_count variable specifies the largest TTL
sent for which we got a response.

The stop_reason and stop_data variables indicate the reason for halting 
the
trace.  The currently defined stop reasons are:

#define SCAMPER_TRACE_STOP_NONE      0x00 /* null reason */
#define SCAMPER_TRACE_STOP_COMPLETED 0x01 /* got an ICMP port unreach */
#define SCAMPER_TRACE_STOP_UNREACH   0x02 /* got an other ICMP unreach code */
#define SCAMPER_TRACE_STOP_ICMP      0x03 /* got an ICMP msg, not unreach */
#define SCAMPER_TRACE_STOP_LOOP      0x04 /* loop detected */
#define SCAMPER_TRACE_STOP_DEAD      0x05 /* unresponsive target */
#define SCAMPER_TRACE_STOP_ERROR     0x06 /* sendto error */

* For the COMPLETED, UNREACH, and ICMP stop reasons, the probe_ttl of the
   packet(s) that caused the stop reason to occur are stored in the stop_data
   slot.
* For the LOOP stop reason, the loop length is stored.
* For the DEAD stop reason, the number of consecutive unresponsive hops
   are stored.
* For the ERROR stop reason, the errno returned from sendto is stored.
   The motivation for this stop reason is to capture codes like ENETUNREACH
   when scamper is fed an address that the machine has no route for.  This
   has happened in the past.  One problem with storing errno numbers are
   that they are system specific, so the same errno numbers will mean
   different things on different platforms.  Perhaps a dictionary might be
   defined for this stop reason.

The trace parameters (flags, attempts, hoplimit, sport, dport) are stored
conditionally if they are different to the parameters for the cycle id.
These parameters are defined below when the cycle parameters are shown.

The scamper_pmtu, scamper_list, and scamper_cycle structs are defined 
below.

The param is a variable not written to disk, but for users of the trace 
struct to record whatever they need to with this struct.  I use it in 
scamper to attach state to each trace during the probe process.

   typedef struct scamper_hop
   {
     /* the address of the hop that responded */
     void               *addr;
     sa_family_t         af;

     /*
      * probe_id:  the attempt # this probe is in response to [count from 0]
      * probe_ttl: the ttl that we sent to the trace->dst
      * turn_ttl:  the ttl of the probe packet received at trace->dst
      * reply_ttl: the ip->ttl that we received from hop->addr [0 if unset]
      */
     uint8_t             probe_id;
     uint8_t             probe_ttl;
     uint8_t             turn_ttl;
     uint8_t             reply_ttl;

     /* icmp type / code returned by this hop */
     uint8_t             icmp_type;
     uint8_t             icmp_code;

     /* time elapsed between sending the probe and receiving this resp */
     struct timeval      rtt;

     /*
      * if the ICMP message was a fragmentation required message, then
      * record the MTU returned.  this variable could probably go into
      * a union in case there are other data items to record as well,
      * as this variable is only necessary for one ICMP message type.
      */
     uint16_t            mtu;

     struct scamper_hop *next;
   } scamper_hop_t;

The scamper_hop struct collects response data from a probe that caused an
ICMP response.  Each hop record written to disk has the address, 
probe_ttl, and reply_ttl recorded unconditionally.

* The probe_id is recorded conditionally if the response is not for
   probe_id zero, i.e. the response was not to the first probe sent.

* The rtt is recorded conditionally if the rtt field is not zero, i.e. the
   RTT of the time from sending a probe to receiving a response was captured.
   It is stored as a 32 bit integer counting microseconds.

* The turn ttl is the TTL value of the probe when the probe caused the ICMP
   response.  It is recorded conditionally if it is not one.  The TTL field
   of the IP packet returned inside of an ICMP TTL expired response is one.

* The icmp_type and icmp_code values are recorded conditionally if the
   ICMP response to the probe is not an ICMP TTL expired in transit response.

* The mtu is recorded conditionally if the ICMP response to the probe is
   a fragmentation required response, and the suggested packet size is not
   zero.

No hop records are stored for unresponsive hops.  scamper can probe each
hop for more than one response as traceroute does.  The request for this
functionality was made by Kenjiro.  If scamper receives more than one
response for a particular probe_ttl during the traceroute phase, it is
linked into the list for this probe_ttl.  The list is ordered by probe_id,
then rtt.

   typedef struct scamper_pmtu
   {
     uint16_t       ifmtu; /* the outgoing interface's MTU */
     uint16_t       pmtu;  /* the packet size we reached the target with */
     scamper_hop_t *hops;  /* list of hops that returned icmp messages */
   } scamper_pmtu_t;

If scamper conducted path MTU discovery towards a destination, then a
scamper_pmtu trace attribute is written to disk with the trace.
The hops are stored in a list, ordered by the turn_ttl.  If we reach the
target using the suggested MTUs returned in ICMP messages, then the size
of the packet is recorded in pmtu.  If we get no response from the target,
then the variable is set to zero.

   typedef struct scamper_list
   {
     uint32_t  id;
     char     *listname;
     char     *descr;
   } scamper_list_t;

   typedef struct scamper_cycle
   {
     uint32_t        id;
     uint8_t         flags;
     uint8_t         attempts;
     uint8_t         hoplimit;
     uint16_t        sport;
     uint16_t        dport;
   } scamper_cycle_t;

These are the variables that I store with the list and cycle start 
records.
The cycle object contains the default parameters supplied for the cycle by
the list manager, or by scamper if the list manager does not specify them.
The currently defined flags are:

#define SCAMPER_TRACE_FLAG_ALLATTEMPTS 0x01 /* send all allotted attempts */
#define SCAMPER_TRACE_FLAG_PMTU        0x02 /* conduct PMTU discovery */

Other flags that could be defined are UDP, TCP, and ICMP, depending on
support being added to scamper to probe using those protocols.